🎉Community Raffle - Win $25

An exclusive raffle opportunity for active members like you! Complete your profile, answer questions and get your first accepted badge to enter the raffle.
Join and Win

Inconsistent results with Optimize Parameters (Grid)

User: "lsevel"
New Altair Community Member
Updated by Jocelyn
Hi all,

I've been working with some extracted connectivity values from fMRI data and am attempting to use Optimize Parameters (Grid) to determine parameter values within a stacked model. (Optimize Parameters-->Cross Validation-->Stacking, etc). I've found that my accuracy values with an optimized model performed in the Optimize Parameters operator (86.67%) are different from those performed with ostensibly the same parameters as those chosen in the Optimize operator but when performed with only cross validation (Cross Validation-->Stacking, etc) (accuracy = 77.50%). Is this difference to be expected? If so, which operator provides the most valid results?

Thank you,

Find more posts tagged with

Sort by:
1 - 5 of 51
    Hi,

    are you sure that your optimization does not yield to overfitting?

    ~Martin
    User: "lsevel"
    New Altair Community Member
    OP
    I suppose that could be the case but then presumably the model would still seem overfit in the cross validation step performed within optimization?
    User: "earmijo"
    New Altair Community Member
    If the sample size is too small it can happen too. I would fix the random seed in the X-val operator. Then you should get exactly the same results.
    User: "JEdward"
    New Altair Community Member
    Out of curiousity have you considered putting your X-Validation inside the optimise parameters? 
    This might help prevent overfitting. 

    See below for a crude example.
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="7.0.000">
      <operator activated="true" class="log" compatibility="7.0.000" expanded="true" height="82" name="Log" width="90" x="581" y="85">
        <parameter key="filename" value="D:\log_values.txt"/>
        <list key="log">
          <parameter key="Count" value="operator.SVM.value.applycount"/>
          <parameter key=" Testing Error" value="operator.Performance.value.performance"/>
          <parameter key="Training Error" value="operator.Performance (2).value.performance"/>
          <parameter key="SVM C" value="operator.SVM.parameter.C"/>
          <parameter key="SVM gamma" value="operator.SVM.parameter.gamma"/>
        </list>
        <parameter key="sorting_type" value="none"/>
        <parameter key="sorting_k" value="100"/>
        <parameter key="persistent" value="false"/>
      </operator>
    </process>
    User: "lsevel"
    New Altair Community Member
    OP
    Sorry if my initial post wasn't clear--I do with x-validation within optimize (and stacking within that cross validation). However, when using just x-validation with the same parameters found in the optimize (with x-val nested) I get different results.