Inconsistent results with Optimize Parameters (Grid)

lsevel
lsevel New Altair Community Member
edited November 2024 in Community Q&A
Hi all,

I've been working with some extracted connectivity values from fMRI data and am attempting to use Optimize Parameters (Grid) to determine parameter values within a stacked model. (Optimize Parameters-->Cross Validation-->Stacking, etc). I've found that my accuracy values with an optimized model performed in the Optimize Parameters operator (86.67%) are different from those performed with ostensibly the same parameters as those chosen in the Optimize operator but when performed with only cross validation (Cross Validation-->Stacking, etc) (accuracy = 77.50%). Is this difference to be expected? If so, which operator provides the most valid results?

Thank you,
Tagged:

Answers

  • MartinLiebig
    MartinLiebig
    Altair Employee
    Hi,

    are you sure that your optimization does not yield to overfitting?

    ~Martin
  • lsevel
    lsevel New Altair Community Member
    I suppose that could be the case but then presumably the model would still seem overfit in the cross validation step performed within optimization?
  • earmijo
    earmijo New Altair Community Member
    If the sample size is too small it can happen too. I would fix the random seed in the X-val operator. Then you should get exactly the same results.
  • JEdward
    JEdward New Altair Community Member
    Out of curiousity have you considered putting your X-Validation inside the optimise parameters? 
    This might help prevent overfitting. 

    See below for a crude example.
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="7.0.000">
      <operator activated="true" class="log" compatibility="7.0.000" expanded="true" height="82" name="Log" width="90" x="581" y="85">
        <parameter key="filename" value="D:\log_values.txt"/>
        <list key="log">
          <parameter key="Count" value="operator.SVM.value.applycount"/>
          <parameter key=" Testing Error" value="operator.Performance.value.performance"/>
          <parameter key="Training Error" value="operator.Performance (2).value.performance"/>
          <parameter key="SVM C" value="operator.SVM.parameter.C"/>
          <parameter key="SVM gamma" value="operator.SVM.parameter.gamma"/>
        </list>
        <parameter key="sorting_type" value="none"/>
        <parameter key="sorting_k" value="100"/>
        <parameter key="persistent" value="false"/>
      </operator>
    </process>
  • lsevel
    lsevel New Altair Community Member
    Sorry if my initial post wasn't clear--I do with x-validation within optimize (and stacking within that cross validation). However, when using just x-validation with the same parameters found in the optimize (with x-val nested) I get different results.