Inconsistent results with Optimize Parameters (Grid)

Question

Hi all,

I've been working with some extracted connectivity values from fMRI data and am attempting to use Optimize Parameters (Grid) to determine parameter values within a stacked model. (Optimize Parameters-->Cross Validation-->Stacking, etc). I've found that my accuracy values with an optimized model performed in the Optimize Parameters operator (86.67%) are different from those performed with ostensibly the same parameters as those chosen in the Optimize operator but when performed with only cross validation (Cross Validation-->Stacking, etc) (accuracy = 77.50%). Is this difference to be expected? If so, which operator provides the most valid results?

Thank you,

lsevel · Answer

Sorry if my initial post wasn't clear--I do with x-validation within optimize (and stacking within that cross validation). However, when using just x-validation with the same parameters found in the optimize (with x-val nested) I get different results.

JEdward · Answer

Out of curiousity have you considered putting your X-Validation inside the optimise parameters? This might help prevent overfitting. See below for a crude example.

earmijo · Answer

If the sample size is too small it can happen too. I would fix the random seed in the X-val operator. Then you should get exactly the same results.