Parameter Otimization - cant replicate results

pilatus

Dear Forum-Members,

I encountered the following problem, that I still can't resolve on my own:

Inside the parameter optimization I put a PCA operator followed by XValidation with Naive Bayes.
This results in an optimal number of principal components and the corresponding performance numbers.
So far, so good.
However, when I construct the same process without optimization and instead configure the PCA with
the optimal number of components taken from step 1, the performance does not match the one from step 1.
Why does this occur?

Regards,
P

land

Hi,
this probably occurs, because the sequence of random numbers used for the splitting in the XValidation is not the same unless you activated local random seed. But if you use a local random seed, you might overfit the number of components to this particular local random seed...

Anyway to rule out the option, that there's simply an error in the process design, please post your process here.

Greetings,
  Sebastian