X-Validation and Feature engenering

janvanrijn

I want to use some feature engineering steps inside a cross-validation loop. For example, I want to evaluate the combination of a PCA (preprocessing) and k-NN (modeling) algorithm. In order to do so, I use the X-Validation operator. However, as PCA changes the features that are available in the original dataset, I cannot in the testing simply connect the model and test set input to the "Apply Model" operator. I probably also need to do something with the preprocessing method.

I can see PCA has a preprocessing model as output, is that what I should use to overcome thing problem? If yes, how?

RalfKlinkenberg

The "Group Models" operator allows to bundle several models into one combined model. You can bundle your pre-processing model (PCA) and your k-NN model into one model that can be passed from the training subprocess of the cross-validation to the test subprocess.

janvanrijn

Dear Ralf,

Thanks for your quick reply. Something like I did here?
[img=http://s9.postimg.org/d36r9xaqj/pca.jpg]

I still get an error, hinting at the wrong set of attributes (included in the screen. )

MartinLiebig

hi,

sadly you need to add a Materialze data infront of your apply model operator. There is a bug in the PCA which is fixed in the next relase.

Cheers,
MArtin

janvanrijn

Awesome. It now works for me. Thanks both

One more question. Suppose I want to use the operators from the "Feature Extraction" extension, that do not output such model (and as far as I'm concerned, just return a weight vector with feature importance) how would I do that?

MartinLiebig

Hi,

i think you simply use select by weights afterwards. Thats it. Usually all models know on which attributes they are applied

By the way: My personal favorite is MRMR

best,
martin

janvanrijn

Seems to work pretty well indeed, thanks

One exception though, k-NN.. Whichever preprocessing method I choose, it always obtains the same scores.

MartinLiebig

Hi,

which version of RM are you using?
There are two ways how a application of the model can be done. Either via the attribute names or the attribute location (first attribute, second attribute.). All models implemented by us should take the col. name. There was something with k-NN, that it consideres something wrong. I know that it is fixed. But if you use 5.3 the bug is of course in.

In this case you need to get the weight vector to the apply side using the through port. For application outside of the X-Val you need to get the Vector out of the x-val. Remember/recall is the way to go here.

Regards,
Martin