Feature Weighting: Filter, Wrapper and Embedded - Some Questions
Hello everyone,
in the last few days I have looked more closely at the Feature Selection in order to compare the different possibilities with each other. Some questions came up and I would be grateful if you could help me.
In general I learned that FS methods can be sorted into "filter", "wrapper" and "embedded" methods, so I tried something from each approach. My dataset consists of about 20 numeric attributes and 10.000 examples, with the goal of identifying the most relevant features for a numeric label.
in the last few days I have looked more closely at the Feature Selection in order to compare the different possibilities with each other. Some questions came up and I would be grateful if you could help me.
In general I learned that FS methods can be sorted into "filter", "wrapper" and "embedded" methods, so I tried something from each approach. My dataset consists of about 20 numeric attributes and 10.000 examples, with the goal of identifying the most relevant features for a numeric label.
Filter Approach
I used the operators "Weight by Correlation" and "Weight by SVM", where I normalized the weights (Click on normalize weights" in the Parameters Panel) and can now compare the two weight directly, because I get values between 0 and 1. For example in Screenshot 1 are shown the results from "Weight by SVM", Left normalized and right not normalized for a later comparison.
Screenshot 1: Weight Results from "Weight by SVM" (left WITH normalization, right WITHOUT normalization)
Screenshot 3: Weight Results from "Modell SVM" (left WITH normalization, right WITHOUT normalization) There are both 18 attributes only the presentation is slightly different, please do not be confused by the height of the two screenshots)
My goal is actually to get weights of the features between 0 and 1 for all methods that I used, so that I can compare all methods concretely with each other.
Thank you very much.
Screenshot 1: Weight Results from "Weight by SVM" (left WITH normalization, right WITHOUT normalization)
Wrapper Approach
In this context I used the "Forward Selection", "Backward Elimination" and the "Optimize Selection (Evolutionary)" operator. Is it possible to get weights of the features between 0 AND 1, because currently only a 0 OR a 1 is output for the weighting? If I understood that correctly, it is not possible, because different subsets are tested and subset with the best performance is selected, right? (I have followed Inga Mierswa's Blog Posts when doing the wrapper approaches, see LINK)
Embedded Approach
In this context I used the weighting function of the models. For example, the "SVM" operator or the "Linear Regression" operator offers the possibility to output the weights. The first question: Is it correct that this weighting by these operators is a so-called embedded approach? Furthermore, I would like to compare the results of the weighting by the models with the weighting by the filter methods. The problem is that I have not yet found a way to normalize the model-based weightings in the same way as with the filter methods. In the filter methods I can click on "normalize weight" in the parameter window. Unfortunately this is not the case with the models. I already tried to use the "Weights to data" operator and then the "normalize" operator (Screenshot 2) with method "range transformation" and min 0 and max 1, but I don't get the right results (Screenshot 3). The weights that are not normalized are approximately the same (see screenshot 1 and 3 on the right), only the normalized weights are not (see screenshot 1 and 3 on the left). For example, the attribute 0.07_ below has a weight of 0 while above it has a weight of 0.539?
Screenshot 2: Normalize the Weights of the model "SVM"
Screenshot 3: Weight Results from "Modell SVM" (left WITH normalization, right WITHOUT normalization) There are both 18 attributes only the presentation is slightly different, please do not be confused by the height of the two screenshots)
My goal is actually to get weights of the features between 0 and 1 for all methods that I used, so that I can compare all methods concretely with each other.
Thank you very much.
Best regards
Moritz