Optimization Problem
Legacy User
New Altair Community Member
Hi,
I would like to find the best classification model w.r.t. accuracy
for a given example set. To achieve best results, my idea is to evaluate
different supervised learners and optimize their parameters. In addition,
different feature selection algorithms should be applied to provide most
suitable input for the parameter optimization of each learner.
So, my idea is something like a nested model:
for each learner that should be evaluated
for each example set determined by particular feature selection
perform parameter optimization for given feature set and learner
return: model model with maximal accuracy
What do you think about this idea? Does it make sense to mix a feature
selection and a learner parameter optimization to find the most accurate
model, i.e. to first Or whould you proceed differently in that case?
Are other approaches more common in practice?
In am of the opinion that the most accurate model can be only found
when different example sets are provided for the parameter optimization
to get a high number of combinations for the performance evaluation.
Correct me if I'm wrong. :-)
If my idea is OK, I would ask you to help me modelling this use case
in RapidMiner. It should be something like sample
05_Features/10_ForwardSelection.xml but not using just the NearestNeighbor
as learner but an parameter optimization problem like
07_Meta/01_ParameterOptimization.xml.
This is the code for the feature selection:
optimization problem. Could you help me?
Regards,
Martin
I would like to find the best classification model w.r.t. accuracy
for a given example set. To achieve best results, my idea is to evaluate
different supervised learners and optimize their parameters. In addition,
different feature selection algorithms should be applied to provide most
suitable input for the parameter optimization of each learner.
So, my idea is something like a nested model:
for each learner that should be evaluated
for each example set determined by particular feature selection
perform parameter optimization for given feature set and learner
return: model model with maximal accuracy
What do you think about this idea? Does it make sense to mix a feature
selection and a learner parameter optimization to find the most accurate
model, i.e. to first Or whould you proceed differently in that case?
Are other approaches more common in practice?
In am of the opinion that the most accurate model can be only found
when different example sets are provided for the parameter optimization
to get a high number of combinations for the performance evaluation.
Correct me if I'm wrong. :-)
If my idea is OK, I would ask you to help me modelling this use case
in RapidMiner. It should be something like sample
05_Features/10_ForwardSelection.xml but not using just the NearestNeighbor
as learner but an parameter optimization problem like
07_Meta/01_ParameterOptimization.xml.
This is the code for the feature selection:
but I don't get to replace the NearestNeighbor but an parameter
<operator name="Root" class="Process" expanded="yes">
<description text="#ylt#p#ygt# Transformations of the attribute space may ease
learning in a way, that simple learning schemes may be able to learn complex
functions. This is the basic idea of the kernel trick. But even without kernel
based learning schemes the transformation of feature space may be necessary to
reach good learning results. #ylt#/p#ygt# #ylt#p#ygt# RapidMiner offers several
different feature selection, construction, and extraction methods. This
selection experiment (the well known forward selection) uses an inner cross
validation for performance estimation. This building block serves as fitness
evaluation for all candidate feature sets. Since the performance of a certain
learning scheme is taken into account we refer to experiments of this type as
#yquot#wrapper approaches#yquot#.#ylt#/p#ygt# #ylt#p#ygt#Additionally the
experiment log operator plots intermediate results. You can inspect them online
in the Results tab. Please refer to the visualization sample experiments or the
RapidMiner tutorial for further details.#ylt#/p#ygt# #ylt#p#ygt# Try the
following: #ylt#ul#ygt# #ylt#li#ygt#Start the experiment and change to
#yquot#Result#yquot# view. There can be a plot selected. Plot the
#yquot#performance#yquot# against the #yquot#generation#yquot# of the feature
selection operator.#ylt#/li#ygt# #ylt#li#ygt#Select the feature selection
operator in the tree view. Change the search directory from forward (forward
selection) to backward (backward elimination). Restart the experiment. All
features will be selected.#ylt#/li#ygt# #ylt#li#ygt#Select the feature selection
operator. Right click to open the context menu and repace the operator by
another feature selection scheme (for example a genetic algorithm).#ylt#/li#ygt#
#ylt#li#ygt#Have a look at the list of the experiment log operator. Every time
it is applied it collects the specified data. Please refer to the RapidMiner
Tutorial for further explanations. After changing the feature selection operator
to the genetic algorithm approach, you have to specify the correct values.
#ylt#table#ygt##ylt#tr#ygt##ylt#td#ygt##ylt#icon#ygt#groups/24/visualization#ylt#/icon#ygt##ylt#/td#ygt##ylt#td#ygt##ylt#i#ygt#Use
the experiment log operator to log values
online.#ylt#/i#ygt##ylt#/td#ygt##ylt#/tr#ygt##ylt#/table#ygt# #ylt#/li#ygt#
#ylt#/ul#ygt# #ylt#/p#ygt#"/>
<operator name="Input" class="ExampleSource">
<parameter key="attributes" value="../data/polynomial.aml"/>
</operator>
<operator name="FS" class="FeatureSelection" expanded="yes">
<operator name="XValidation" class="XValidation" expanded="yes">
<parameter key="sampling_type" value="shuffled sampling"/>
<operator name="NearestNeighbors" class="NearestNeighbors">
<parameter key="k" value="5"/>
</operator>
<operator name="ApplierChain" class="OperatorChain" expanded="yes">
<operator name="Applier" class="ModelApplier">
<list key="application_parameters">
</list>
</operator>
<operator name="Performance" class="Performance">
</operator>
</operator>
</operator>
<operator name="ExpLog" class="ProcessLog">
<list key="log">
<parameter key="generation" value="operator.FS.value.generation"/>
<parameter key="performance" value="operator.FS.value.performance"/>
</list>
</operator>
</operator>
</operator>
optimization problem. Could you help me?
Regards,
Martin
Tagged:
0
Answers
-
Hi Martin,
just replace it with an operator delivering an Model. You could do anything inside an OperatorChain if a model will be returned at the end.
The general idea is great, this could solve all our problems. Small problem: Computational costs will be a little bit too high with real data sets, because of the double or tripple exponential nature of the search space...
Good luck
Greetings,
Sebastian0 -
Hi Sebastian et all,
thank you for your answer.
However, I'm still not able to replace the simple NearestNeighbor
model (used for the feature selection optimization) by an optimized
learner model returned by a GridParameterOptimization. This
is my current non-working model:
The problem is the first operand of operator XValidation which
<operator name="Root" class="Process" expanded="yes">
<description text="#ylt#p#ygt# Transformations of the attribute
space may ease learning in a way, that simple learning schemes may be
able to learn complex functions. This is the basic idea of the kernel
trick. But even without kernel based learning schemes the
transformation of feature space may be necessary to reach good learning
results. #ylt#/p#ygt# #ylt#p#ygt# RapidMiner offers several different
feature selection, construction, and extraction methods. This selection
experiment (the well known forward selection) uses an inner cross
validation for performance estimation. This building block serves as
fitness evaluation for all candidate feature sets. Since the
performance of a certain learning scheme is taken into account we refer
to experiments of this type as #yquot#wrapper
approaches#yquot#.#ylt#/p#ygt# #ylt#p#ygt#Additionally the experiment
log operator plots intermediate results. You can inspect them online in
the Results tab. Please refer to the visualization sample experiments
or the RapidMiner tutorial for further details.#ylt#/p#ygt#
#ylt#p#ygt# Try the following: #ylt#ul#ygt# #ylt#li#ygt#Start the
experiment and change to #yquot#Result#yquot# view. There can be a plot
selected. Plot the #yquot#performance#yquot# against the
#yquot#generation#yquot# of the feature selection
operator.#ylt#/li#ygt# #ylt#li#ygt#Select the feature selection
operator in the tree view. Change the search directory from forward
(forward selection) to backward (backward elimination). Restart the
experiment. All features will be selected.#ylt#/li#ygt#
#ylt#li#ygt#Select the feature selection operator. Right click to open
the context menu and repace the operator by another feature selection
scheme (for example a genetic algorithm).#ylt#/li#ygt# #ylt#li#ygt#Have
a look at the list of the experiment log operator. Every time it is
applied it collects the specified data. Please refer to the RapidMiner
Tutorial for further explanations. After changing the feature selection
operator to the genetic algorithm approach, you have to specify the
correct values.
#ylt#table#ygt##ylt#tr#ygt##ylt#td#ygt##ylt#icon#ygt#groups/24/visualization#ylt#/icon#ygt##ylt#/td#ygt##ylt#td#ygt##ylt#i#ygt#Use
the experiment log operator to log values
online.#ylt#/i#ygt##ylt#/td#ygt##ylt#/tr#ygt##ylt#/table#ygt#
#ylt#/li#ygt# #ylt#/ul#ygt# #ylt#/p#ygt#"/> <operator name="Input"
class="ExampleSource"> <parameter key="attributes"
value="../data/polynomial.aml"/> </operator> <operator name="FS"
class="FeatureSelection" expanded="yes"> <operator name="XValidation"
class="XValidation" expanded="yes"> <parameter
key="sampling_type" value="shuffled sampling"/> <operator
name="GridParameterOptimization" class="GridParameterOptimization"
expanded="yes"> <operator name="XValidation (2)" class="XValidation"
expanded="yes"> <operator name="NearestNeighbors"
class="NearestNeighbors"> </operator> <operator name="OperatorChain"
class="OperatorChain" expanded="yes"> <operator name="ModelApplier"
class="ModelApplier"> </operator> <operator
name="ClassificationPerformance" class="ClassificationPerformance">
</operator> </operator> </operator> </operator> <operator
name="ApplierChain" class="OperatorChain" expanded="yes"> <operator
name="Applier" class="ModelApplier"> <list
key="application_parameters"> </list> </operator> <operator
name="Performance" class="Performance"> </operator> </operator>
</operator> <operator name="ExpLog" class="ProcessLog"> <list
key="log"> <parameter key="generation"
value="operator.FS.value.generation"/> <parameter
key="performance" value="operator.FS.value.performance"/>
</list> </operator> </operator> </operator>
expects a model but gets a ParameterSet and PerformanceVector.
I have no idea how I can return the best model found by
GridParameterOptimization and pass it to the next operator
ApplierChain. Could you please help me extending my model?
Thank you.
Regards,
Martin
0 -
Hi Martin,
you simply have to put another learner into your process after the [tt]GridParameterOptiomization[/tt] operator. For that learner you should set the optimized parameters via a [tt]ParameterSetter[/tt]. The following process gives you an idea, how to do that:
Hope that helps,
<operator name="Root" class="Process" expanded="yes">
<operator name="Input" class="ExampleSource">
<parameter key="attributes" value="../data/polynomial.aml"/>
</operator>
<operator name="FS" class="FeatureSelection" expanded="yes">
<operator name="XValidation" class="XValidation" expanded="yes">
<parameter key="sampling_type" value="shuffled sampling"/>
<operator name="IOMultiplier" class="IOMultiplier">
<parameter key="io_object" value="ExampleSet"/>
</operator>
<operator name="GridParameterOptimization" class="GridParameterOptimization" expanded="yes">
<operator name="XValidation (2)" class="XValidation" expanded="yes">
<operator name="NearestNeighbors" class="NearestNeighbors">
</operator>
<operator name="OperatorChain" class="OperatorChain" expanded="yes">
<operator name="ModelApplier" class="ModelApplier">
</operator>
<operator name="ClassificationPerformance" class="ClassificationPerformance">
</operator>
</operator>
</operator>
</operator>
<operator name="ParameterSetter" class="ParameterSetter">
<list key="name_map">
<parameter key="knn" value="knn"/>
</list>
</operator>
<operator name="knn" class="NearestNeighbors">
</operator>
<operator name="ApplierChain" class="OperatorChain" expanded="yes">
<operator name="Applier" class="ModelApplier">
<list key="application_parameters">
</list>
</operator>
<operator name="Performance" class="Performance">
</operator>
</operator>
</operator>
<operator name="ExpLog" class="ProcessLog">
<list key="log">
<parameter key="generation" value="operator.FS.value.generation"/>
<parameter key="performance" value="operator.FS.value.performance"/>
</list>
</operator>
</operator>
</operator>
Tobias0 -
Hi Tobias,
yes, that was exactly what I was looking for. Thank you.
Regards,
Martin0
