"GridParameterOptimization and performances"

Username

Hi,

how can I use the GridParameterOptimizationOperator to save the PerformanceVectors after each iteration? The result should be the currently selected parameters and the resulting performances, e.g.:

k  Sim  | Accuracy
1  cos    0.3
2  cos    0.5
3  cos    0.4
1  Euc    0.2
2  Euc    0.4
3  Euc    0.3

P.S.: Didn't older RM versions have more similarity measures for the kNN learner?


Thanks

IngoRM

Hello,

you can use the ProcessLog operator for this (please refer to the sample process "...Meta/01_ParameterOptimization.xml"). Here is the basic setup:

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="number_examples"	value="200"/>
        <parameter key="number_of_attributes"	value="2"/>
        <parameter key="target_function"	value="checkerboard classification"/>
    </operator>
    <operator name="ParameterOptimization" class="GridParameterOptimization" expanded="yes">
        <list key="parameters">
          <parameter key="NearestNeighbors.k"	value="[1.0;3.0;2;linear]"/>
        </list>
        <operator name="Validation" class="XValidation" expanded="yes">
            <parameter key="sampling_type"	value="shuffled sampling"/>
            <operator name="NearestNeighbors" class="NearestNeighbors">
                <parameter key="k"	value="3"/>
            </operator>
            <operator name="ApplierChain" class="OperatorChain" expanded="yes">
                <operator name="Test" class="ModelApplier">
                    <list key="application_parameters">
                    </list>
                </operator>
                <operator name="ClassificationPerformance" class="ClassificationPerformance">
                    <parameter key="accuracy"	value="true"/>
                    <list key="class_weights">
                    </list>
                </operator>
            </operator>
        </operator>
        <operator name="Log" class="ProcessLog">
            <parameter key="filename"	value="paraopt.log"/>
            <list key="log">
              <parameter key="k"	value="operator.NearestNeighbors.parameter.k"/>
              <parameter key="Accuracy"	value="operator.Validation.value.performance"/>
            </list>
        </operator>
    </operator>
</operator>

You can add a filename for the ProcessLog operator into which the results should be written. The results are of course also available in the Log tab of the Results View.

P.S.: Didn't older RM versions have more similarity measures for the kNN learner?

Yes, and they will again be integrated. We just completely revised the KNN learner which led to a speed-up of factor 10. However, we had to remove the usual similarity measure in order to get this speed-up and did not manage to re-implement all measures for KNN yet. But (at least most of them) will be available again in future releases.

Cheers,
Ingo