"Customized X-fold cross-validation"

_paul_

Hi,

I want to perform an X-fold cross-validation which however does not
operate on sets that are defined by RapidMiner's XValidation "sampling_type"
parameter but on sets which are constructed using a "marker" in the
examples provided by an ExampleSource operator.

To be more accurate, my input examples (pairs of feature vectors and
labels) used for classification contain an attribute that defines the
application this particular example was extracted from. Let's say the
examples come from three applications "A", "B", and "C" and each
example contains an attribute holding one of the three characters.

Based on this, I would like to perform a 3-fold cross-validation where in
a first run, examples from "A" are excluded and tested on examples from
"B" and "C" ...

Is there an operator for that in RapidMiner?

Regards,
Paul

land

Hi Paul,
there is a special variant of the XValidation called BatchXValidation, where it uses an attribute with the special role batch to define the splitting sets. I post a process below, making use of this operator.

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="target_function"	value="sum classification"/>
    </operator>
    <operator name="AttributeSubsetPreprocessing" class="AttributeSubsetPreprocessing" expanded="yes">
        <parameter key="condition_class"	value="attribute_name_filter"/>
        <parameter key="attribute_name_regex"	value="att1"/>
        <operator name="BinDiscretization" class="BinDiscretization">
            <parameter key="number_of_bins"	value="3"/>
            <parameter key="range_name_type"	value="short"/>
        </operator>
    </operator>
    <operator name="ChangeAttributeRole" class="ChangeAttributeRole">
        <parameter key="name"	value="att1"/>
        <parameter key="target_role"	value="batch"/>
    </operator>
    <operator name="BatchXValidation" class="BatchXValidation" expanded="yes">
        <operator name="DecisionTree (2)" class="DecisionTree" breakpoints="before,after">
        </operator>
        <operator name="TestChain (2)" class="OperatorChain" expanded="no">
            <operator name="ModelApplier (2)" class="ModelApplier">
                <list key="application_parameters">
                </list>
            </operator>
            <operator name="Performance (2)" class="Performance">
            </operator>
        </operator>
    </operator>
</operator>

Greetings,
  Sebastian