Standardization as a preprocessor

Legacy User

In addition to normalization as a preprocessing operator, a standardization preprocessing operator would be very handy.

TobiasMalbrecht

Hi,

what exactly do you mean with standardization?

Regards,
Tobias

Legacy User

Normalize -
A collection of numeric data is normalized by subtracting the minimum value from all values and dividing by the range of the data. This yields data with a similarly shaped histogram but with all values between 0 and 1.

Standardize -
A collection of numeric data is standardized by subtracting a measure of central location (such as the mean or median) and by dividing by some measure of spread (such as the standard deviation, interquartile range or range). This yields data with a similarly shaped histogram with values centered around 0.

-- It would be useful to be able to specify with a dropdown the the measure of the spread whether it is standard deviation, interquartile range, 5-95% etc, does this make sense? Also the selection of what you call doing a z-transform looks to be mutally exclusive with scaling, there is not a reason for this that I can tell and thus requires two operations.

TobiasMalbrecht

Hi,

ok, I got the point. These two tasks are indeed both contained in the [tt]Normalization[/tt] operator where the normalization task is executed if the [tt]z-transform[/tt] parameter is disabled. If it is enabled, the mean is substracted from all values and the difference is divided by the standard deviation of the values. We will think about extending the operator to support other scaling strategies as well for the next release.

Regards,
Tobias

Stefan_E

Tobias,

related I'm looking for a method to just subtract the mean without then dividing by the standard deviation.

This is eg. useful for NearestNeighbor learning, in case of measurement offsets: You don't want a z-Normalization there as that would effectively change influence of some parameters to the classification.

Regards,
Stefan

IngoRM

Hi,

related I'm looking for a method to just subtract the mean without then dividing by the standard deviation.

That's possible with the latest CVS version (and the following nice process):

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="number_examples"	value="200"/>
        <parameter key="target_function"	value="sum classification"/>
    </operator>
    <operator name="FeatureNameFilter" class="FeatureNameFilter">
        <parameter key="filter_special_features"	value="true"/>
        <parameter key="skip_features_with_name"	value="label"/>
    </operator>
    <operator name="ExampleSetTranspose" class="ExampleSetTranspose">
    </operator>
    <operator name="AttributeAggregation" class="AttributeAggregation">
        <parameter key="aggregation_attributes"	value="att_.*"/>
        <parameter key="aggregation_function"	value="average"/>
        <parameter key="attribute_name"	value="mean"/>
    </operator>
    <operator name="FeatureIterator" class="FeatureIterator" expanded="yes">
        <parameter key="filter"	value="att_.*"/>
        <parameter key="work_on_input"	value="false"/>
        <operator name="AttributeConstruction" class="AttributeConstruction">
            <list key="function_descriptions">
              <parameter key="mean_%{loop_feature}"	value="%{loop_feature}-mean"/>
            </list>
        </operator>
    </operator>
    <operator name="AttributeFilter" class="AttributeFilter">
        <parameter key="condition_class"	value="attribute_name_filter"/>
        <parameter key="parameter_string"	value="mean_.*"/>
    </operator>
    <operator name="ExampleSetTranspose (2)" class="ExampleSetTranspose">
    </operator>
    <operator name="AttributeFilter (2)" class="AttributeFilter">
        <parameter key="condition_class"	value="attribute_name_filter"/>
        <parameter key="invert_filter"	value="true"/>
        <parameter key="parameter_string"	value="id"/>
    </operator>
</operator>

Cheers,
Ingo