OutlierDistanceBasedDetection

alexman

Hi,

I have an exampleSet with different attributes and I would like to apply Outlier in every numerical attribute but separately is this possible? which is the best way to do ?  I can apply outlier over the whole table but i would like to do it in every attribute, for example:

heigth weigth .....  // more attributtes
188      80
185      150
186      83
189      89
190      87
192      86
145      88

I would like to get 145 (heigth) and 150 (weigth) separately ... [Probably a process for each attribute applying DBoutlierOperator would be a solution but not efficient...]
DBOutlierOperator(OperatorDescription description)  is not applyable for an attribute of an exampleSet. Probably AttributeSelectionExampleSet which filters what attributes I want in exampleset would be useful but how to apply the Outlier function for each attribute?

thanks

land

Hi,
you could use a combination of the feature iterator and the Attribute Subset Preprocessing, which will deliver only a subset of the exampleset's attributes to its child operators. Since this is a complex process, I will post a sample below:

<operator name="Root" class="Process" expanded="yes">
    <operator name="ExampleSetGenerator" class="ExampleSetGenerator">
        <parameter key="target_function" value="sum classification"/>
    </operator>
    <operator name="IOStorer" class="IOStorer">
        <parameter key="name" value="Store"/>
        <parameter key="io_object" value="ExampleSet"/>
        <parameter key="remove_from_process" value="false"/>
    </operator>
    <operator name="FeatureIterator" class="FeatureIterator" expanded="yes">
        <parameter key="work_on_input" value="false"/>
        <operator name="IORetriever" class="IORetriever">
            <parameter key="name" value="Store"/>
            <parameter key="io_object" value="ExampleSet"/>
        </operator>
        <operator name="AttributeSubsetPreprocessing" class="AttributeSubsetPreprocessing" expanded="yes">
            <parameter key="condition_class" value="attribute_name_filter"/>
            <parameter key="parameter_string" value="att5"/>
            <parameter key="attribute_name_regex" value="%{loop_feature}"/>
            <operator name="DetectionOnSingleAttribute" class="DensityBasedOutlierDetection">
                <parameter key="distance" value="1.0"/>
                <parameter key="proportion" value="0.5"/>
            </operator>
            <operator name="DoingSomething" class="OperatorChain" expanded="yes">
                <operator name="ChangeAttributeRole" class="ChangeAttributeRole">
                    <parameter key="name" value="Outlier"/>
                </operator>
                <operator name="ChangeAttributeName" class="ChangeAttributeName">
                    <parameter key="old_name" value="Outlier"/>
                    <parameter key="new_name" value="Outlier_%{loop_feature}"/>
                </operator>
            </operator>
        </operator>
        <operator name="IOStorer (2)" class="IOStorer">
            <parameter key="name" value="Store"/>
            <parameter key="io_object" value="ExampleSet"/>
            <parameter key="remove_from_process" value="false"/>
        </operator>
    </operator>
    <operator name="IOConsumer" class="IOConsumer">
        <parameter key="io_object" value="ExampleSet"/>
    </operator>
    <operator name="IORetriever (2)" class="IORetriever">
        <parameter key="name" value="Store"/>
        <parameter key="io_object" value="ExampleSet"/>
    </operator>
</operator>


The problem is the behavior of the FeatureIterator, which will not deliver the changed exampleset after finishing. That's why we have to use the IOStore and IORetrieve operators to save the generated ExampleSet on our own. We actually only need the macro defined by the FeatureIterator giving us every regular attribute name, so that we can use it in the attributeSubsetPreprocessing condition.

This sample only renames the attributes, but you very well might do something more intelligent like unification of the results of each attribute using an attributeConstruction, or something else.

Greetings,
  Sebastian