Aggregate Duplicates

Hello

The aggregate operator is your friend - here's an example

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.011">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.011" expanded="true" name="Process">
    <process expanded="true" height="206" width="413">
      <operator activated="true" class="generate_data" compatibility="5.1.011" expanded="true" height="60" name="Generate Data" width="90" x="112" y="75">
        <parameter key="target_function" value="three ring clusters"/>
        <parameter key="number_of_attributes" value="2"/>
      </operator>
      <operator activated="true" class="aggregate" compatibility="5.1.011" expanded="true" height="76" name="Aggregate" width="90" x="313" y="75">
        <list key="aggregation_attributes">
          <parameter key="label" value="count"/>
        </list>
        <parameter key="group_by_attributes" value="|label"/>
      </operator>
      <connect from_op="Generate Data" from_port="output" to_op="Aggregate" to_port="example set input"/>
      <connect from_op="Aggregate" from_port="example set output" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

regards

Andrew

Thank you for your reply.
If I understand correctly, you suggest aggregating duplicates using the aggregate operator and "group by" all attributes.
How can this be utilized to make a k-NN faster?
Having 20 million samples with 20 attributes but only 1 million possible attribute combinations will result in a dataset of 1 million examples with 21 attributes.
How will k-NN work on that (ie use the 21st attribute as weight/count or something).

Hello

I think k-NN would still work, the new aggregation attribute would need to be carefully selected in order to ensure that unseen data is near to representative examples.

As always, an experiment is needed.

regards

Andrew