"[SOLVED] Aggregation: generating collections grouped by attrbutes"

Ugo

Hello,

I have a data set with three attributes: first, second and rank.
I have sorted the data-set so that it is ordered first by "first" and then rank.
I need to:
1. generate subsets of the data grouped by "first"
2. for each group data-set select the first k elements

In the end I would like to select any group data-set via the "first" attribute.
 
Anyone know if this is possible?
If so how?

TIA,
Hugo F

P.S: I have looked at Aggregation and Collections to no avail

MariusHelf

Hi Hugo,

try to use the Loop Values operator to loop over the values of "first". In the inner process, use "Filter Examples" such that you have only examples with the current value of first (you need the attribute_value_filter with an expression like "first=%{loop_value}". Then Sort by rank and use Filter Example Range to keep only the first k elements. Outside of the loop, you can use Append to recombine the results of the loop operator.

Best regards,
Marius

Ugo

Hi Marius,

Thanks once again for the input. Worked as expected. 
Took some time though (36.34 min) because I am dealing with
roughly 8.36 m records.

Which brings me to my next question. Yesterday I considered
using a DB which I expect is useful due to the indexing. So
my question is: do these "Filter" operators also use indexes or
would I be better off using SQL for the "filtering"?

TIA,
Hugo F.

MariusHelf

You should do it directly in the database. Once you imported the data into RapidMiner it's just a normal example set without any indices.

Ugo

Ok. Pretty quick for 8M records though.

Thanks.
Hugo F.