processing across batches

Question

I am wondering what is the easiest way of doing a bunch of data ETL preprocessing on a large Exampleset in batches and keeping everything in RapidMiner.
There is an operator called Loop Batches, which in theory looks ideal, except it doesn't return the results from the batches and also doesn't include any batch macro to allow storage of intermediate results!  It looks like it is only suitable for use with databases, so it actually has very limited value.  Perhaps that operator could be expanded in the future to be more useful.
In the meantime, if you had 10k total records and wanted to divide them into 10 batches of 1000 records each, do a bunch of preprocessing, and then unify the resulting data at the end, what is the most efficient way to do this in RapidMiner?
Right now it looks like creating a batch attribute manually and then using the generic Loop operator, using Store on each intermediate batch/iteration result, and then using Append at the end.  But there might be some other creative way of using loops and macros that I am missing?

MartinLiebig · Accepted Answer

Hi @Telcontar120 ,
usually I use the Group Into Collection operator for it. You can simply create a collection with one example set per batch. Than you can use Loop Collection to loop over it and do what ever transformation on it.
The Group Collection + Loop Collection combo allows you to have somewhat a GROUP BY statement for any subprocess.
Its worth of note, that this duplicates the data once, since the items in the collection are a copy of the original data.

Loop Collection is also a loop operator which does 
not run in parallel. To overcome this I have implemented the option to 
extract the collection size in Extract Macro Format. This way you can 
then use the normal loop operator like this:
And inside the loop you then select the i-th element:

Attached is the example as a full work flow.
Cheers,Martin

MartinLiebig · Accepted Answer

Hi @Telcontar120 ,
you are right, you need some kind of batch attribute. But very frequently it is:id%10

which gives you numbers from 1-10 orround(id/max_id*10)

which gives you increasing numbers from 1-10.
I agree that this is an extra step and one could enhance the Loop Batches operator to do in a more handy manner.

Best,Martin