AssociationRuleGenerator out of memory

Legacy User

I'm using RapidMiner for extracting association rules from a table with 20 attributes and 66 tuples, stored in Microsoft SQL Server. I need to extract itemsets with a length of at least 6 items, with minimum support 0.03, and I need all the rules with minimum confidence 0.2. FPGrowth finds about 500000 frequent itemsets, but unfortunately the Association Rules Generator goes out of memory. It works only if I limit the maximum size of the itemset to 3, but it's too few for me. I used RapidMiner giving it a memory of 1.4 GB (my system has 3 GB available, but it doesn't let me create a JVM with more than 1500 MB in the -Xmx option).

More in detail, my process tree contains the following blocks: DatabaseExampleSource, Preprocessing (Nominal2Binominal), FPGrowth, AssociationRuleGenerator.

Does someone know if there is a way to make my process work without going out of memory?

Thanks in advance.

IngoRM

Hello,

with those settings the number of rules is probably going to be really huge and hence there will be probably always be some memory problems. This will lead to somewhat like 500000 over 6 rules which is probably even too large to even display the set of rules in a table (without pagination, table views (not RapidMiner!) are restricted to about 150 million rules).

You can use more than 1.4Gb on a 64 Bit machine (we ourself use a 16Gb machine here ourself for larger analysis projects) but you would have to use a 64 bit OS and the 64 bit version of RapidMiner. See here for more informaiton:

http://rapid-i.com/rapidforum/index.php/topic,547.msg2756.html#msg2756


But as I said before I have no idea if even this would be enough for that amount of rules.

Cheers,
Ingo