"Association Rules hangs machine"

mimesis

Hi,
I am working on a text analysis project of news articles grabbed from the internet. I have been successful in extracting the content, preprocessing (tokenizing, filtering, transforming case etc) and generating a binary term occurrence exampleset. I followed Neil M's video and used Process documents from files, Numerical to Binomial and FP Growth. Using breakpoints, all these stages appear to be fine and run quickly. When I attempt to create Association rules, the machine runs and seems to hang, regardless of the memory allocated when I start java (I use the command line and for instance, assigned as follows:

java -Xmx12000m -jar rapidminer.jar. Right now, the program seems locked out (still calculating the rules at 13 minutes) and one CPU is at 100%. My dataset only has five documents in it and the rest of the process ran in 3 seconds or less on an 8 core Mac Pro. I noticed that the memory usage  would increment and max out, regardless of how much I allocated (I have tried various lower and upper limits and read on a Weka site that one should not set both -Xms and -Xmx at the same time).

Any ideas?

William

haddock

Hi William,

Sadly it looks like the code is, shall we say, 'bold'....

// generating rule by splitting set in every two parts for head and body of rule 
			if (set.getItems().size() > 1) {
				PowerSet<Item> powerSet = new PowerSet<Item>(set.getItems());

Which would mean quite a wait for longer frequent item sets, as there are 2^N subsets of an N length frequent item set. So a 50 word itemset would need the space for just 1,125,899,906,842,624 sets. Bold indeed.