"Association Rules with Thousands of Products"

a_isabel_marque

Hi,

I am trying to create association rules.
My dataset have 30496 transactions and 18270 products and it is stored in a MySQL database like this:

transaction | product
1 | 0987
1 | 6543
2 | 1234
... | ...

Before use FP-Growth operator I convert data with Nominal to Numerical operator and aggregate the rows of the same transaction. After that I convert data again with Numerical to Binomial operator.

I run the process with RapidAnalytics but it returns the error "java.lang.OutOfMemoryError: GC overhead limit exceeded ". And I used Free Memory operator.

How can I solve this problem? It is possible to create association rules with this number of products?
Other question, it is possible to convert data to a sparse matrix?

I am studing other approach in other database. I installed in a PostgreSQL database the MADlib library. And with the same dataset, MADlib function create the association rules in 30 seconds without any transformation of the data!!!

Best regards.

veve

That is funny, I have somehow the same problem, but nobody bothered to answer..