"Association Rules with Thousands of Products"

a_isabel_marque
a_isabel_marque New Altair Community Member
edited November 5 in Community Q&A
Hi,

I am trying to create association rules.
My dataset have 30496 transactions and 18270 products and it is stored in a MySQL database like this:

transaction | product
1 | 0987
1 | 6543
2 | 1234
... | ...

Before use FP-Growth operator I convert data with Nominal to Numerical operator and aggregate the rows of the same transaction. After that I convert data again with Numerical to Binomial operator.

I run the process with RapidAnalytics but it returns the error "java.lang.OutOfMemoryError: GC overhead limit exceeded ". And I used Free Memory operator.

How can I solve this problem? It is possible to create association rules with this number of products?
Other question, it is possible to convert data to a sparse matrix?

I am studing other approach in other database. I installed in a PostgreSQL database the MADlib library. And with the same dataset, MADlib function create the association rules in 30 seconds without any transformation of the data!!!

Best regards.
Tagged:

Answers

  • veve
    veve New Altair Community Member
    That is funny, I have somehow the same problem, but nobody bothered to answer..

Categories