[SOLVED] Performance RAM Rapid Miner
Hi,
at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:
criterion: accurancy
minimal size for split: 4
minimal leaf size: 2
minimal gain : 0,1
maximal depth 10
confidence 0,25
Cross Validation 5
sampling type : stratifield sampling
Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?
Best Regards and thank you
Marco
at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:
criterion: accurancy
minimal size for split: 4
minimal leaf size: 2
minimal gain : 0,1
maximal depth 10
confidence 0,25
Cross Validation 5
sampling type : stratifield sampling
Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?
Best Regards and thank you
Marco
Find more posts tagged with
Sort by:
1 - 3 of
31
Thanks a lot for the replay.
It's funny I am doing at the moment the SVM with optimization ;-)
I am eager to use the new release.
Best Regards
Marco
It's funny I am doing at the moment the SVM with optimization ;-)
I am eager to use the new release.
Best Regards
Marco
For SVM optimization and understanding you may be interested in this thread: http://rapid-i.com/rapidforum/index.php/topic,6194.0.html
I'll mark this topic as solved now.
Best regards,
Marius
I'll mark this topic as solved now.
Best regards,
Marius
the decision tree is one of the worst choices for data with many attributes. (It is perfectly fit to process data with a huge amount of examples though, with only few attributes).
For data with many attributes, but only a few examples as in your case, try a Linear SVM. Please note that the parameter C of the SVM must be optimized. You can use the Optimize Parameters (Grid) for that. For a choice of the search range for C, please have a look at this thread. Please let me know if you have any problems.
Concerning the usage of RAM, we are planning to make the maximum amount of memory available to RapidMiner configurable with the next release, which is planned to be available to the public on next monday.
Best regards,
Marius