How to use all Processor Cores in the RapidMiner Models

Marcello_Sandi

Hello staff,

I attract attention to the fact of the RapidMiner use all Processor Cores only with Meta.Parameter.ParameterIterationParallel and Meta.Parameter.EvolutionaryParameterOptimizationParallel.

Considering that I'm a beginner RapidMiner user so....

How could I use all quad cores of my processor over the models without these components? Is it possible?

Is there a jvm parameter that I can use to this problem?

I appreciate all kinds of help and comments!

Marcello Sandi

wessel

Does the latest jvm already support quad cores?

Marcello_Sandi

Well,

I found this http://documentation.skillsoft.com/ikb/23851.htm

this http://forums.sun.com/thread.jspa?forumID=31&;threadID=5347654

and this http://forums.sun.com/thread.jspa?forumID=7&;threadID=5255292

Of course, RapidMiner it's a complex system, and need to consider this factor.

I developed an application that use java multi-thread.....and run so fast. Just one example of a method is below. It's based on an consumer and producer system:

   private Thread factoryThread(ThreadGroup tg, final Class<Consumidor> consumidorClass){
Thread th =null;
try {
th = new Thread(tg, new Runnable() {
Consumidor consumidor =  consumidorClass.newInstance();
public void run() {
consumidor.startup();
Object obj = null;
while ((obj = next()) != null && continua) {
consumidor.execute(obj);

}
consumidor.finish();
}
});
} catch (InstantiationException e) {
log.severe(e.getMessage());
} catch (IllegalAccessException e) {
log.severe(e.getMessage());
    } catch(Exception e){
log.severe(e.getMessage());
    }
return th;
   
   }

land

Hi Marcello,
in usuall data mining applications, the most time consuming part is the training and not the application. 300.000 examples with 40.000 sparse attributes are very well applied in around an hour on one core.
The most computationally expensive part is finding the best parameters for your learning algorithm. Since you need to estimate the performance for this purpose, you will have to use a cross validation. Hence there are two major points, where you can make use of your multicore cpu: Parallelizing the cross-validation and parallelizing the parameter optimization. That's exactly what the enterprise edition provides, in order to speed up the manual training process.
If you have really much data to classify in your domain, probably a few million examples with hundreds of non sparse attributes per day, then you will indeed need a solution for parallel applying. This can be achieved by starting rapid miner a few times (as much as you have cores) and use the non parallel model applier. Another solution would be to misuse the parallel cross-validation, because the model is applied there parallely. But the most elegant way would be, that you use your status as enterprise customer and contact us, so that we will find a way to provide you with that operator.

Kind regards,
  Sebastian Land

Marcello_Sandi

Sebastian,

You are right. But any way I want to make clear that, I didn't want to damage the reputation and the prestige that this software and all team have.

I am a great admirer of this tool, and for this reason, I wanted to buy it. And I bought with great difficulty. Not just to receive support and so on. I want to understand and to know RapidMiner in depth.

My last post was just try to understand the some limitations, and you explained me very well. I am very grateful.

About of the starting rapid miner a few times (as much as I has cores), I had already noticed it, because I stay monitoring it using top command in Linux. So, it is truth. 

Frankly, I would like to have the opportunity to meet all team, and make all courses that this company has. But...I can't.

So, thanks for your attention and patience.
Marcello Sandi