Parallel Processing - Multiple server instances

Nikouy

Dear Community,

I am currently working on my MSc research thesis, which is based on Rapidminer  and cloud deployment models. Therefore, I will probably be bugging you a bit as I research and questions arise .

One of the questions I intend to answer is: Can Rapidminer support parallel processing or have multiple instances (different servers deployed) mining collaboratively the same data set?

I would appreciate if you can help me answer these questions and point me in the right direction to explore various options. While the Rapidminer platform may not directly support this set up, some work arounds or custom configuracions or data mining strategies might.

Any help or feedback is very welcome.

Thanks in advance,

Nicolas

BalazsBaranyRM

Hi Nicolas,

Studio and Server share the same execution engine. Therefore, parallel processes in Studio also work in parallel on Server.

Sharing data mining algorithms between multiple independent systems is an active research topic. For most machine learning algorithms, it's not worth the effort, as network communication latencies would make the entire process much more inefficient than executing a smaller number of processes on one system.

There are exceptions like neural network building using TensorFlow with the Deep Learning extension. This can use multiple systems with or without GPUs to work one problem.

There are also related operations that are very easy to parallelize. E. g. you can score large datasets in parallel with existing models on multiple servers. You can use Radoop with a Hadoop cluster to parallelize preprocessing, filtering, etc. and build a few model types.

Regards,

Balázs