Python script running slowly in RapidMiner

Bingle

Hi There,

I developed a python script to transfer about 80K raw system logs. It works in both RapidMiner and my local python IDE. But it took more than 4 hours to finish the job in RapidMiner, and it only took around 2 mins in local Python IDE. 

Is there anyway I can improve the efficiency of RapidMiner? 

Thanks

MichaelKnopf

Hi @Bingle ,

Are you by any chance running Execute Python once for each of the 80k log files? Since the operator fires up a new Python instance and sets up the communication between RapidMiner and Python from scratch every time the operator is executed, this could easily explain the long runtime.

If this is the case, I suggest you change your code to loop over the log files in Python instead of RapidMiner.

If you are already doing this, more details on the process and script would indeed be helpful for troubleshooting.

Thanks,

Michael

YYH

Hi @Bingle,

Thanks for sharing your experience. To improve the performance of python integration, could you share your process with the python scripts? Our python experts @mschmitz, @pschlunder, @Michael would help investigate into details.

Thanks!

YY

MichaelKnopf

Hi @Bingle ,

Are you by any chance running Execute Python once for each of the 80k log files? Since the operator fires up a new Python instance and sets up the communication between RapidMiner and Python from scratch every time the operator is executed, this could easily explain the long runtime.

If this is the case, I suggest you change your code to loop over the log files in Python instead of RapidMiner.

If you are already doing this, more details on the process and script would indeed be helpful for troubleshooting.

Thanks,

Michael

Bingle

Hi @yyhuang

I guess you are right, will try later.

Thanks and best regards,
Bingle