Altair RISE
A program to recognize and reward our most engaged community members
Nominate Yourself Now!
Home
Discussions
Community Q&A
Python script running slowly in RapidMiner
Bingle
Hi There,
I developed a python script to transfer about 80K raw system logs. It works in both RapidMiner and my local python IDE. But it took more than 4 hours to finish the job in RapidMiner, and it only took around 2 mins in local Python IDE.
Is there anyway I can improve the efficiency of RapidMiner?
Thanks
Find more posts tagged with
AI Studio
Python
Execute Script
Accepted answers
MichaelKnopf
Hi
@Bingle
,
Are you by any chance running
Execute Python
once for each of the 80k log files? Since the operator fires up a new Python instance and sets up the communication between RapidMiner and Python from scratch every time the operator is executed, this could easily explain the long runtime.
If this is the case, I suggest you change your code to loop over the log files in Python instead of RapidMiner.
If you are already doing this, more details on the process and script would indeed be helpful for troubleshooting.
Thanks,
Michael
All comments
YYH
Hi
@Bingle
,
Thanks for sharing your experience. To improve the performance of python integration, could you share your process with the python scripts? Our python experts
@mschmitz
,
@pschlunder
,
@Michael
would help investigate into details.
Thanks!
YY
MichaelKnopf
Hi
@Bingle
,
Are you by any chance running
Execute Python
once for each of the 80k log files? Since the operator fires up a new Python instance and sets up the communication between RapidMiner and Python from scratch every time the operator is executed, this could easily explain the long runtime.
If this is the case, I suggest you change your code to loop over the log files in Python instead of RapidMiner.
If you are already doing this, more details on the process and script would indeed be helpful for troubleshooting.
Thanks,
Michael
Bingle
Hi
@yyhuang
I guess you are right, will try later.
Thanks and best regards,
Bingle
Quick Links
All Categories
Recent Discussions
Activity
Unanswered
日本語 (Japanese)
한국어(Korean)
Groups