"taskScheduler loops in infinity on server with Loop Remote Files"
mugicagonzalez_
New Altair Community Member
Hello all!
I am using the "Loop Remote Files" from the Jackhammer extension. But irregularly it happens that the process runs into infinity (and doesn't time-out) on the server but doesn't loop the remote files anymore.
My gut feeling says it's due to an overloaded FTP server, but I wanted to check here first before I start moving to another FTP server.
Any hint on how I can investigate the real problem?
My server log file produces this in bulk:
I am using the "Loop Remote Files" from the Jackhammer extension. But irregularly it happens that the process runs into infinity (and doesn't time-out) on the server but doesn't loop the remote files anymore.
My gut feeling says it's due to an overloaded FTP server, but I wanted to check here first before I start moving to another FTP server.
Any hint on how I can investigate the real problem?
My server log file produces this in bulk:
<div>2019-03-06 08:41:28.510 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:41:28.510 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:41:32.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:37.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:42.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:47.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:52.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:57.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:02.081 INFO 72445 --- [taskScheduler-4] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:07.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:12.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:17.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:22.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:27.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:42:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:42:32.082 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:37.082 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:42.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:47.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:52.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:57.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:02.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:07.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:12.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:17.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:22.084 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:27.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:43:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:43:32.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:37.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:42.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:47.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:52.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:57.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:02.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:07.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:12.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:17.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:22.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:27.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:44:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:44:32.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:37.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:42.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:47.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:52.088 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:57.088 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:02.088 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:07.088 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:12.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:17.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:22.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:27.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:45:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:45:32.089 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:37.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:42.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:47.090 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:52.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:57.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:02.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:07.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:12.091 INFO 72445 --- [taskScheduler-2] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:17.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:22.092 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:27.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:46:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:46:32.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:37.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:42.092 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:47.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:52.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:57.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:02.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:07.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:12.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:17.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:22.094 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:27.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:47:28.512 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:47:32.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:37.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:42.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:47.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:52.095 INFO 72445 --- [taskScheduler-2] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB ....</div>
Tagged:
0
Best Answer
-
Hi,
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz1
Answers
-
Hello,
I am part of the Old World Computing team (as a Student) and I would like to try and reproduce said bahavior to locate the issue and hopefully fix it.
But for that I am in need for additional Information, since the cause of the issue could be anything:- Which version of the Jackhammer extension are you using?
(Rapidminer menu: Extensions -> About installed Extensions-> About Jackhammer Extension) - Which Rapidminer server version are you using?
- Do you know if your FTP server has a timeout defined, if yes what is the defined value?
- Do you know the name of your FTP server ?
- You said the issue occurs irregularly, but if the issue occurs, how do you get your process to work again? Do you just restart your server, or is it sufficient to restart the process?
It would be a big help if you could provide answers to as many of the above questions as you can, so I can investigate this issue in a quick manner.
kind regards
Burhan Yildiz
3 - Which version of the Jackhammer extension are you using?
-
good to have you here @Burhan_Yildiz!1
-
Thanks @Burhan_Yildiz for reaching out. Hope this helps:
- Which version of the Jackhammer extension are you using?
(Rapidminer menu: Extensions -> About installed Extensions-> About Jackhammer Extension)
2.1.784 - Which Rapidminer server version are you using?
9.1 - Do you know if your FTP server has a timeout defined, if yes what is the defined value?
No, and no access to config. But could experiment with another server though.
- Do you know the name of your FTP server ?
46.30.211.164:21 - You said the issue occurs irregularly, but if the issue occurs, how do you get your process to work again? Do you just restart your server, or is it sufficient to restart the process?
I manually stop the process and re-run it without any issue (often a couple of hours later, because I check the server executions once a day)
0 - Which version of the Jackhammer extension are you using?
-
Hi,
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz1
Altair Employee