"taskScheduler loops in infinity on server with Loop Remote Files"
New Altair Community Member
Updated by Jocelyn
Hello all!
I am using the "Loop Remote Files" from the Jackhammer extension. But irregularly it happens that the process runs into infinity (and doesn't time-out) on the server but doesn't loop the remote files anymore.
My gut feeling says it's due to an overloaded FTP server, but I wanted to check here first before I start moving to another FTP server.
Any hint on how I can investigate the real problem?
My server log file produces this in bulk:
I am using the "Loop Remote Files" from the Jackhammer extension. But irregularly it happens that the process runs into infinity (and doesn't time-out) on the server but doesn't loop the remote files anymore.
My gut feeling says it's due to an overloaded FTP server, but I wanted to check here first before I start moving to another FTP server.
Any hint on how I can investigate the real problem?
My server log file produces this in bulk:
<div>2019-03-06 08:41:28.510 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:41:28.510 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:41:32.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:37.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:42.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:47.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:52.080 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:41:57.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:02.081 INFO 72445 --- [taskScheduler-4] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:07.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:12.081 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:17.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:22.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:27.082 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:42:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:42:32.082 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:37.082 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:42.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:47.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:52.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:42:57.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:02.083 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:07.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:12.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:17.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:22.084 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:27.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:43:28.510 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:43:32.084 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:37.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:42.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:47.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:52.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:43:57.085 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:02.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:07.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:12.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:17.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:22.086 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:27.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:44:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:44:32.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:37.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:42.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:47.087 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:52.088 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:44:57.088 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:02.088 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:07.088 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:12.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:17.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:22.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:27.089 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:45:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:45:32.089 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:37.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:42.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:47.090 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:52.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:45:57.090 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:02.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:07.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:12.091 INFO 72445 --- [taskScheduler-2] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:17.091 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:22.092 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:27.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:46:28.511 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:46:32.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:37.092 INFO 72445 --- [taskScheduler-7] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:42.092 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:47.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:52.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:46:57.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:02.093 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:07.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:12.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:17.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:22.094 INFO 72445 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:27.094 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:28.511 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.</div><div>2019-03-06 08:47:28.512 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.RemoteRepositoryService : Skipped updating the JWT token as old one equals the new one.</div><div>2019-03-06 08:47:32.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:37.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:42.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:47.095 INFO 72445 --- [taskScheduler-5] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB</div><div>2019-03-06 08:47:52.095 INFO 72445 --- [taskScheduler-2] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 2272MB / 6144MB ....</div>
Find more posts tagged with
Sort by:
1 - 5 of
51
- Which version of the Jackhammer extension are you using?
(Rapidminer menu: Extensions -> About installed Extensions-> About Jackhammer Extension) - Which Rapidminer server version are you using?
- Do you know if your FTP server has a timeout defined, if yes what is the defined value?
- Do you know the name of your FTP server ?
- You said the issue occurs irregularly, but if the issue occurs, how do you get your process to work again? Do you just restart your server, or is it sufficient to restart the process?
- Which version of the Jackhammer extension are you using?
(Rapidminer menu: Extensions -> About installed Extensions-> About Jackhammer Extension)
2.1.784 - Which Rapidminer server version are you using?
9.1 - Do you know if your FTP server has a timeout defined, if yes what is the defined value?
No, and no access to config. But could experiment with another server though.
- Do you know the name of your FTP server ?
46.30.211.164:21 - You said the issue occurs irregularly, but if the issue occurs, how do you get your process to work again? Do you just restart your server, or is it sufficient to restart the process?
I manually stop the process and re-run it without any issue (often a couple of hours later, because I check the server executions once a day)
David_A
New Altair Community Member
@land can you perhaps help?
New Altair Community Member
Hello,
I am part of the Old World Computing team (as a Student) and I would like to try and reproduce said bahavior to locate the issue and hopefully fix it.
But for that I am in need for additional Information, since the cause of the issue could be anything:
It would be a big help if you could provide answers to as many of the above questions as you can, so I can investigate this issue in a quick manner.
kind regards
Burhan Yildiz
I am part of the Old World Computing team (as a Student) and I would like to try and reproduce said bahavior to locate the issue and hopefully fix it.
But for that I am in need for additional Information, since the cause of the issue could be anything:
It would be a big help if you could provide answers to as many of the above questions as you can, so I can investigate this issue in a quick manner.
kind regards
Burhan Yildiz
Altair EmployeeNew Altair Community Member
OPThanks @Burhan_Yildiz for reaching out. Hope this helps:
New Altair Community Member
Accepted Answer
Hi,
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz
Sort by:
1 - 1 of
11
New Altair Community Member
Accepted Answer
Hi,
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz
we have invastigated and isolated and fixed the issue. Your guess was right, in case of an overload of the server the operator would run into an infinite loop, since it could not set up the ftp enviroment in a correct manner and in order to not mess anything up on your remote file system (FTP server), it would keep failing during settiung up the evniroment and not be able to continue, hence run into an infinite loop.
Now the operator would simply fail, allowing you to handle the exception with the (HandleException Operator).
A release was made this morning and can be downloaded from the rapidminer marketplace (https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_toolkit).
Please make sure to update your extension to the latest version to resolve this issue.
Since there are many different FTP servers out there, you still might see issues for your specific ftp server. In that case just reach out to us (Old World Computing) again and we will resolve it.
kind regards
Burhan Yildiz