"Running Process on Server"
Alexandros_Tzekas51
New Altair Community Member
Hello Guys,
I 'm trying to run processes on Server but i fail everytime! I did go to VIEW DETAILS and get the ERROR MESSAGE:
"Out of memory- The process has failed as it ran out of memory. Try to run the process again with less data or choose another queue with larger Job Containers."
I did as Edin_Klapic had said to me ("You need to assign more RAM to the job agent. The file is agent.properties located in job-agent/home/config.") and assign more RAM to the job agent. I even tried to run a proccess with very very low RAM requirements (some kB) but didn't work!
Any ideas?
I give you the LOG:
2018-12-19 15:00:42.111 INFO 5855 --- [main] c.r.execution.jobcontainer.Application : Starting Application on alex-HP-Compaq-8000-Elite-CMT-PC with PID 5855 (/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container/rapidminer-jobcontainer-0.4.7.jar started by root in /home/alex/rapidminer-server/rapidminer-server-9.0.4/bin)
2018-12-19 15:00:42.135 INFO 5855 --- [main] c.r.execution.jobcontainer.Application : No active profile set, falling back to default profiles: default
2018-12-19 15:00:50.205 INFO 5855 --- [main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@2d1ef81a: startup date [Wed Dec 19 15:00:50 EET 2018]; root of context hierarchy
2018-12-19 15:00:54.428 INFO 5855 --- [main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler'
2018-12-19 15:00:58.428 INFO 5855 --- [main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2018-12-19 15:00:58.442 INFO 5855 --- [main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2018-12-19 15:00:58.602 INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.
2018-12-19 15:00:58.600 INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 544MB / 500000MB
2018-12-19 15:00:58.623 INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService : Updated JWT token from auth properties file.
2018-12-19 15:00:58.686 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Changing 'user.home' system property to '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4'
2018-12-19 15:00:58.686 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Generating new in-memory cipher key
2018-12-19 15:00:59.017 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading JDBC properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/jdbc/jdbc_properties.xml'
2018-12-19 15:00:59.580 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Setting pre-defined RapidMiner properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4/rapidminer.properties'
2018-12-19 15:00:59.597 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Initializing the RapidMiner Core
2018-12-19 15:00:59.597 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Setting execution mode to 'JOB_CONTAINER'
2018-12-19 15:00:59.711 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading bundled extension from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container/plugins'
2018-12-19 15:00:59.713 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading custom extensions from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/extensions'
2018-12-19 15:00:59.730 INFO 5855 --- [main] com.rapidminer : Performing upgrade to version 9.0.003
2018-12-19 15:00:59.946 INFO 5855 --- [main] com.rapidminer : Set locale to en.
2018-12-19 15:01:00.106 INFO 5855 --- [main] PlatformUtilities : rapidminer.home is '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container'.
2018-12-19 15:01:00.164 INFO 5855 --- [main] com.rapidminer : Initializing license manager.
2018-12-19 15:01:00.438 INFO 5855 --- [main] com.rapidminer : Cannot access file system in execution mode JOB_CONTAINER. Not loading repositories.
2018-12-19 15:01:03.663 INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 544MB / 500000MB
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Model Simulator
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Time Series
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Professional
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Process Scheduling
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: PMML
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: JDBC Connectors
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Advanced File Connectors
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Productivity
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Social Media
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Legacy Result Access
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Concurrency
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Remote Repository
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: H2O
2018-12-19 15:01:05.309 INFO 5855 --- [main] com.rapidminer : Register plugin: Cloud Connectivity
2018-12-19 15:01:06.900 INFO 5855 --- [main] com.rapidminer : Registered configurator Twitter Connection.
2018-12-19 15:01:07.458 INFO 5855 --- [main] com.rapidminer : Registered configurator Salesforce Connection.
2018-12-19 15:01:07.458 INFO 5855 --- [main] com.rapidminer : Registered configurator Amazon S3 Connection.
2018-12-19 15:01:07.459 INFO 5855 --- [main] com.rapidminer : Registered configurator Azure Blob Storage Connection.
2018-12-19 15:01:07.472 INFO 5855 --- [main] com.rapidminer : Registered configurator Google Cloud Storage Connection.
2018-12-19 15:01:07.472 INFO 5855 --- [main] com.rapidminer : Registered configurator Dropbox Connection.
2018-12-19 15:01:08.664 INFO 5855 --- [taskScheduler-3] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 576MB / 500000MB
2018-12-19 15:00:42.135 INFO 5855 --- [main] c.r.execution.jobcontainer.Application : No active profile set, falling back to default profiles: default
2018-12-19 15:00:50.205 INFO 5855 --- [main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@2d1ef81a: startup date [Wed Dec 19 15:00:50 EET 2018]; root of context hierarchy
2018-12-19 15:00:54.428 INFO 5855 --- [main] o.s.s.c.ThreadPoolTaskScheduler : Initializing ExecutorService 'taskScheduler'
2018-12-19 15:00:58.428 INFO 5855 --- [main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2018-12-19 15:00:58.442 INFO 5855 --- [main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2018-12-19 15:00:58.602 INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService : Trying to update JWT token from auth properties.
2018-12-19 15:00:58.600 INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 544MB / 500000MB
2018-12-19 15:00:58.623 INFO 5855 --- [taskScheduler-2] c.r.e.j.service.RemoteRepositoryService : Updated JWT token from auth properties file.
2018-12-19 15:00:58.686 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Changing 'user.home' system property to '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4'
2018-12-19 15:00:58.686 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Generating new in-memory cipher key
2018-12-19 15:00:59.017 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading JDBC properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/jdbc/jdbc_properties.xml'
2018-12-19 15:00:59.580 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Setting pre-defined RapidMiner properties from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/data/jobs/75d9e747-7381-45dc-963b-dd18ba50b7a4/rapidminer.properties'
2018-12-19 15:00:59.597 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Initializing the RapidMiner Core
2018-12-19 15:00:59.597 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Setting execution mode to 'JOB_CONTAINER'
2018-12-19 15:00:59.711 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading bundled extension from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container/plugins'
2018-12-19 15:00:59.713 INFO 5855 --- [main] c.r.e.core.RapidMinerInitializer : Loading custom extensions from '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/home/resources/extensions'
2018-12-19 15:00:59.730 INFO 5855 --- [main] com.rapidminer : Performing upgrade to version 9.0.003
2018-12-19 15:00:59.946 INFO 5855 --- [main] com.rapidminer : Set locale to en.
2018-12-19 15:01:00.106 INFO 5855 --- [main] PlatformUtilities : rapidminer.home is '/home/alex/rapidminer-server/rapidminer-server-9.0.4/job-agent/lib/container'.
2018-12-19 15:01:00.164 INFO 5855 --- [main] com.rapidminer : Initializing license manager.
2018-12-19 15:01:00.438 INFO 5855 --- [main] com.rapidminer : Cannot access file system in execution mode JOB_CONTAINER. Not loading repositories.
2018-12-19 15:01:03.663 INFO 5855 --- [taskScheduler-1] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 544MB / 500000MB
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Model Simulator
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Time Series
2018-12-19 15:01:05.307 INFO 5855 --- [main] com.rapidminer : Register plugin: Professional
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Process Scheduling
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: PMML
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: JDBC Connectors
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Advanced File Connectors
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Productivity
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Social Media
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Legacy Result Access
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Concurrency
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: Remote Repository
2018-12-19 15:01:05.308 INFO 5855 --- [main] com.rapidminer : Register plugin: H2O
2018-12-19 15:01:05.309 INFO 5855 --- [main] com.rapidminer : Register plugin: Cloud Connectivity
2018-12-19 15:01:06.900 INFO 5855 --- [main] com.rapidminer : Registered configurator Twitter Connection.
2018-12-19 15:01:07.458 INFO 5855 --- [main] com.rapidminer : Registered configurator Salesforce Connection.
2018-12-19 15:01:07.458 INFO 5855 --- [main] com.rapidminer : Registered configurator Amazon S3 Connection.
2018-12-19 15:01:07.459 INFO 5855 --- [main] com.rapidminer : Registered configurator Azure Blob Storage Connection.
2018-12-19 15:01:07.472 INFO 5855 --- [main] com.rapidminer : Registered configurator Google Cloud Storage Connection.
2018-12-19 15:01:07.472 INFO 5855 --- [main] com.rapidminer : Registered configurator Dropbox Connection.
2018-12-19 15:01:08.664 INFO 5855 --- [taskScheduler-3] c.r.e.j.service.OutOfMemoryWatchdog : Current memory usage is 576MB / 500000MB
Best Regards,
Tzekas Alexandros
0
Best Answers
-
Hi Alexandros,you should check the memory and number of containers assigned to the job agent. The safest configuration would be to use only one job container with memory equal to the system memory (this will run only one process at the time!). You can find the configuration file in:$PATH_TO_SERVER/job_agent/home/config/agent.propertiesThe file is very small and is easy to understand. Let me know if it has worked out.Regards,Sebastian5
-
Hi I have some experience with the job agents.
Go to your agents file and get the config.properties file and edit it with a notepad.
File should be on
[RM Folder]\job-agent\config
The line you need to change is:
jobagent.container.memoryLimit = 10000
The number is on MB so check how many GB you may allocate. remember that the agent shares memory with the server and all the things that are set up on it so you may have a constraint there.
After setting the new memory of the agent you will need to restart it.
If you are working with Windows Server go to the bin folder of the agent and run
stop-job-agent.bat and then rapidminer-jobagent.bat
Then launch your studio or server an trigger your process.
Do you really have 500GB on that server? The log you shared doesn´t show the actual part in which your process failed and it was just using 576MB.
Other things you may check is trying to run your process with the LoopBaches operator in order to handle the information. Hope this helps you.
1
Altair EmployeeAnswers
-
Hello,
Is your process something you can execute on studio without memory or any other errors?
0 -
Yes, at Studio i get no Errors.
0 -
Guys, please i really really need some answers on these questions cause i have to start my experiment. I have only 2 weeks to get my results!
0 -
hi @Alexandros_Tzekas51 I'm sorry but I don't know job agents well enough to help you here. Most RM folks are on holiday right now...
Scott0 -
Hi Alexandros,you should check the memory and number of containers assigned to the job agent. The safest configuration would be to use only one job container with memory equal to the system memory (this will run only one process at the time!). You can find the configuration file in:$PATH_TO_SERVER/job_agent/home/config/agent.propertiesThe file is very small and is easy to understand. Let me know if it has worked out.Regards,Sebastian5
-
Hi I have some experience with the job agents.
Go to your agents file and get the config.properties file and edit it with a notepad.
File should be on
[RM Folder]\job-agent\config
The line you need to change is:
jobagent.container.memoryLimit = 10000
The number is on MB so check how many GB you may allocate. remember that the agent shares memory with the server and all the things that are set up on it so you may have a constraint there.
After setting the new memory of the agent you will need to restart it.
If you are working with Windows Server go to the bin folder of the agent and run
stop-job-agent.bat and then rapidminer-jobagent.bat
Then launch your studio or server an trigger your process.
Do you really have 500GB on that server? The log you shared doesn´t show the actual part in which your process failed and it was just using 576MB.
Other things you may check is trying to run your process with the LoopBaches operator in order to handle the information. Hope this helps you.
1 -
Happy New Year to you.As @MarcoBarradas already said: The log file extract you posted does not state where the process failed so helping is difficult at this point.How much RAM does your machine have?Happy Mining,EdinP.S.:When you write the name of the community members with a leading @ - it results in a notification service and the names are colored in orange.1
