Purging Old Job Agent Logs
bmatt
New Altair Community Member
I've been searching around documentation and the community, but I can't seem to find anything about purging old job logs? We have accumulated nearly 200GB of job data that I don't think we really need any longer. It appears to be linked to a DB and provided via the RM server UI, but no way to clean them out there. I assume if I just delete the data we'd have some UI errors following.
# pwd
/opt/rapidminer/rapidminer-server/rapidminer-server-home/job-agent-home/data
# du -shx jobs
167G jobs
0
Best Answer
-
Hi bmatt,
welcome to the RapidMiner Community!
The automatic purging of logs and job data will be released soon - so stay tuned
In the meantime you could manually delete the jobs directory. Obviously the consequence is that the UI will not longer be able to display the logs of the old executions, but I guess this is fine from your point of view.
Cheers,
Marcel1
Answers
-
What I've found is there are jars copied to jobs as they run, they are not cleaned up afterwards. Particularly Radoop jobs have a 44MB hive jar that is copied every time. I create a python script to just purge these jars on inactive jobs that I can run on demand for now. That freed up about 165GB.2
-
Hi bmatt,
welcome to the RapidMiner Community!
The automatic purging of logs and job data will be released soon - so stay tuned
In the meantime you could manually delete the jobs directory. Obviously the consequence is that the UI will not longer be able to display the logs of the old executions, but I guess this is fine from your point of view.
Cheers,
Marcel1 -
Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.1
-
Thanks! The script I made purges the jars which shouldn't, and doesn't seem to, impact the display of the old logs/etc.We are also working on a new execution strategy so that this jar duplication is not longer required. But thanks for pointing it out!1
Altair Employee