After completing a process, does RM release memory it consumed during execution?

tennenrishin

I have a process that, under certain conditions, consumes about 14GB of memory during its execution. This process does not have any outputs, and therefore no RM results are generated. It only generates reports as files.

Isn't RM supposed to release this memory after the process has completed?

MariusHelf

Not necessarily, it is up to the java garbage collection. I don't know how much you are familiar with the guts of java, but even if RapidMiner declares memory as free, it does not necessarily mean that java frees it up for system use directly.
However, to trigger the garbage collection manually and free all memory that RapidMiner does not need anymore, you can use the Free Memory operator.

Additionally, RapidMiner stores intermediate results at each output of each operator, which may also consume some memory.


Best regards,
Marius

tennenrishin

Thanks Marius. More technically, I should perhaps have asked: Does RM retain references to objects in that memory (thus preventing their collection by the GC)?

Because to all appearances, it does. I have experimented quite a bit in the past (including trying the Free Memory operator), but exiting and restarting RM seems to be the only way to get things back to normal after running (and completing/stopping) this memory-intensive process.

Typically, I have to run a process or two, restart RM, run a process or two, restart RM, run a process or two, etc. If I just keep running processes without restarting RM, then RM grinds to a halt with an error message about insufficient memory.

(I can't recall why, but my suspicion is that it may be something in the reporting extension holding onto objects after the report-generating process has long completed. Is such a thing possible, or is there some internal policy through which RM ensures that no references of resources used by operators survive?)

Any advice/info would be appreciated.
Thanks in advance
Isak

MariusHelf

It seems that the Reporting Extension does not free all memory. I don't know yet the exact reason why, but I already created an internal issue for that.

tennenrishin

Thanks Marius!