Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Poor data scrolling, GUI issues !

Hi,

I am relatively new to RabidMiner nevertheless I have successfully installed rapidminer-5.1.006x64.exe on my Windows 7 x64 (8 GB RAM) and found terrible scrolling performance even with small results as less than 1000 rows in data view. Interesting - dragging scroll view sliders is much more efficient while actually clicking the slider with a mouse and drag the cursor out of the RapidMiner window while keeping the mouse button constantly clicked!

Anyway - I do not cope while scrolling result view of a simple table having ~800 rows and ~10 columns with plain numeric data is consuming more than 1GB of memory by the javaw.exe process??? Please notice - this memory is never released even in case a completely new experiment is opened and performed. In my humble opinion there is some issue with memory management as I was easily able to clog the system previewing a few table of the similar size as previously mentioned what resulted in GUI issues like annoying interactivity lagging, turning toolbar buttons into white "blanks", etc.

Find more posts tagged with

AI Studio

Accepted answers

All comments

Marco_Boeck

Hi,

this sounds like some problem with java.. I'm also using RapidMiner on a Win7 x64 machine, and there are no such problems whatsoever.
However, a quick google search revealed that quite a few people are having memory problems with different java programs and Win 7 x64.
Are you using the latest java version (version 6 update 26)? Are you using the 32 or 64 bit java version?

Regards,
Marco

IngoRM

Hi,

and also a quick jump-in from me as well: I frequently use RapidMiner for millions of rows and hundreds of attributes with no scrolling or GUI problems at (as many others as well): so I also think that this is indeed a problem with your specific configuration - not with RapidMiner in general. Actually, I hear quite often that people are surprised about the good GUI performance of RapidMiner after they learned that it's written in Java. I made a quick test and generated some data sets, starting with 1.000.000 examples and ending with 10.000.000 with 10 attributes and I was able to fluently scroll across all data points in the data view.

So in addition to Marco's comments I would like to ask if you have the option to install RapidMiner on a different machine and test there again. Maybe this helps to find out what's going on on your system.

About the memory usage: The Windows version of RapidMiner allocate the memory for the virtual machine right at start up of the system. This behaviour is described in the installation manual under "Starting RapidMiner: Windows" at

http://rapid-i.com/content/view/17/211/

(which you certainly have read

). Actually, the most recent versions allocate 90% of the free memory instead of 80%. It is allocated in any case and will not be released, no matter what you do.

If you don't like this behaviour, you could use the other starting options or tweak the default one yourself. This also means that you seem to have many programs running already if out of your 8 Gb memory only about 1 Gb was available to RapidMiner (used by javaws). If you start a few more, RapidMiner will begin swapping and then, indeed, using RapidMiner is a pain. With the default settings, starting RapidMiner as the last program is always a good idea (to avoid swapping) or you should adapt the memory settings like it was described on the web page stated above.

By the way: thanks for your comments - but instead of bashing a program without really knowing the backgrounds and what's going on internally does not really motivate others to help and usually lead to a pretty fast "ok, somebody to ignore"-reaction from my side. Stating the same by asking questions ("is it normal that...?") certainly is the more elegant way

Cheers,
Ingo

Ziggi

Hi,

First of all - please (Ingo !) do not blame me for "bashing" RapidMiner as it was not my intention at all. But please notice, it's too easy to push any responsibility for application performance on end-user burden. It is also a responsibility of a software developer to make some effort in a purpose to identify nature of the common problems.

Now, lets discuss the issue:

You may see 1.16 GB memory usage by javaw.exe process. Overall memory usage was ~6GB from 8GB available for the system. Please believe me - RapidMiner was the only foreground application running! Normally, my Windows system with all (many!) background processes I keep running on my machine consume about 4GB. Please notice, just starting RapidMiner increases this to ~5GB. While I just put data loader, load Excel file with data and run the process, switch to result and display example set data, it may increase to ~5.2GB. Then, just through scrolling the data view up and down it can grow up to 6GB. 800 MB just because of scrolling 1000 rows ???

And the scrolling performance is a disaster!

I have just uninstalled all Java and installed latest JRE from Sun Microsystem. No improvemnt at all. Update 26 performs the same poor as Update 24 I had installed until today. But RapidMiner is the only Java application having such problems on my system - I use Eclipse, I use some graphics stuff in Java, etc and it all is working smooth and fine. Just RapidMiner...

Unfortunately, I had yet no possibility to try RapidMiner on another configuration.
I believe you enjoy RapidMiner excellet performance but the question is - what might be wrong with my Java (just fresh re-installed), so it performs that bad in terms of RapidMiner scrolling GUI???

Thanks for feedback and thanks for your FANTASTIC RapidMiner !!!

IngoRM

Hi,

Thanks for feedback and thanks for your FANTASTIC RapidMiner !!!

Now we are talking

First of all - please (Ingo !) do not blame me for "bashing" RapidMiner as it was not my intention at all. But please notice, it's too easy to push any responsibility for application performance on end-user burden. It is also a responsibility of a software developer to make some effort in a purpose to identify nature of the common problems.

You are absolutely right. The problem is: this is not a common problem. Out of about 200.000 frequently running RapidMiner installations there are only few with serious performance problems like those described by you. Believe me: we take these problems absolutely serious but please understand that only very limited resources can be offered by Rapid-I to fix these problems for this relatively small amount of users of the free community edition. However, we will try our best (again for free!) that even those few users can enjoy RapidMiner in the future - and we are relying on your help here since we can hardly fix something we do not experience and never had.

You may see 1.16 GB memory usage by javaw.exe process. Overall memory usage was ~6GB from 8GB available for the system. Please believe me - RapidMiner was the only foreground application running!

It does actually not seem so. As far as I can see from your task manager screenshot, RapidMiner indeed only allocated 1.16 Gb memory (which probably was around 90% of the free memory during starting RapidMiner as indicated in the guide I linked in my first post) - exactly as you have written. If there is 1.16 Gb allocated, it's 1.16 Gb - not more. I don't know what the rest is used for but it is certainly not RapidMiner, sorry...

Please notice, just starting RapidMiner increases this to ~5GB.

This is actually the weird part: If you have 8 Gb and only 4 Gb have been used before RapidMiner has been started (please check this again in the resource view of your task manager where also the amount of free memory and swap is shown), RapidMiner should actually allocated about 3.8 Gb during startup, not only 1.16 Gb.

Could it be that you accidentally have installed the 32 Bit version of RapidMiner? This one indeed is restricted to an amount of memory in this region.

The picture below shows the System Monitor view of my RapidMiner (64 Bit system, 4 Gb memory, RapidMiner allocated 90% of the free memory during startup which is about 1.7 Gb as you can see in the monitor). The picture was taken directly after startup:

From the maximum amount which is available (1.7 Gb), less than 1 / 10, hence about 140 Mb are actually used for RapidMiner itself directly after startup which is fine from my point of view. By the way: if I add the operator "Free Memory" to a process and execute it, the memory usage drops to less than 1/20 which is about 80 Mb. This is the amount of memory which is necessary for the RapidMiner core which is quite good (compare that to Excel or the IE

)

The rest of the 1.7 Gb on my system is available for data, models etc. If I load a data set with 800 examples and 30 attributes (about your data size), the monitor hardly moves, so only something like 10 Mb or so have been used for all data, meta data, and visualization. Not optimal but still fine. The following picture show the result:

So now I start scrolling and indeed memory usage is increasing (this is quite typical for Java tables but nevertheless...). But you stated that this memory is never freed again and this is simply not true. Look at the next picture. Here I scrolled the data (which increased used memory) and re-executed the process which directly freed the used memory:

This is perfectly normal: the virtual machine needs memory for the visualization and directly frees it again as soon as necessary or asked for by the garbage collector. So everything is working like a charm and no performance issue at all (I could produce a video of scrolling a couple of million of examples but I am sure that you believe me).

Did you check the monitor yourself? How does it look for you direct after startup? After loading the data?

I still don't have a clue what's causing the problems on your system, but I notices several quite weird things

Why is your empty Windows system using about 4 Gb of memory (for comparison: mine uses about 1.4 Gb)
Why is RapidMiner only allocating 1.16 Gb of memory? 90% of the remaining free should be between 3.5 and 4 Gb (really using the 64 Bit version)?
Is RapidMiner swapping? This is the No 1 cause for performance issues?

I have just uninstalled all Java and installed latest JRE from Sun Microsystem. No improvemnt at all. Update 26 performs the same poor as Update 24 I had installed until today.

Only installing a new Java might not be enough! RapidMiner is delivered with its own runtime environment - so you will have to tell RapidMiner which Java should be used. More information can be found in the installation guide I linked to above.

Cheers,
Ingo

wessel

Dear All,

I have also experienced this issue. Not sure how I solved it. I believe the trick is simply using a computer with lots of memory.

I tried loading this data set: (It is 121 MB)
http://www.few.vu.nl/~wln320/val0.arff

Then I went into task manage to see how much memory Java was using: 3.5GB!
Then I loaded the data again: memory went up to 4.3GB.
Loading the data again and memory increased to 4.9GB.
Then I went into data view, scrolling worked nice and smooth.

This might be a Java issue, instead of a Rapid Miner issue.
Java collections always take up lots of memory.
(Although I believe Rapid Miner is using the colt primitive collection package).
To find out better what is the problem you could download:
http://www.ej-technologies.com/products/jprofiler/overview.html
Profiling Rapid Miner will be slow, but should give you a clear overview of CPU and memory usage by process.

Best regards,

Wessel

Ziggi

Hi guys!

First of all - I am 100% sure I have x64 version of RM installed. Second - various open processes on my system use ~2.5GB memory besides what Windows needs by itself. I have a good reason for keeping these processes running unless I really need some extra memory. But now I started to scrutinize all performance indicators and have suddenly noticed a **** high hard faults rate of my memory chips, so I need to test RAM to see what is going on.

BTW - RapidMiner system monitor show 2.3GB memory usage.
Anyway - the biggest issue is not memory usage but scrolling. I repeat and I would like to comment this point:

Scrolling performance is much much better in case I click a scroll slider and keeping mouse button pressed, move the cursor out of the RM window. Then I can actually scroll the table. But in case the cursor remains within the boundary of RM window, scrolling is jagged, practically impossible.

Please - let me check the memory chips first - page faults are normal for Windows, but I want to verify (indeed, a few weeks ago I have installed new chips) - I'll be back!

Ziggi

OK,

RAM test have proven there is no problem with chips. The question remains - why scrolling is so poor in case cursor is within RM window ???

IngoRM

Dear Ziggi,

I would love to try to help but this means that I definite will need some answers to my questions. I am missing information like:

Are you sure that you are not swapping? Sorry for getting back to memory, but bad scrolling performance is using memory in Java and performance issues during increased memory usage are most often caused by memory swapping done by the system. This can be taken from the task monitor of your system, you could also try to disable swapping for this test.

And related to that (would explain WHY you are swapping): Why is RapidMiner only allocating 1.16 Gb (or 2.3 Gb as indicated in your later reply) of memory? 90% of the remaining free should be between 3.5 and 4 Gb? This does simply not fit and since swapping is the reason for about 95% of all performance problems I have to come back to this...

Sorry for getting back to the memory issue but since this is a likely cause I would like to try to sort this our first before trying to find more exotic problems remotely...

And last but not least: Did you had the chance to try RapidMiner on a different machine? Maybe this is only a question of perception...

Cheers,
Ingo

wessel

What I said earlier: http://www.ej-technologies.com/products/jprofiler/overview.html
This can also really help you find out what is the problem on the problem PC.
It gives a nice overview of CPU, memory, swap, function calls, etc.
It will even give you "hot spots", things that eat away much performance.

Best regards,

Wessel