"Java Heap Space"

ada

Dear specialists,

I have tried almost everything but I seem to be stuck with a Java maximum memory of 1.5 Gig RAM though I tried on different computers (both windows 2000 and windows xp) with 3 Gig Ram.
It doesn't help to increase the variable in the .bat - file nor in the java console - it always tells me cannot create Virtual Memory... if I enter anything above 1.5 Gig.

so the maximum I always end up with is 1.4 Gig (as it says in the GUI)
The process I want to run is a Weka-Adaboost, admittedly the training-data size is quite large (more than 2000000 rows and 25 attributes) but I tried the same process with a smaller training size (500000 and 25 attributes) and it worked ok.

So is there ANY way to convince this stupid JVM to use all my RAM??????

below is the log-file of my process:


G Mar 3, 2009 2:19:12 PM: Checking properties...

G Mar 3, 2009 2:19:12 PM: Properties are ok.

G Mar 3, 2009 2:19:12 PM: Checking process setup...

G Mar 3, 2009 2:19:12 PM: Inner operators are ok.

G Mar 3, 2009 2:19:12 PM: Checking i/o classes...

G Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.

G Mar 3, 2009 2:19:12 PM: Process ok.

P Mar 3, 2009 2:19:12 PM: Logging: log file is 'ada_log.log'...

P Mar 3, 2009 2:19:12 PM: Initialising process setup

P Mar 3, 2009 2:19:12 PM: Checking properties...

P Mar 3, 2009 2:19:12 PM: Properties are ok.

P Mar 3, 2009 2:19:12 PM: Checking process setup...

P Mar 3, 2009 2:19:12 PM: Inner operators are ok.

P Mar 3, 2009 2:19:12 PM: Checking i/o classes...

P Mar 3, 2009 2:19:12 PM: i/o classes are ok. Process output: Model.

P Mar 3, 2009 2:19:12 PM: Process ok.

P Mar 3, 2009 2:19:13 PM: Process initialised

P Mar 3, 2009 2:19:13 PM: [NOTE] Process starts

P Mar 3, 2009 2:19:13 PM: Process:

  Root[0] (Process)

  +- ExampleSource[0] (ExampleSource)

  +- W-AdaBoostM1[0] (W-AdaBoostM1)

  |  +- W-REPTree[0] (W-REPTree)

  +- ModelWriter[0] (ModelWriter)

G Mar 3, 2009 2:27:48 PM: [Fatal] OutOfMemoryError occured in 1st application of W-AdaBoostM1 (W-AdaBoostM1)

G Mar 3, 2009 2:27:48 PM: [Fatal] Process failed: Java heap space

          Root[1] (Process)

          +- ExampleSource[1] (ExampleSource)

here ==> +- W-AdaBoostM1[1] (W-AdaBoostM1)

          |  +- W-REPTree[0] (W-REPTree)

          +- ModelWriter[0] (ModelWriter)

keith

Are you booting XP/2000 with the /3GB boot parameter?

http://msdn.microsoft.com/en-us/library/ms791558.aspx

ada

thanks for the reply!

but no I don't, why should I?

I have more the 3.4 Gig RAM available in my system and those I want the JVM to use.

would it help if I did this boot thing?

keith

In short, it's a WIndows problem, not a Java or RapidMiner problem. 

The link I posted explains it in detail, but it summarized by:  "On Windows, by default, the lower 2 GB are reserved for user-mode programs and the upper 2 GB are reserved for kernel-mode programs."

Basically, when you run a 32-bit Windows operating system (as I assume you are), regardless of the amount of RAM you have installed, by default, it will only allocate a maximum of 2GB to an application (reserving the other 2GB for the OS).  With Java running, it basically means only 1.5GB is actually available to RapidMiner.

The switch changes the operating system allocation to allot 3 GB to applications (and 1 GB for the OS), so you may be able to run RM with more memory.

ada

I did try it now with the /3GB boot-option but this doesn't change my JVM problem, since it still doesn't allow me to use more than 1.4 GB

is there any way around??? or is there any other java version that allows for a bigger max memory????

keith

OK, I just checked on my own system, and the /3GB switch allowed me to get a little more memory allocated to RM (about 1.4 GB), but not as much as I had thought.  A bit of Googling explained why.

Basically, Java requires heap space to be allocated in a single contiguous block of memory, but Windows 32-bit address space is fragmented, and can't allocate a single block of memory larger than about 1.5GB, even with the /3GB switch.  In other words, Windows 32-bit can't give the memory to Java the way Java needs it.

A more complete answer is here:  http://blogs.sun.com/moazam/entry/why_can_t_i_allocate

There are other applications (e.g. I've done it with "R" statistical software), which can deal with multiple chunks of memory, and can take advantage of the /3GB to allocate a total of about 2.8GB of RAM.  But anything Java-based such as RapidMiner is going to have the same problem you've encountered here.  The best solution is to upgrade to a 64-bit version of Windows or Linux, which doesn't have the memory limitations of 32-bit to contend with. 

ada

thanks a lot for your help!!!

so I'll have to finally upgrade.....grrrr

laurab

Hi Ada,

I think I was having the trouble as you a while ago.  I seem to rember changing to a newer version of java and the problem resolved its self.

Dont know if this will help you but might be worth a try...

Laura

Marcello_Sandi

Hi people,

I was reading this forum and I was interested about this issue.

First of all, windows 32bits has no chance with big dataset and java.

To use all memory with java, as you know, it's necessary use 64bits operational systems and install 64bits jvm (http://java.sun.com/javase/downloads/index.jsp)

Now, windows 64bits don't has the same performance compared to linux 64bits, both with the same hardware.

I my case, I installed Mandriva 64bits, but I recommend Fedora 64bits too over desktops or laptops.

I has a Sony Vaio VGN-FZ470E with with 4GB RAM. I run big dataset to text mining tasks it. Over 2000 attribute and 3000 examples. It's no problem, in 10 to 20 minutes all k-means, EM, SVM, Trees ends.

I learned to optimize my laptop, and as a result, my laptop, literally, fly.   ;D

First, if you have 4GB RAM try to put this jvm arguments to starts your RapidMiner:

-Xms3078m -Xmx3078m -Xmn2048m -Xss128k -d64 -server -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:+UseParallelOldGC  -XX:+AggressiveOpts -XX:ParallelGCThreads=20

THIS PARAMETERS WORK OVER WINDOWS TOO.

References: http://java.sun.com/performance/reference/whitepapers/tuning.html

It's possible running faster, but now you need do the instructions in:
  http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html

After, your jvm arguments should changes to:

-Xms2048m -Xmx2048m -Xmn1024m -Xss128k -d64 -server -XX:SurvivorRatio=8 -XX:+UseParallelGC -XX:+UseParallelOldGC -XX:+UseLargePages -XX:+AggressiveOpts -XX:ParallelGCThreads=20

LargePages it's possible only in Linux and Unix based systems (or Windows 2003 Advanced Server, but who has this SO?). It's the difference about windows systems.

With the last changes, my laptop is flying ....  ;D

Good luck,
Marcello Sandi

land

Hi Marcello,
this seems to be very interesting. Have you already performed some reliable tests? I would be very interested in the influence of the different systems and JVM settings. Do you have a nice setup there?

Greetings,
  Sebastian

Marcello_Sandi

Hi,

Specifically, I don't do any logger to prove my theory about tunning.

It was based in do and test. But with those site, everyone can performe your jvm and Linux. I researched many sites and foruns.

Actually, I'm working with a text mining process, specifically with opinion mining.

I needed to process a very large BOW and, before this perform, I could not do.

So, first of all, I studied jvm arguments tunning. With it I was able to do something with my BOW.

After, I studied how to improve perform Linux to running jmv fastest. And now it's possible to process everything so fast.

I found the best fit for my laptop. Each one needs to find its.

The secret is one. JVM running fast with continuous memory, because the kernel architecture it's based to work like that.

Recalling, who invented the JVM and what operating system it has?

My friend, Sun invented Java to running in any operational system, it's truth, but just Solaris and a few Unix operational systems has large memories by default.

In Linux it's possible to configure large memories, but in windows I heard that only Windows Advanced Server it's capable running with large memories. The others can not because are based in DLL kernels and, for this reason, it is not possible its configurations.

Pass me your configurations about hardware and software that I'll try to fit, or start, one for you. 

Sincerely,
Marcello Sandi