Large data set model apply

Question

Hi, I try to apply model on 10mln records database. I use "read database" operator but it copies all data from database to memory in my computer so it coses out of memory exception, moreover there is timeout on database. "Stream database" looks nice but it looks like it works only to make model not to apply (I got an error when applaying with this operator). I think about building a loop to get data with parametrized SQL limit - limiting data f.eg. to 10 000 records  is working very well in applying model. Please help - I think there is smarter way than making loops. Most of ETL got streaming DB read.
Thanks

Krystian · Answer

I got: Apr 11, 2012 1:19:44 PM SEVERE: Process failed: operator cannot be executed. Check the log messages... Apr 11, 2012 1:19:44 PM SEVERE: Here: Process[1] (Process) subprocess 'Main Process' ==> +- Stream Database[1] (Stream Database) +- Write CSV[0] (Write CSV) Apr 11, 2012 1:19:44 PM SEVERE: java.lang.NullPointerException with Stream database connected only to CSV output or even to screen: Now I'am testing to export RMiner PMML export and use it in streaming process in Pentaho. I will write how it works. Thanks

MariusHelf · Answer

Hi Krystian,

using a loop is a perfect work-around if Stream Database does not work for you. As always, posting your process setup and the details of the error message could be useful.

Best, Marius