"De-Pivot Memory issue"

jlabado

Hello Everyone,

 

I am running a process that needs to use 2 De-pivot operators. The dataset is large (aprox 900k examples). The first De-pivot runs fine but the second one crashes out of memory, even though I have a machine with 16GB of RAM.

 

I know there should be some workaround by using the Loop Batches (inserting inside the second de-pivot) and Append operators, but I just can't get it to work. Could you please explain me how to exactly set these operators to work?

Thanks a lot for your help,

Best,

Telcontar120

Without seeing your process or your data it is difficult to give you an exact solution.  But take a look at this example, which is a simple modification of the de-pivot tutorial process.  This loops through each example (one-by-one) and depivots it, then at the end it takes all the individual de-pivoted examples and appends them into one combined dataset.  Something like this should work for you (assuming your resulting dataset is in fact able to be contained in memory).

 

Regards,

 

Telcontar120

I would suggest that you do some testing first to determine the maximum number of records that you can get to run successfully through your two-step de-pivoting process.  You should be able to do that manually by adding a "filter example range" operator and just trying increasingly larger values until the process fails.  

 

One you know how many examples you can successfully process, then you should be able to create a loop to go through the examples in suitable chunks, complete the de-pivoting, and then store the results to a repository entry.  All those repositories should then be able to be appended together as the last step.  

 

 

jlabado

Hi Brian,

 

Thanks a lot for your input. Would you please give me more detail on which operator to select (just "loop" or another one?) and how to connect it with the corresponding appends? Would you mind sharing an XML?

Thanks once again,

Telcontar120

Without seeing your process or your data it is difficult to give you an exact solution.  But take a look at this example, which is a simple modification of the de-pivot tutorial process.  This loops through each example (one-by-one) and depivots it, then at the end it takes all the individual de-pivoted examples and appends them into one combined dataset.  Something like this should work for you (assuming your resulting dataset is in fact able to be contained in memory).

 

Regards,

 

jlabado

Thanks a lot Brian, that solved my problem!!!

Telcontar120

@jlabado I am glad that solved your problem!  You may want to "accept as solution" that post so if other community members are searching for this topic or something similar then it will show up in the results as solved.