"De-Pivot Memory issue"

I would suggest that you do some testing first to determine the maximum number of records that you can get to run successfully through your two-step de-pivoting process.  You should be able to do that manually by adding a "filter example range" operator and just trying increasingly larger values until the process fails.  

One you know how many examples you can successfully process, then you should be able to create a loop to go through the examples in suitable chunks, complete the de-pivoting, and then store the results to a repository entry.  All those repositories should then be able to be appended together as the last step.  

Hi Brian,

Thanks a lot for your input. Would you please give me more detail on which operator to select (just "loop" or another one?) and how to connect it with the corresponding appends? Would you mind sharing an XML?

Thanks once again,

Without seeing your process or your data it is difficult to give you an exact solution.  But take a look at this example, which is a simple modification of the de-pivot tutorial process.  This loops through each example (one-by-one) and depivots it, then at the end it takes all the individual de-pivoted examples and appends them into one combined dataset.  Something like this should work for you (assuming your resulting dataset is in fact able to be contained in memory).

Regards,

Thanks a lot Brian, that solved my problem!!!

@jlabado I am glad that solved your problem!  You may want to "accept as solution" that post so if other community members are searching for this topic or something similar then it will show up in the results as solved.