Hi...so I know the old parallel processing extension is no longer functional and I understand that you're working on parallelization of other operators. I have a simple problem - I have 50,000 examples which need to send a text attribute through an API one at a time. I need to make the API calls separately as sometimes the operator (Enrich Data by Webservice) will return an error and hence I put it into a Handle Exception operator to keep going. This all works fine but it is SLOW - maybe 1-4 API calls/second? I cannot tell the bottleneck but I would like to parallelize this by splitting the data set into 10 partitions (i.e. 5,000 per partition) and working on all 10 partitions at the same time. See attached process for a general framework (some stuff deleted).
With the old parallel processing extension, this would work fine. But now it is still going through each subprocess (i.e. each partition) one at a time. Ugh. SO I had an idea - could I somehow trick the fancy new Cross Validation operator to basically do the same thing? Thoughts?
Scott
