Train/Apply Model in "Sliding" Batches
Laszlo
New Altair Community Member
Hi everyone.
I am trying to do the following: Let's say I have 10000 examples in my data set. I would like to take the
first 1000 to train my learner, then apply that model on the next 1000 examples for prediction. Then "slide"
the training examples forward to 1001 to 2000, train, then apply to examples 2001 to 3000. Repeat through the
whole data set. In the end, I would have 9000 predictions. The learner is being re-trained every 1000 data points.
I've been trying the "Loop Batches" operator, but that doesn't quite do what I want. Sliding the window forward
manually is a bit tedious ( esp. if I expand this to several million examples!)
Any thoughts would be greatly appreciated.
I am trying to do the following: Let's say I have 10000 examples in my data set. I would like to take the
first 1000 to train my learner, then apply that model on the next 1000 examples for prediction. Then "slide"
the training examples forward to 1001 to 2000, train, then apply to examples 2001 to 3000. Repeat through the
whole data set. In the end, I would have 9000 predictions. The learner is being re-trained every 1000 data points.
I've been trying the "Loop Batches" operator, but that doesn't quite do what I want. Sliding the window forward
manually is a bit tedious ( esp. if I expand this to several million examples!)
Any thoughts would be greatly appreciated.
Tagged:
0
Answers
-
Use the SlidingWindowValidation operator?0
-
I don't see this operator. Is it in the Series folder? Maybe has a different name?0
-
Series > Evaluation > Validation
By the way, you can hit ctrl + I, to find any operator using text search.0 -
Ah, thank you. I needed to update to the latest Series Extension.
I will experiment with it.0