LSTM Deep Learning Forecast - Validation

Oprick

Hello,
Yesterday I posted here a question about a walkforward validation of forecasting model that was solved.
But @SGolbert pointed that LSTMs are becomimg very important in forecasting and because I've already start studying this subject I decided it was time to give it a try. Results are quite fair though tuning is more thorny.

I'm using this extension operators:
https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_deeplearning

My question is basically if it is possible to validate the models built using this extension with, for example, sliding window validation operator.
When I try to connect mod outport with validation operator mod ports I get a error. I understand the reason of the error, but I'm stucked. 

How can we backtest models build under this extension.

Enclosed mock example set and process.

Thanks for your help

hughesfleming68

Hi Opick. This can work with the sliding window operator. Here is your process with the validation.

In the real world, I have had mixed results with LSTM's. I have had better results using dilated causal convolution networks following the Wavenet model. Unfortunately you would need some experience with Python/Keras/Tensorflow but it can all work inside of Rapidminer with the execute Python operator.

The warning here is the enormous amount of time it takes to get Deep Learning networks tuned. Between the number of layers, neurons and activation functions, it really requires a commitment.

It might also be interesting for you to take a look at the M4 Forecasting competition and see how some of the top solutions worked. Plenty of discussion as well about if these techniques out perform classical methods or not. That answer isn't absolutely clear sometimes.

regards,

Alex

lionelderkrikor

HI @Oprick,

I just imported and executed the process you shared with your data and ..... everything works fine...
Can you tell me more about your error ?

Regards,

Lionel

hughesfleming68

Hi Opick. This can work with the sliding window operator. Here is your process with the validation.

In the real world, I have had mixed results with LSTM's. I have had better results using dilated causal convolution networks following the Wavenet model. Unfortunately you would need some experience with Python/Keras/Tensorflow but it can all work inside of Rapidminer with the execute Python operator.

The warning here is the enormous amount of time it takes to get Deep Learning networks tuned. Between the number of layers, neurons and activation functions, it really requires a commitment.

It might also be interesting for you to take a look at the M4 Forecasting competition and see how some of the top solutions worked. Plenty of discussion as well about if these techniques out perform classical methods or not. That answer isn't absolutely clear sometimes.

regards,

Alex

Oprick

Hi,
Thanks @hughesfleming68
I saw the red warning at mod port and didn't try to run the process :S  Dumb be.

Thanks a lot also for the tips.

Regards,
Pedro

SGolbert

You really need an expensive GPU for that. I've trained a simple RNN model based on reading for one stock of the Nasdaq exchange from the last 5 years, and it took good 20 minutes to train with CPU (csv file was 65 kb!). In comparison, a VAR model for the same task trains almost instantenously, and it's a good model.

I think Deep Learning makes sense only when the task is intrinsictly very complex, so that simple models just don't cut it.

Regards,

Sebastian

hughesfleming68

Hi @SGolbert, one advantage of CNN's for time series is that they are generally much lighter for similar or better results than LSTM. Worth checking out for the time savings. In many cases a GPU is not necessary. I am using the CPU with Tensorflow and it is not unbearable in terms of training time.

regards,

Alex

SGolbert

Hi @hughesfleming68

the problem is that you have several parameters and NN architectures. If you really want to optimize it, it becomes impossible (unless you plan on taking a holiday). If you want to train a given NN with given parameters, then it is doable, but it's not a realistic situation.

Do you have some experience with pre-trained networks?

Regards,

Sebastian

hughesfleming68

Hi @SGolbert, I agree with you on the time it takes to optimize. It is a part of your life that you never get back! On the other hand, there are known network architectures like AlexNet, LeNet, RezNet and in my case WaveNet which provide good starting points for many different tasks. It makes more sense to fine tune a known architecture than to build a complex one from scratch.

I have no experience with pre-trained networks except from what I have read.

Regards,

Alex

WT

Hi Alex @hughesfleming68  following your post above I have looked at Wavenet for time series prediction. My project involves using both the underlying stock price and other time series data with the same frequency i.e. daily (alpha, sector performance and some technical indicators) to predict the stock price five days forward. My questions are a) is there an existing implementation of Wavenet in Rapidminer already? b) if not, are there any existing time dilated causal neural network implementations or libraries which could solve for this problem in Rapidminer? Thank you - any assistance welcome as I could not find those answers in any of the documentation or online tutorials.

hughesfleming68

Hi WT, there isn't an existing off the shelf implementation of wavenet or modified wavenet architectures in Rapidminer. You can however do multivariate CNN with the DL4J Deep Learning extension but unless something has changed, dilation is not part of the extension. That may or may not be a problem if you intend to use multiple attributes. I have had good (repeatable) results with the Deep learning extension. 

What you can do and what I use is Tensorflow inside Rapidminer with the execute python operator which opens more possibilities. Tensorflow is convenient because you can find lots of code on Github which makes learning by example possible. For an example of dilated cnn for time series forecasting, I would look at seriesnet.py by Krist Papadopoulos which he has kindly made available on Github. https://github.com/kristpapadopoulos/seriesnet. You would however have to install Anaconda, Keras and Tensorflow. That can be quite a big jump for many users unless you already have that working. There are also a few extra things to learn about getting repeatable results in Tensorflow and it is slow. There are also gated convolution NN's and at least five or six variations that I can think of. You might have to explore quite a few different options until you find a technique that works for you.

regards,

Alex

WT

Thank you Alex - this is super helpful and I so appreciate this as its been harder than I thought to get help on this so thrilled to read your reply today and very grateful. I will try those possible solutions and sure I can find one that will work. Kind regards Warren

Yunes

Thanks a lot @hughesfleming68  Mr. Alex  please how I can used for forcastting two output for LSTM and MLP , Thank you!