Web mining, get numerical data through going through provided calendar

felix_w

Dear Rapidminer Community, 

 

only recently I discovered that RM also offers webmining operators which is of great help for me! 

But unfortunately I encountered a problem when I tried to download data from this webpage 

https://www.eex.com/en/market-data/power/spot-market/auction#!/2018/01/03

I would like to download all hourly data 00-01 etc for a specific time (e.g. 01.01.2015 - 30.09.2017) but I can only do this through manually changing the dates day per day in the provided calendar on the website, which is not very handy. 

 

My question now, is there any way in RM to configure the Webcrawler to go through the calender automatically and draw the hourly data?

 

I hope you can help me with my question. 

 

Best regards

Felix

 

Thomas_Ott

This might be tricky indeed. I don't think the webcrawler from RapidMiner can do this and you might have to resort to some outside script to do it (like node.js).  You can use the Execute Program operator to call the script, change the date, load the new page and then run the RM crawler.