How to download dynamic information

gortipavan

Hi

I have a problem at hand and am wondering if RapidMiner can be used to solve it.

I need to get some information from a website. It is a website that reports scientific information for different lakes. I would like to fetch all the data corresponding to a lake attribute (example, Lake depth). This data is loaded on the website dynamically and is not part of the initial markup (The database is read to dynamically populate certain fields). Is tehre a way I can mine this information using Rapid miner? If so, can someone point me in the right direction?

Thanks

el_chief

scraping websites that use javascript can be difficult.

i would recommend looking at Selenium, which lets you use say Chrome browser to fetch a web page and create the markup for you, and return that as text.

http://seleniumhq.org/

you could also try the watij library which is a java library for scraping sites with dynamic content

http://watij.com/

regards,

neil