Webcrawler download images?

mrbigglesworth
mrbigglesworth New Altair Community Member
edited November 5 in Community Q&A
Hi,

I haven't posted on this forum before but I've been using Rapidminer recently to do webcrawling and run some analysis on the websites. In certain cases, I would like to use rapidminer's webcrawler save the full page rather than just the html. Specifically, the page may have some jpg files, and I'd like to archive those in certain cases. Is there an easy way to do this (other than writing a custom groovy script)?

Thanks!

PS - Thank you very much for the great support and great program. Rapidminer keeps getting better.
Tagged:

Answers

  • colo
    colo New Altair Community Member
    Hi,

    as far as I know the embedded WebSphinx crawler doesn't support storing additional media. But better check the documentation to be sure: http://www.cs.cmu.edu/~rcm/websphinx/

    There was a discussion about replacing this ancient crawler some time ago. I would also be glad if a better crawler was available in RapidMiner, but since there is one that is working certainly other things have priority. Maybe I will integrate another crawler after hopefully finishing my thesis this month, if I start missing the development of RapidMiner add-ons ;)

    Regards
    Matthias

Categories