Loop operator issues

rur68

Hello, everyone
i want to crawl a web with loop operator. the website defines first page is the latest pages. but the information i want to get is the first five page of the newest.
can loop operator in rapidminer do iteration backwards?

rfuentealba

Hello @rur68,

For the sake of simplicity, I'll be using this url instead of the one you provide:

http://jakarta.id/page/%{iteration}/site.

You are iterating numbers 1, 2 and 3. Right?

Well, with number 1, that URL becomes:

http://jakarta.id/page/1/site.

With number 2, that URL becomes:

http://jakarta.id/page/2/site.

With number 3, that URL becomes:

http://jakarta.id/page/1/site.

If you have a known page number (e.g., 500), then you may use the Generate Macro operator, giving a name to the macro generated (like: "calculated_page", with the following code:

<b>500</b> + 1 - eval(%{iteration})

That way, with number 1 you will get:

http://jakarta.id/page/500/site.

With number 2, that URL becomes:

http://jakarta.id/page/499/site.

With number 3, that URL becomes:

http://jakarta.id/page/498/site.

However, that is for a known number. If you are seeking for an unknown number, (e.g., some 1000 new results come every day and you want to crawl those), then you might be out of luck (but people in this community is amazing, they might come up with a solution) and I would recommend you to use something not-so-rapidminer-ish like httrack on UNIX machines (Linux, Mac) to grab an updated copy of the site and use indexes or other tricks under the sleeve to handle these as files.

Word of caution: httrack and other site crawlers might be prohibited in your country, your mileage may vary.

Hope this helps, if I can come up with a better solution, then I'll be back to this thread.

All the best,

Rodrigo.