Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Crawl web operator does not return any results

Hi I am having issues with the simplest web crawler. It does not return any results at all. I have no rules, just a simple web crawl out of the box. What am I doing wrong here?

Find more posts tagged with

AI Studio

Web Mining

Accepted answers

All comments

Amiro

Lab7_WebCrawler.rmp

Amiro

I have attached my rmp file, any help would be appreciated? Many thanks

sgenzer

hi @Amiro - so that website does not have much content and it is a very heavily-stylized site. My guess is that the site is blocking very basic crawlers like "Crawl Web". You're better off using Get Page, grabbing URLs, and so on.

Scott

Telcontar120

Or it could be related to the https problem that came up in the other thread as well. At this point I think Crawl Web operator is not a totally reliable option. Using Get Page within some kind of loop is probably a better alternative.