"Cannot retrieve data from the specified URL" error

cbaslan

Hi,

 

I have a process containing a collection of example sets including URLs. I will loop through every example set using loop collection operand. Inside there is a get pages operand and store operand. Total number of websites to be crawled is about 20k from 10 different domains. But when started the process it gives this error and stops. But I know that I have connection and that page causing the error exists. What could be the problem here? Sceenshot is attached. 

 

Thanks

IngoRM

Hard to say.  Could be some Javascript or something to throw it off...

 

In situations like this, where you loop through thousands of sources and might run into trouble with a few of them, I always recommend to use the operator "Handle Exception" as part or actually around the extraction.  This way the rest of the process and the data collection can still finish even if one source throws the system off (or you are offline for 1 second or...)

 

Hope this helps,

Ingo