"Web mining a list of URLs"

adommnich
adommnich New Altair Community Member
edited November 5 in Community Q&A
Hello,

how can I loop over a list of different URLs and extract the content? The operator Crawl Web has no input but the parameter 'URL'. Is there a way to fill it dynamically using a Loop operator?

Thanks for your attention.

Regards
adommnich
Tagged:

Answers

  • colo
    colo New Altair Community Member
    Hi adommnich,

    I would use a "Loop Examples" operator for the URL list (if you read it into an example set before). Inside the loop you can easily extract the current URL with "Extract Macro" and use the value for the "Get Page" operator. If you want to crawl and not just retrieve one website you can replace "Get Page" by "Crawl Web".

    Regards,
    Matthias
  • adommnich
    adommnich New Altair Community Member
    Hi Matthias,

    thanks a lot for your quick reply. I am afraid I wasn' t aware of the Macro concept.  :-[

    It works fine now.

    Regards
    Alex

Categories