"Crawl Web with follow_link_with_matching_url returning empty"

alanbontempo
alanbontempo New Altair Community Member
edited November 5 in Community Q&A

Hi, I discovered rapidminer recently and I am impressed with its usability.

I am crowling the website and downloading some information from there (it is in portuguese):

 

http://www.portaldatransparencia.gov.br/servidores/Servidor-ListaServidores.asp?bogus=1&Pagina=1

 

I have a rule that download all pages with the matching regular expression:

 

.+Servidor-DetalhaServidor.+|.+Servidor-DetalhaRemuneracao.+

 

And it is working great. But in this page there is a "next" button that shows more samples. The "next" button send me to the following url:

 

http://www.portaldatransparencia.gov.br/servidores/Servidor-ListaServidores.asp?bogus=1&Pagina=2

 

So I inserted a "follow_link_with_matching_url" rule with the following regular expression:

 

.+Servidor-ListaServidores.+

 

But when I insert this rule I get empty results. Why is this happening?

 

Best Regards

Alan

Best Answer

  • Edin_Klapic
    Edin_Klapic New Altair Community Member
    Answer ✓

    Hi Alan,

     

    as @Thomas_Ott pointed out I highly assume that the problem results in the reason that the content of the pagination is based on JavaScript. Unfortunately, the Web crawler is not capable of accessing this.

    If I get you right you want to extract the data from every page. The website seems quite easy structured so you could e.g. access every page incrementally.

    In this case you could use the Loop operator and set the parameter number of iterations to the maximum number of pages.

    Then you could directly access each page using the URL

    http://www.portaldatransparencia.gov.br/servidores/Servidor-ListaServidores.asp?bogus=1&Pagina=%{iteration}

    In this case the macro iteration reflects the number of the page.

    Please turn off the Parallel execution for the Loop operator, otherwise your IP might get blacklisted.

     

    I hope this gets you nearer to the expected result,

    Edin

     

Answers

  • Thomas_Ott
    Thomas_Ott New Altair Community Member

    Hi, Would you please share the process? Would make it easier to inspect. Thanks!

  • alanbontempo
    alanbontempo New Altair Community Member

    Hi, The process is attached! Tnaks.

  • alanbontempo
    alanbontempo New Altair Community Member

    Hi Thomas,

     

    When you say "process" you mean this xml document:

     

    <?xml version="1.0" encoding="UTF-8"?><process version="7.5.000">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="7.5.000" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
    <operator activated="true" class="web:crawl_web_modern" compatibility="7.3.000" expanded="true" height="68" name="Crawl Web" width="90" x="112" y="34">
    <parameter key="url" value="http://www.portaldatransparencia.gov.br/servidores/Servidor-ListaServidores.asp"/>
    <list key="crawling_rules">
    <parameter key="store_with_matching_url" value=".+Servidor-DetalhaServidor.+|.+Servidor-DetalhaRemuneracao.+"/>
    <parameter key="follow_link_with_matching_url" value=".+Servidor-ListaServidores.+"/>
    </list>
    <parameter key="max_crawl_depth" value="2"/>
    <parameter key="retrieve_as_html" value="true"/>
    <parameter key="enable_basic_auth" value="false"/>
    <parameter key="add_content_as_attribute" value="false"/>
    <parameter key="write_pages_to_disk" value="false"/>
    <parameter key="include_binary_content" value="false"/>
    <parameter key="output_dir" value="D:\Users\alan\Documents\webcrowling"/>
    <parameter key="output_file_extension" value="html"/>
    <parameter key="max_pages" value="1000"/>
    <parameter key="max_page_size" value="1000"/>
    <parameter key="delay" value="200"/>
    <parameter key="max_concurrent_connections" value="100"/>
    <parameter key="max_connections_per_host" value="50"/>
    <parameter key="user_agent" value="rapidminer-web-mining-extension-crawlerMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36"/>
    <parameter key="ignore_robot_exclusion" value="true"/>
    </operator>
    <connect from_op="Crawl Web" from_port="example set" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="0"/>
    </process>
    </operator>
    </process>

    Best Regards,

    Alan

     

  • Thomas_Ott
    Thomas_Ott New Altair Community Member

    Hi, I was speaking with @Edin_Klapic regarding this. It might be related to some Javascript in that page. I believe he might have a workaround. 

  • Edin_Klapic
    Edin_Klapic New Altair Community Member
    Answer ✓

    Hi Alan,

     

    as @Thomas_Ott pointed out I highly assume that the problem results in the reason that the content of the pagination is based on JavaScript. Unfortunately, the Web crawler is not capable of accessing this.

    If I get you right you want to extract the data from every page. The website seems quite easy structured so you could e.g. access every page incrementally.

    In this case you could use the Loop operator and set the parameter number of iterations to the maximum number of pages.

    Then you could directly access each page using the URL

    http://www.portaldatransparencia.gov.br/servidores/Servidor-ListaServidores.asp?bogus=1&amp;Pagina=%{iteration}

    In this case the macro iteration reflects the number of the page.

    Please turn off the Parallel execution for the Loop operator, otherwise your IP might get blacklisted.

     

    I hope this gets you nearer to the expected result,

    Edin

     

  • alanbontempo
    alanbontempo New Altair Community Member

    Hi Edin,

     

    Thank you very much. The for loop worked perfecly.

     

    Best Regards

    Alan

  • alanbontempo
    alanbontempo New Altair Community Member

    Hi @Thomas_Ott

     

    I used the Loop operator but I am facing a very annoying problem. In each loop interation the WebCrawler operator ovewrite the file, once the files are written, for example, 0.tml to 14.html.

     

    Is is possible the WebCrawler give another name for saved files? I am trying to rename the files after the loop but I still did not find a solution.

     

    Best Regards,

    Alan

     

     

  • Edin_Klapic
    Edin_Klapic New Altair Community Member

    Hi Alan,

     

    Let me try a wild guess - You might need to change the filename which you write by combining macros.

    Or try using the macro %{a}. It reflects the number of executions for the corresponding operator.

    In case this does not work, could you please share your process again?

     

    Best,

    Edin

  • Thomas_Ott
    Thomas_Ott New Altair Community Member

    I think you can use a macro value to give them all a unique name. You might try appending the file extension name with txt_%{iteration} or txt_%{t}, the %{t} macro is the system time.