"Web Mining crawling prices of an internet page"

Hi Luiz-Vidal,

I came across that issue a few days ago.

Just copy&paste the xml from google wont work due to namespace

Google gives

//*[@id="product_383527"]/div/div[1]/div[3]/div[1]/a/span for the first product: Paulistânia Puro Malte Premium Lager Garrafa 600 ml 1 Unidade and

//*[@id="product_383527"]/div/div[2]/div[1]/div[1]/a/span for the price 14,99

In RM you have to use //*[@id="product_383527"]/h:div/dh:iv[1]/h:div[3]/h:div[1]/h:a/h:span

and //*[@id="product_383527"]/h:div/h:div[2]/h:div[1]/h:div[1]/h:a/h:span

See the discussion here: https://community.rapidminer.com/t5/RapidMiner-Studio-Forum/Extracting-Information-With-XPath/td-p/9883

Cheers

miner

Hey,

Thanks for your reply

Although I still can't make it..

<?xml version="1.0" encoding="UTF-8"?><process version="8.0.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.0.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="false" class="concurrency:loop" compatibility="8.0.001" expanded="true" height="103" name="Loop" width="90" x="179" y="34">
        <parameter key="iteration_macro" value="page"/>
        <process expanded="true">
          <operator activated="true" class="web:crawl_web_modern" compatibility="7.3.000" expanded="true" height="68" name="Crawl Web" width="90" x="246" y="238">
            <parameter key="url" value="https://www.buscape.com.br/cerveja?pagina=%{page}"/>
            <list key="crawling_rules">
              <parameter key="follow_link_with_matching_url" value="cerveja"/>
            </list>
            <parameter key="retrieve_as_html" value="true"/>
            <parameter key="add_content_as_attribute" value="true"/>
            <parameter key="write_pages_to_disk" value="true"/>
            <parameter key="output_dir" value="C:\Users\luiz.vidal\Desktop\Cloudera\SEFA-PA\Cerveja\"/>
          </operator>
          <operator activated="true" class="rename_file" compatibility="8.0.001" expanded="true" height="82" name="Rename File" width="90" x="514" y="238">
            <parameter key="file" value="C:\Users\luiz.vidal\Desktop\Cloudera\SEFA-PA\Cerveja\0.txt"/>
            <parameter key="new_name" value="%{page}.txt"/>
          </operator>
          <operator activated="true" class="generate_macro" compatibility="8.0.001" expanded="true" height="82" name="Generate Macro" width="90" x="112" y="34">
            <list key="function_descriptions">
              <parameter key="page" value="%{page} + 1"/>
            </list>
          </operator>
          <connect from_port="input 1" to_op="Generate Macro" to_port="through 1"/>
          <connect from_op="Crawl Web" from_port="example set" to_op="Rename File" to_port="through 1"/>
          <connect from_op="Rename File" from_port="through 1" to_port="output 2"/>
          <connect from_op="Generate Macro" from_port="through 1" to_port="output 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
          <portSpacing port="sink_output 3" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="text:process_document_from_file" compatibility="8.1.000" expanded="true" height="82" name="Process Documents from Files" width="90" x="112" y="187">
        <list key="text_directories">
          <parameter key="cerveja" value="C:\Users\luiz.vidal\Desktop\Cloudera\SEFA-PA\Cerveja"/>
        </list>
        <parameter key="encoding" value="ISO-8859-1"/>
        <process expanded="true">
          <operator activated="true" class="text:cut_document" compatibility="8.1.000" expanded="true" height="68" name="Cut Document" width="90" x="45" y="34">
            <parameter key="query_type" value="XPath"/>
            <list key="string_machting_queries"/>
            <list key="regular_expression_queries"/>
            <list key="regular_region_queries"/>
            <list key="xpath_queries">
              <parameter key="product" value="//*[@id=&amp;quot;product_383527&quot;]/h:div/dh:iv[1]/h:div[3]/h:div[1]/h:a/h:span"/>
            </list>
            <list key="namespaces"/>
            <parameter key="assume_html" value="false"/>
            <list key="index_queries"/>
            <list key="jsonpath_queries"/>
            <process expanded="true">
              <operator activated="true" class="text:extract_information" compatibility="8.1.000" expanded="true" height="68" name="Extract Information" width="90" x="447" y="85">
                <parameter key="query_type" value="XPath"/>
                <list key="string_machting_queries"/>
                <list key="regular_expression_queries"/>
                <list key="regular_region_queries"/>
                <list key="xpath_queries">
                  <parameter key="product" value="//*[@id=&amp;quot;product_383527&quot;]/h:div/h:div[2]/h:div[1]/h:div[1]/h:a/h:span"/>
                </list>
                <list key="namespaces"/>
                <list key="index_queries"/>
                <list key="jsonpath_queries"/>
              </operator>
              <connect from_port="segment" to_op="Extract Information" to_port="document"/>
              <connect from_op="Extract Information" from_port="document" to_port="document 1"/>
              <portSpacing port="source_segment" spacing="0"/>
              <portSpacing port="sink_document 1" spacing="0"/>
              <portSpacing port="sink_document 2" spacing="0"/>
            </process>
          </operator>
          <connect from_port="document" to_op="Cut Document" to_port="document"/>
          <connect from_op="Cut Document" from_port="documents" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Process Documents from Files" from_port="example set" to_port="result 1"/>
      <connect from_op="Process Documents from Files" from_port="word list" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Any idea what am I doing wrong?

I´m not quite sure.

The website is using product-id for reference.

For the first product I took it was //*[@id="product_383527"] - assuming the id is changing for every product the xpaht is only working for this specific product.

Then you would have to go up the tree to get a "non-id-related" node and then pick the detail from there.

That would be /html/body/main/div[3]/div/div[3]/section?

Sorry,

I know nothing about xpath, I've been trying all day to get ..

<input value="Brahma Pilsen Lata 350 ml 1 Unidade" name="productName" type="hidden">

<input value="6.75" name="priceProduct" type="hidden">

I try, try try and the extract document returs me only true or false or ?

name="productName"], it returns TRUE or FALSE.. but what I want is the value for productName and for priceProduct.. which will probably have to be return on a list.. or a huge string to be split.. I dont know yet.
A victory would be just getting one value returned

Ugh,

After almost giving up I was able to retrieve the piece of data I want, the thing is that it brings only the first that it finds..

I need to find a way to fetch all products names and prices 

//*[@name="priceProduct"]

//*[@name="productName"]

Hi @luiz_vidal

xpath can be a mess...

A good way to test xpath-strings is to use google docs where you can quickly copy the xpath from chrome to the spreadsheet and test the result. This is much faster than testing the structure in RM.

On Youtube you find a lot of tutorials to xpath and google docs.

My recommendation is the video of community member el chief - find it here: https://www.youtube.com/watch?v=UG6223p9fZE

Cheers

miner

Overall,

It was a matter of getting to know how to use xpath and configuring it correctly along the operators.

Thanks for your help

"xpath can be a mess..." 

Definately agree, but it's powerful when it works. 

Help me please. Which Currency is best to mine. https://en.bitcoinwiki.org/wiki/Web_mining here it is written that experts advice "monero".

Hi,

Trying the XPaths in a shell environment can make things faster.

A simple command line tool is XML Shell:

http://www.xmlsh.org/CommandXPath

You can also find the same functionality in Python's scrapy, but it is overkill for your actual needs.

Regards,

Sebastian