Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Article categorization from web

Hi!
My name is Andrea, i'm trying to to get "all the articles" in the "repubblica.it" home page, then i have to categorize them.
For the first part (access to the articles) i thought it was useful to access the content of <p> tags of the page (www.repubblica.it).
I mean, i chosed to use the operators Crawl Web and Enrich Data by Webservice (to access via XPATH to the meaning content). I setted the Enrich operator with an xpath query (attribute name=Article, query expression =//h:p), but i receive as output a file with the entire page (not the portion i need) as if the xpath query doesn't have effect. Did i choose them wrongly or anything else?
Can someone help me, please?

If possible, i'd like to post here the XML code of the project: can i?

Thanks,
Andrea

Find more posts tagged with

AI Studio

Accepted answers

All comments

Skirzynski

Hey Andrea,

i don't know what is wrong with your query, but i have experimented a little bit with the two extensions "Text Processing" and "Web Mining". I get the front page of your site and extract all links to the news articles, get these pages and extract the content. This isn't the perfect solution, but something to build upon.

Have fun with this
Marcin


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.008" expanded="true" name="Process">
    <process expanded="true" height="505" width="681">
      <operator activated="true" class="subprocess" compatibility="5.1.008" expanded="true" height="76" name="Get news pages" width="90" x="45" y="30">
        <process expanded="true" height="580" width="667">
          <operator activated="true" class="web:get_webpage" compatibility="5.1.001" expanded="true" height="60" name="Get Page" width="90" x="45" y="30">
            <parameter key="url" value="http://www.repubblica.it/"/>
            <parameter key="random_user_agent" value="true"/>
            <list key="query_parameters"/>
          </operator>
          <operator activated="true" class="text:cut_document" compatibility="5.1.002" expanded="true" height="60" name="Cut Document" width="90" x="179" y="30">
            <parameter key="query_type" value="XPath"/>
            <list key="string_machting_queries"/>
            <list key="regular_expression_queries"/>
            <list key="regular_region_queries"/>
            <list key="xpath_queries">
              <parameter key="article-link" value="//h:h3/h:a[contains(@href,'news')]/@href"/&gt;
            </list>
            <list key="namespaces"/>
            <list key="index_queries"/>
            <process expanded="true">
              <connect from_port="segment" to_port="document 1"/>
              <portSpacing port="source_segment" spacing="0"/>
              <portSpacing port="sink_document 1" spacing="0"/>
              <portSpacing port="sink_document 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="text:documents_to_data" compatibility="5.1.002" expanded="true" height="76" name="Documents to Data" width="90" x="313" y="30">
            <parameter key="text_attribute" value="news-links"/>
            <parameter key="add_meta_information" value="false"/>
          </operator>
          <operator activated="true" class="web:retrieve_webpages" compatibility="5.1.001" expanded="true" height="60" name="Get Pages" width="90" x="447" y="30">
            <parameter key="link_attribute" value="news-links"/>
            <parameter key="page_attribute" value="content"/>
            <parameter key="random_user_agent" value="true"/>
          </operator>
          <connect from_op="Get Page" from_port="output" to_op="Cut Document" to_port="document"/>
          <connect from_op="Cut Document" from_port="documents" to_op="Documents to Data" to_port="documents 1"/>
          <connect from_op="Documents to Data" from_port="example set" to_op="Get Pages" to_port="Example Set"/>
          <connect from_op="Get Pages" from_port="Example Set" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="subprocess" compatibility="5.1.008" expanded="true" height="76" name="Extract content" width="90" x="246" y="30">
        <process expanded="true" height="598" width="570">
          <operator activated="true" class="text:data_to_documents" compatibility="5.1.002" expanded="true" height="60" name="Data to Documents" width="90" x="45" y="30">
            <parameter key="select_attributes_and_weights" value="true"/>
            <list key="specify_weights">
              <parameter key="content" value="1.0"/>
            </list>
          </operator>
          <operator activated="true" class="loop_collection" compatibility="5.1.008" expanded="true" height="76" name="Loop Collection" width="90" x="180" y="30">
            <process expanded="true">
              <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.001" expanded="true" name="Extract Content (2)">
                <parameter key="minimum_text_block_length" value="10"/>
              </operator>
              <connect from_port="single" to_op="Extract Content (2)" to_port="document"/>
              <connect from_op="Extract Content (2)" from_port="document" to_port="output 1"/>
              <portSpacing port="source_single" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="text:documents_to_data" compatibility="5.1.002" expanded="true" height="76" name="Documents to Data (2)" width="90" x="315" y="30">
            <parameter key="text_attribute" value="content"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="5.1.008" expanded="true" height="76" name="Select Attributes" width="90" x="450" y="30">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attributes" value="|content|URL"/>
          </operator>
          <connect from_port="in 1" to_op="Data to Documents" to_port="example set"/>
          <connect from_op="Data to Documents" from_port="documents" to_op="Loop Collection" to_port="collection"/>
          <connect from_op="Loop Collection" from_port="output 1" to_op="Documents to Data (2)" to_port="documents 1"/>
          <connect from_op="Documents to Data (2)" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Get news pages" from_port="out 1" to_op="Extract content" to_port="in 1"/>
      <connect from_op="Extract content" from_port="out 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

andrea11

Thank you very much, Parostatek! I'll have fun with this to see how to get what i want!
Thanks! ;D

andrea11

Hi there! I started to try to solve the second part of the job: take a document to classify it in some categories. Now, i've whatched Neil's tutorial http://www.youtube.com/watch?v=BRvjWLwSScQ&feature=player_embedded#!, and i followed his suggestions.
But my case has a difference beetween Neil's one: his data source is a database, while i need to get data from a file.
This is the problem: i've got a file with a text. I want to classify it in a category (like: sport, politic, economy...).

Here's what i produced until now.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.008" expanded="true" name="Process">
    <process expanded="true" height="386" width="413">
      <operator activated="true" class="text:process_document_from_file" compatibility="5.1.002" expanded="true" height="76" name="Process Documents from Files" width="90" x="45" y="30">
        <list key="text_directories">
          <parameter key="ResultDir" value="C:\Users\Andrea\Desktop\Result"/>
        </list>
        <process expanded="true" height="415" width="346">
          <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.002" expanded="true" height="60" name="Extract Content" width="90" x="45" y="30"/>
          <operator activated="true" class="text:transform_cases" compatibility="5.1.002" expanded="true" height="60" name="Transform Cases" width="90" x="45" y="120"/>
          <operator activated="true" class="text:replace_tokens" compatibility="5.1.002" expanded="true" height="60" name="Replace Tokens" width="90" x="45" y="255">
            <list key="replace_dictionary">
              <parameter key="e-mail" value="email"/>
            </list>
          </operator>
          <operator activated="true" class="text:tokenize" compatibility="5.1.002" expanded="true" height="60" name="Tokenize" width="90" x="112" y="345"/>
          <operator activated="true" class="text:filter_stopwords_english" compatibility="5.1.002" expanded="true" height="60" name="Filter Stopwords (English)" width="90" x="246" y="255"/>
          <operator activated="true" class="text:stem_snowball" compatibility="5.1.002" expanded="true" height="60" name="Stem (Snowball)" width="90" x="179" y="120"/>
          <operator activated="true" class="text:filter_by_length" compatibility="5.1.002" expanded="true" height="60" name="Filter Tokens (by Length)" width="90" x="246" y="30">
            <parameter key="min_chars" value="2"/>
            <parameter key="max_chars" value="999"/>
          </operator>
          <connect from_port="document" to_op="Extract Content" to_port="document"/>
          <connect from_op="Extract Content" from_port="document" to_op="Transform Cases" to_port="document"/>
          <connect from_op="Transform Cases" from_port="document" to_op="Replace Tokens" to_port="document"/>
          <connect from_op="Replace Tokens" from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_op="Filter Stopwords (English)" to_port="document"/>
          <connect from_op="Filter Stopwords (English)" from_port="document" to_op="Stem (Snowball)" to_port="document"/>
          <connect from_op="Stem (Snowball)" from_port="document" to_op="Filter Tokens (by Length)" to_port="document"/>
          <connect from_op="Filter Tokens (by Length)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="5.1.008" expanded="true" height="76" name="Select Attributes" width="90" x="45" y="210">
        <parameter key="attribute_filter_type" value="no_missing_values"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="5.1.008" expanded="true" height="76" name="Set Role" width="90" x="179" y="255">
        <parameter key="name" value="label"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.1.008" expanded="true" height="130" name="Validation" width="90" x="246" y="30">
        <parameter key="number_of_validations" value="5"/>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="k_nn" compatibility="5.1.008" expanded="true" height="76" name="k-NN" width="90" x="45" y="120">
            <parameter key="k" value="3"/>
            <parameter key="weighted_vote" value="true"/>
            <parameter key="measure_types" value="NumericalMeasures"/>
            <parameter key="numerical_measure" value="CosineSimilarity"/>
          </operator>
          <connect from_port="training" to_op="k-NN" to_port="training set"/>
          <connect from_op="k-NN" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="apply_model" compatibility="5.1.008" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="5.1.008" expanded="true" height="76" name="Performance" width="90" x="60" y="141"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
          <portSpacing port="sink_averagable 3" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Process Documents from Files" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Validation" to_port="training"/>
      <connect from_op="Validation" from_port="model" to_port="result 1"/>
      <connect from_op="Validation" from_port="training" to_port="result 2"/>
      <connect from_op="Validation" from_port="averagable 1" to_port="result 3"/>
      <connect from_op="Validation" from_port="averagable 2" to_port="result 4"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
    </process>
  </operator>
</process>

I'm sure i've got problems with "labels", as you can see from the error i get; but i think it's not my only problem ;D in fact, i don't know how to set up the training set.

Please, can someone help me?

andrea11

Anyone can help me? Need to solve it...

Thanks...!

colo

Hi Andrea,

the labels are indeed missing in your case. If you use the "Process Documents from Files" operator, each class name used for a directory to import files from is used as label for those files. This makes sense, if you put the documents from different categories into different directories and add all of them to the text directories parameter of the operator. But you are importing from only one directory, so there exists only one category. What should the classifier learn in this case? If there are no categories to distinguish, classifying doesn't make much sense.
You have to assign the categories as label/class to each document in your training set, otherwise there is nothing to learn. Using different directories for the classes is certainly the easiest way to achieve this.

Regards
Matthias

andrea11

Thanks man, i fixed it!
But where do i put the file with the text to categorize?
As you said i divided into folders the texts for the training. Now i want to use the system (trained) to categorize a text. I don't know where to put it.

Thanks in advance!

colo

Hi Andrea,

you can either import the files in the same way you use for training (if you have some pre-categorized test data) or import unknown documents from only one directory using any label you want for "Process Documents from Files". You have to assign the same preprocessing steps to the data as you did before training the model (the existing attributes have to be the same). Then you should be able to use "Apply Model" and get some predictions.

If the necessary steps aren't clear to you, consider providing your process XML to show the current progress.

Regards
Matthias

andrea11

Man, i'm sorry but i didn't understand very well... Here's my XML:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.008" expanded="true" name="Process">
    <process expanded="true" height="386" width="413">
      <operator activated="true" class="text:process_document_from_file" compatibility="5.1.002" expanded="true" height="76" name="Process Documents from Files" width="90" x="45" y="30">
        <list key="text_directories">
          <parameter key="EconomyAndPolitic" value="C:\Users\Andrea\Desktop\Result\EconomyAndPolitic"/>
          <parameter key="Sport" value="C:\Users\Andrea\Desktop\Result\Sport"/>
        </list>
        <parameter key="use_file_extension_as_type" value="false"/>
        <parameter key="encoding" value="UTF-8"/>
        <process expanded="true" height="415" width="346">
          <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.002" expanded="true" height="60" name="Extract Content" width="90" x="45" y="30"/>
          <operator activated="true" class="text:transform_cases" compatibility="5.1.002" expanded="true" height="60" name="Transform Cases" width="90" x="45" y="120"/>
          <operator activated="true" class="text:replace_tokens" compatibility="5.1.002" expanded="true" height="60" name="Replace Tokens" width="90" x="45" y="255">
            <list key="replace_dictionary">
              <parameter key="e-mail" value="email"/>
            </list>
          </operator>
          <operator activated="true" class="text:tokenize" compatibility="5.1.002" expanded="true" height="60" name="Tokenize" width="90" x="179" y="345"/>
          <operator activated="true" class="text:filter_stopwords_english" compatibility="5.1.002" expanded="true" height="60" name="Filter Stopwords (English)" width="90" x="246" y="255"/>
          <operator activated="true" class="text:stem_snowball" compatibility="5.1.002" expanded="true" height="60" name="Stem (Snowball)" width="90" x="179" y="120"/>
          <operator activated="true" class="text:filter_by_length" compatibility="5.1.002" expanded="true" height="60" name="Filter Tokens (by Length)" width="90" x="246" y="30">
            <parameter key="min_chars" value="2"/>
            <parameter key="max_chars" value="999"/>
          </operator>
          <connect from_port="document" to_op="Extract Content" to_port="document"/>
          <connect from_op="Extract Content" from_port="document" to_op="Transform Cases" to_port="document"/>
          <connect from_op="Transform Cases" from_port="document" to_op="Replace Tokens" to_port="document"/>
          <connect from_op="Replace Tokens" from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_op="Filter Stopwords (English)" to_port="document"/>
          <connect from_op="Filter Stopwords (English)" from_port="document" to_op="Stem (Snowball)" to_port="document"/>
          <connect from_op="Stem (Snowball)" from_port="document" to_op="Filter Tokens (by Length)" to_port="document"/>
          <connect from_op="Filter Tokens (by Length)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="5.1.008" expanded="true" height="76" name="Select Attributes" width="90" x="45" y="210">
        <parameter key="attribute_filter_type" value="no_missing_values"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="5.1.008" expanded="true" height="76" name="Set Role" width="90" x="179" y="255">
        <parameter key="name" value="label"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.1.008" expanded="true" height="130" name="Validation" width="90" x="246" y="30">
        <parameter key="number_of_validations" value="2"/>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="k_nn" compatibility="5.1.008" expanded="true" height="76" name="k-NN" width="90" x="45" y="75">
            <parameter key="k" value="2"/>
            <parameter key="weighted_vote" value="true"/>
            <parameter key="measure_types" value="NumericalMeasures"/>
            <parameter key="numerical_measure" value="CosineSimilarity"/>
          </operator>
          <connect from_port="training" to_op="k-NN" to_port="training set"/>
          <connect from_op="k-NN" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="apply_model" compatibility="5.1.008" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
            <parameter key="create_view" value="true"/>
          </operator>
          <operator activated="true" class="performance" compatibility="5.1.008" expanded="true" height="76" name="Performance" width="90" x="60" y="141"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
          <portSpacing port="sink_averagable 3" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Process Documents from Files" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Validation" to_port="training"/>
      <connect from_op="Validation" from_port="model" to_port="result 1"/>
      <connect from_op="Validation" from_port="training" to_port="result 2"/>
      <connect from_op="Validation" from_port="averagable 1" to_port="result 3"/>
      <connect from_op="Validation" from_port="averagable 2" to_port="result 4"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
    </process>
  </operator>
</process>

I thought to create a txt file with the text to categorize, but i don't know where to put it. I can't put it neither in economyandpolitic folder, nor in sport one. ???
Thanks!

colo

Hi Andrea,

you have to add something like the operator chain I added to the bottom of your process. Hope this gives you some hints on how to classify new texts.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.009">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.009" expanded="true" name="Process">
    <process expanded="true" height="627" width="614">
      <operator activated="true" class="text:process_document_from_file" compatibility="5.1.001" expanded="true" height="76" name="Process Documents from Files" width="90" x="45" y="30">
        <list key="text_directories">
          <parameter key="EconomyAndPolitic" value="C:\Users\Andrea\Desktop\Result\EconomyAndPolitic"/>
          <parameter key="Sport" value="C:\Users\Andrea\Desktop\Result\Sport"/>
        </list>
        <parameter key="use_file_extension_as_type" value="false"/>
        <parameter key="encoding" value="UTF-8"/>
        <process expanded="true" height="415" width="346">
          <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.002" expanded="true" height="60" name="Extract Content" width="90" x="45" y="30"/>
          <operator activated="true" class="text:transform_cases" compatibility="5.1.001" expanded="true" height="60" name="Transform Cases" width="90" x="45" y="120"/>
          <operator activated="true" class="text:replace_tokens" compatibility="5.1.001" expanded="true" height="60" name="Replace Tokens" width="90" x="45" y="255">
            <list key="replace_dictionary">
              <parameter key="e-mail" value="email"/>
            </list>
          </operator>
          <operator activated="true" class="text:tokenize" compatibility="5.1.001" expanded="true" height="60" name="Tokenize" width="90" x="179" y="345"/>
          <operator activated="true" class="text:filter_stopwords_english" compatibility="5.1.001" expanded="true" height="60" name="Filter Stopwords (English)" width="90" x="246" y="255"/>
          <operator activated="true" class="text:stem_snowball" compatibility="5.1.001" expanded="true" height="60" name="Stem (Snowball)" width="90" x="179" y="120"/>
          <operator activated="true" class="text:filter_by_length" compatibility="5.1.001" expanded="true" height="60" name="Filter Tokens (by Length)" width="90" x="246" y="30">
            <parameter key="min_chars" value="2"/>
            <parameter key="max_chars" value="999"/>
          </operator>
          <connect from_port="document" to_op="Extract Content" to_port="document"/>
          <connect from_op="Extract Content" from_port="document" to_op="Transform Cases" to_port="document"/>
          <connect from_op="Transform Cases" from_port="document" to_op="Replace Tokens" to_port="document"/>
          <connect from_op="Replace Tokens" from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_op="Filter Stopwords (English)" to_port="document"/>
          <connect from_op="Filter Stopwords (English)" from_port="document" to_op="Stem (Snowball)" to_port="document"/>
          <connect from_op="Stem (Snowball)" from_port="document" to_op="Filter Tokens (by Length)" to_port="document"/>
          <connect from_op="Filter Tokens (by Length)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="5.1.009" expanded="true" height="76" name="Select Attributes" width="90" x="45" y="210">
        <parameter key="attribute_filter_type" value="no_missing_values"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="5.1.009" expanded="true" height="76" name="Set Role" width="90" x="179" y="210">
        <parameter key="name" value="label"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.1.009" expanded="true" height="130" name="Validation" width="90" x="246" y="30">
        <parameter key="number_of_validations" value="2"/>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="k_nn" compatibility="5.1.009" expanded="true" height="76" name="k-NN" width="90" x="45" y="75">
            <parameter key="k" value="2"/>
            <parameter key="weighted_vote" value="true"/>
            <parameter key="measure_types" value="NumericalMeasures"/>
            <parameter key="numerical_measure" value="CosineSimilarity"/>
          </operator>
          <connect from_port="training" to_op="k-NN" to_port="training set"/>
          <connect from_op="k-NN" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="347" width="145">
          <operator activated="true" class="apply_model" compatibility="5.1.009" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
            <parameter key="create_view" value="true"/>
          </operator>
          <operator activated="true" class="performance" compatibility="5.1.009" expanded="true" height="76" name="Performance" width="90" x="60" y="141"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
          <portSpacing port="sink_averagable 3" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="text:process_document_from_file" compatibility="5.1.001" expanded="true" height="76" name="Process Documents from Files (2)" width="90" x="45" y="390">
        <list key="text_directories">
          <parameter key="unclassified" value="C:\Users\Andrea\Desktop\Result\TestFiles"/>
        </list>
        <parameter key="use_file_extension_as_type" value="false"/>
        <parameter key="encoding" value="UTF-8"/>
        <process expanded="true" height="607" width="763">
          <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.002" expanded="true" height="60" name="Extract Content (2)" width="90" x="45" y="30"/>
          <operator activated="true" class="text:transform_cases" compatibility="5.1.001" expanded="true" height="60" name="Transform Cases (2)" width="90" x="180" y="30"/>
          <operator activated="true" class="text:replace_tokens" compatibility="5.1.001" expanded="true" height="60" name="Replace Tokens (2)" width="90" x="315" y="30">
            <list key="replace_dictionary">
              <parameter key="e-mail" value="email"/>
            </list>
          </operator>
          <operator activated="true" class="text:tokenize" compatibility="5.1.001" expanded="true" height="60" name="Tokenize (2)" width="90" x="450" y="30"/>
          <operator activated="true" class="text:filter_stopwords_english" compatibility="5.1.001" expanded="true" height="60" name="Filter Stopwords (2)" width="90" x="585" y="30"/>
          <operator activated="true" class="text:stem_snowball" compatibility="5.1.001" expanded="true" height="60" name="Stem (2)" width="90" x="45" y="120"/>
          <operator activated="true" class="text:filter_by_length" compatibility="5.1.001" expanded="true" height="60" name="Filter Tokens (2)" width="90" x="380" y="75">
            <parameter key="min_chars" value="2"/>
            <parameter key="max_chars" value="999"/>
          </operator>
          <connect from_port="document" to_op="Extract Content (2)" to_port="document"/>
          <connect from_op="Extract Content (2)" from_port="document" to_op="Transform Cases (2)" to_port="document"/>
          <connect from_op="Transform Cases (2)" from_port="document" to_op="Replace Tokens (2)" to_port="document"/>
          <connect from_op="Replace Tokens (2)" from_port="document" to_op="Tokenize (2)" to_port="document"/>
          <connect from_op="Tokenize (2)" from_port="document" to_op="Filter Stopwords (2)" to_port="document"/>
          <connect from_op="Filter Stopwords (2)" from_port="document" to_op="Stem (2)" to_port="document"/>
          <connect from_op="Stem (2)" from_port="document" to_op="Filter Tokens (2)" to_port="document"/>
          <connect from_op="Filter Tokens (2)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="5.1.009" expanded="true" height="76" name="Select Attributes (2)" width="90" x="179" y="390">
        <parameter key="attribute_filter_type" value="no_missing_values"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="apply_model" compatibility="5.1.009" expanded="true" height="76" name="Apply Model (2)" width="90" x="447" y="210">
        <list key="application_parameters"/>
      </operator>
      <connect from_op="Process Documents from Files" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Validation" to_port="training"/>
      <connect from_op="Validation" from_port="model" to_op="Apply Model (2)" to_port="model"/>
      <connect from_op="Validation" from_port="training" to_port="result 1"/>
      <connect from_op="Validation" from_port="averagable 1" to_port="result 2"/>
      <connect from_op="Validation" from_port="averagable 2" to_port="result 3"/>
      <connect from_op="Process Documents from Files (2)" from_port="example set" to_op="Select Attributes (2)" to_port="example set input"/>
      <connect from_op="Select Attributes (2)" from_port="example set output" to_op="Apply Model (2)" to_port="unlabelled data"/>
      <connect from_op="Apply Model (2)" from_port="labelled data" to_port="result 4"/>
      <connect from_op="Apply Model (2)" from_port="model" to_port="result 5"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="126"/>
      <portSpacing port="sink_result 5" spacing="0"/>
      <portSpacing port="sink_result 6" spacing="0"/>
    </process>
  </operator>
</process>

Regards
Matthias

andrea11

Thank you colo!! Really helped! I'll work on it!!

andrea11

Men, need your help once more. Be patient

I'm taking from "www.repubblica.it" all the articles/link couples (thanks to Parostatek

).
Now what i need is to save each article in a format like this: "value_of_the_attribute_URL.txt" (in the file there should be the value of the "content" attribute.

I'd like to specify the output folder, where the process can create the files.

Here's what i've done right now:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.1.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.1.008" expanded="true" name="Process">
    <process expanded="true" height="505" width="681">
      <operator activated="true" class="subprocess" compatibility="5.1.008" expanded="true" height="76" name="Get news pages" width="90" x="45" y="120">
        <process expanded="true" height="580" width="667">
          <operator activated="true" class="web:get_webpage" compatibility="5.1.002" expanded="true" height="60" name="Get Page" width="90" x="45" y="30">
            <parameter key="url" value="http://www.repubblica.it"/>
            <parameter key="random_user_agent" value="true"/>
            <list key="query_parameters"/>
          </operator>
          <operator activated="true" class="text:cut_document" compatibility="5.1.002" expanded="true" height="60" name="Cut Document" width="90" x="112" y="120">
            <parameter key="query_type" value="XPath"/>
            <list key="string_machting_queries"/>
            <list key="regular_expression_queries"/>
            <list key="regular_region_queries"/>
            <list key="xpath_queries">
              <parameter key="article-link" value="//h:h3/h:a[contains(@href,'news')]/@href"/&gt;
            </list>
            <list key="namespaces"/>
            <list key="index_queries"/>
            <process expanded="true" height="355" width="497">
              <connect from_port="segment" to_port="document 1"/>
              <portSpacing port="source_segment" spacing="0"/>
              <portSpacing port="sink_document 1" spacing="0"/>
              <portSpacing port="sink_document 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="text:documents_to_data" compatibility="5.1.002" expanded="true" height="76" name="Documents to Data" width="90" x="246" y="75">
            <parameter key="text_attribute" value="news-links"/>
            <parameter key="add_meta_information" value="false"/>
          </operator>
          <operator activated="true" class="web:retrieve_webpages" compatibility="5.1.002" expanded="true" height="60" name="Get Pages" width="90" x="380" y="165">
            <parameter key="link_attribute" value="news-links"/>
            <parameter key="page_attribute" value="content"/>
            <parameter key="random_user_agent" value="true"/>
          </operator>
          <connect from_op="Get Page" from_port="output" to_op="Cut Document" to_port="document"/>
          <connect from_op="Cut Document" from_port="documents" to_op="Documents to Data" to_port="documents 1"/>
          <connect from_op="Documents to Data" from_port="example set" to_op="Get Pages" to_port="Example Set"/>
          <connect from_op="Get Pages" from_port="Example Set" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="subprocess" compatibility="5.1.008" expanded="true" height="112" name="Extract content" width="90" x="246" y="120">
        <process expanded="true" height="598" width="614">
          <operator activated="true" class="text:data_to_documents" compatibility="5.1.002" expanded="true" height="60" name="Data to Documents" width="90" x="45" y="30">
            <parameter key="select_attributes_and_weights" value="true"/>
            <list key="specify_weights">
              <parameter key="content" value="1.0"/>
            </list>
          </operator>
          <operator activated="true" class="loop_collection" compatibility="5.1.008" expanded="true" height="76" name="Loop Collection" width="90" x="112" y="210">
            <process expanded="true" height="373" width="602">
              <operator activated="true" class="web:extract_html_text_content" compatibility="5.1.002" expanded="true" height="60" name="Extract Content (2)" width="90" x="265" y="30">
                <parameter key="minimum_text_block_length" value="10"/>
              </operator>
              <connect from_port="single" to_op="Extract Content (2)" to_port="document"/>
              <connect from_op="Extract Content (2)" from_port="document" to_port="output 1"/>
              <portSpacing port="source_single" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="text:documents_to_data" compatibility="5.1.002" expanded="true" height="76" name="Documents to Data (2)" width="90" x="179" y="30">
            <parameter key="text_attribute" value="content"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="5.1.008" expanded="true" height="76" name="Select Attributes" width="90" x="313" y="165">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attributes" value="|content"/>
          </operator>
          <operator activated="true" class="loop_examples" compatibility="5.1.008" expanded="true" height="94" name="Loop Examples" width="90" x="479" y="217">
            <parameter key="iteration_macro" value="testo"/>
            <process expanded="true" height="367" width="700">
              <operator activated="true" class="write_special" compatibility="5.1.008" expanded="true" height="60" name="Write Special Format" width="90" x="279" y="178">
                <parameter key="example_set_file" value="C:\Users\Andrea\Desktop\Result\Articles.txt"/>
                <parameter key="special_format" value="$a"/>
              </operator>
              <connect from_port="example set" to_op="Write Special Format" to_port="input"/>
              <connect from_op="Write Special Format" from_port="through" to_port="example set"/>
              <portSpacing port="source_example set" spacing="0"/>
              <portSpacing port="sink_example set" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <connect from_port="in 1" to_op="Data to Documents" to_port="example set"/>
          <connect from_op="Data to Documents" from_port="documents" to_op="Loop Collection" to_port="collection"/>
          <connect from_op="Loop Collection" from_port="output 1" to_op="Documents to Data (2)" to_port="documents 1"/>
          <connect from_op="Documents to Data (2)" from_port="example set" to_op="Select Attributes" to_port="example set input"/>
          <connect from_op="Select Attributes" from_port="example set output" to_port="out 1"/>
          <connect from_op="Select Attributes" from_port="original" to_op="Loop Examples" to_port="example set"/>
          <connect from_op="Loop Examples" from_port="example set" to_port="out 2"/>
          <connect from_op="Loop Examples" from_port="output 1" to_port="out 3"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Get news pages" from_port="out 1" to_op="Extract content" to_port="in 1"/>
      <connect from_op="Extract content" from_port="out 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

As you can see, now i'm creating ONE file called Articles.txt where i save all the articles. I need to fix this, as i explained above.
Really thanks for your help!

colo

Hi Andrea,

inside "Loop Examples" you could do something like the solution described here: http://rapid-i.com/rapidforum/index.php/topic,4055.0.html
You can extract the URL you want to use as file name in a similar way using macros. But you have to ensure that there aren't symbols inside the URL string that are invalid within filenames.

I think you might simplify the chain "Data to Documents" - "Loop Collection" - "Documents to Data" by using a single "Process Documents from Data" instead. This will not affect functionality but require two operators less.

Regards
Matthias

andrea11

colo wrote:

Hi Andrea,

inside "Loop Examples" you could do something like the solution described here: http://rapid-i.com/rapidforum/index.php/topic,4055.0.html
You can extract the URL you want to use as file name in a similar way using macros. But you have to ensure that there aren't symbols inside the URL string that are invalid within filenames.

I think you might simplify the chain "Data to Documents" - "Loop Collection" - "Documents to Data" by using a single "Process Documents from Data" instead. This will not affect functionality but require two operators less.

Regards
Matthias

Thanks! I adapted that solution with my needs! Thanks a lot!