Problem: Textmining with specific set of words

ngaved91

Hello everyone :smileywink:

 

I currently face a problem with searching for specific keywords within texts that are stored in a Example Set / Dataset. 

What I did so far:

 

Main Process 1: Selected the attribute "text" from the table --> Data to Documents --> Process Documents ( with a Wordlist as input)

Process 2: (getting the wordlist): Retrieve Example Set --> Selecting the Attribute for the keywords with a filter --> process Documents from Data --> connecting the "wor" output to the "wor" input from "Process Documents"

 

Problem: The results for occurences of the keywords are always the same, even if I change the text files to complete new ones ( I have multiple Example Sets containing multiple text files). The total occurences and Document Occurences are also always the same.

 

Here is the XML code:

<?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="8.2.000" expanded="true" height="68" name="Retrieve Names" width="90" x="45" y="391">
        <parameter key="repository_entry" value="../../Data/Bundesliga Spieler"/>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="8.2.000" expanded="true" height="82" name="Select the Surname" width="90" x="246" y="391">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="Nachname"/>
      </operator>
      <operator activated="true" class="text:process_document_from_data" compatibility="8.1.000" expanded="true" height="82" name="Process Documents from Data" width="90" x="514" y="340">
        <parameter key="select_attributes_and_weights" value="true"/>
        <list key="specify_weights">
          <parameter key="Nachname" value="1.0"/>
        </list>
        <process expanded="true">
          <operator activated="true" class="text:tokenize" compatibility="8.1.000" expanded="true" height="68" name="Tokenize" width="90" x="112" y="34"/>
          <operator activated="true" class="text:filter_by_length" compatibility="8.1.000" expanded="true" height="68" name="Filter Tokens (2)" width="90" x="313" y="34">
            <parameter key="min_chars" value="3"/>
            <parameter key="max_chars" value="100"/>
          </operator>
          <connect from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_op="Filter Tokens (2)" to_port="document"/>
          <connect from_op="Filter Tokens (2)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="retrieve" compatibility="8.2.000" expanded="true" height="68" name="DFL Services" width="90" x="45" y="34">
        <parameter key="repository_entry" value="../../Data/DFL &amp; Agenda"/>
      </operator>
      <operator activated="true" class="retrieve" compatibility="8.2.000" expanded="true" height="68" name="Content Texte" width="90" x="45" y="136">
        <parameter key="repository_entry" value="../../Data/ContentTexte"/>
      </operator>
      <operator activated="true" class="concurrency:join" compatibility="8.2.000" expanded="true" height="82" name="Join" width="90" x="179" y="85">
        <parameter key="use_id_attribute_as_key" value="false"/>
        <list key="key_attributes">
          <parameter key="article_id" value="metadata_file"/>
        </list>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="8.2.000" expanded="true" height="82" name="Select Attributes" width="90" x="380" y="85">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="text"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="text:data_to_documents" compatibility="8.1.000" expanded="true" height="68" name="Data to Documents" width="90" x="581" y="85">
        <parameter key="select_attributes_and_weights" value="true"/>
        <list key="specify_weights">
          <parameter key="text" value="1.0"/>
        </list>
      </operator>
      <operator activated="true" class="text:process_documents" compatibility="8.1.000" expanded="true" height="103" name="Process Documents" width="90" x="916" y="136">
        <parameter key="keep_text" value="true"/>
        <process expanded="true">
          <operator activated="true" class="text:tokenize" compatibility="8.1.000" expanded="true" height="68" name="Tokenize (2)" width="90" x="313" y="85"/>
          <connect from_port="document" to_op="Tokenize (2)" to_port="document"/>
          <connect from_op="Tokenize (2)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve Names" from_port="output" to_op="Select the Surname" to_port="example set input"/>
      <connect from_op="Select the Surname" from_port="example set output" to_op="Process Documents from Data" to_port="example set"/>
      <connect from_op="Process Documents from Data" from_port="word list" to_op="Process Documents" to_port="word list"/>
      <connect from_op="DFL Services" from_port="output" to_op="Join" to_port="left"/>
      <connect from_op="Content Texte" from_port="output" to_op="Join" to_port="right"/>
      <connect from_op="Join" from_port="join" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
      <connect from_op="Data to Documents" from_port="documents" to_op="Process Documents" to_port="documents 1"/>
      <connect from_op="Process Documents" from_port="example set" to_port="result 1"/>
      <connect from_op="Process Documents" from_port="word list" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

I would really appreciate any help!

 

Kind regards

ngaved91

 

ngaved91

I've found the solution. Can someone delete this post. 

 

Thanks!

jczogalla

Hi @ngaved91!

 

Welcome to the community. Instead of deleting this post, would you like to share how you solved the problem? This might help others who stumbled across a similar problem.

Cheers

Jan