trivial question on a process (newbie)

lavramu

I created a process that takes in a file from disk (I used ProcessDocumentsFromFiles operator). Within (nested) that I created Tranformcases operator and ReplaceTokens. All the connections were given correctly and through breakpoints I saw that they were performing like I wanted.

Here is the question. After running the entire process..why is that I see the old text (without case transformation and replacement) in the example set in the results section while is see the actual processing of texts through breakpoints.

How do i see the output after all the transformation? When I connect the "wor" to res I seem to see the transformations in the final result. So what is the different between example set and wordlist set? when do you connect which to results?
And my main question is how to I see the transformed text? Even while writing to disk after transformation I see the text without any processing written.

This is the XML :

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="text:process_document_from_file" compatibility="5.3.001" expanded="true" height="76" name="Process Documents from Files" width="90" x="45" y="30">
        <list key="text_directories">
          <parameter key="test" value="C:\Users\Uma\Desktop\nvivo\test"/>
          <parameter key="high" value="C:\Users\Uma\Desktop\nvivo\High"/>
        </list>
        <parameter key="use_file_extension_as_type" value="false"/>
        <parameter key="create_word_vector" value="false"/>
        <parameter key="keep_text" value="true"/>
        <process expanded="true">
          <operator activated="true" breakpoints="after" class="text:transform_cases" compatibility="5.3.001" expanded="true" height="60" name="Transform Cases" width="90" x="45" y="30"/>
          <operator activated="true" breakpoints="after" class="text:replace_tokens" compatibility="5.3.001" expanded="true" height="60" name="Replace Tokens" width="90" x="179" y="30">
            <list key="replace_dictionary">
              <parameter key="reference.*coverage" value=" "/>
              <parameter key="&lt;Internals.*coverage]" value=" "/>
            </list>
          </operator>
          <connect from_port="document" to_op="Transform Cases" to_port="document"/>
          <connect from_op="Transform Cases" from_port="document" to_op="Replace Tokens" to_port="document"/>
          <connect from_op="Replace Tokens" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Process Documents from Files" from_port="example set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

MariusHelf

Hi,

Process Documents is supposed to output the unmodified text, so everything is fine here. Usually the operator is used to calculate word frequencies, not to transform texts.

Since all you want to do is some text modification, you should probably work on the output of Process Documents and transform the text attribute with standard operators. Actually you will need the Replace operator. Here you can use regular expressions to perform everything you are doing currently in Process Documents.

Another alternative in your special case would be to take the wor output and add a Wordlist to Data operator to transform the wordlist to an example set that contains the transformed texts.

Best regards,
Marius