Generate n-grams characters or terms operator?

ikayunida123

Hello everyone!

I'm still new to rapidminer and today I'm going to do a pre-processing for sentiment analysis using rapidminer :catvery-happy:

But I'm confused with the generate n-grams operator. There are two generate n-grams files in rapidminer, generate n-grams (character) and generate n-grams (terms). What's the difference between those two actually? I'm trying to figure it out with read the help section, but I'm still don't understand the difference.

I need your help. Thank you!

 

sgenzer

hello @ikayunida123 - that is a good question. Have you read the KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.

 

I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).

 

<?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="Create Document" width="90" x="45" y="34">
        <parameter key="text" value="This is a sentence to explain n-grams. This is a sentence. This is a character."/>
      </operator>
      <operator activated="true" class="text:tokenize" compatibility="7.5.000" expanded="true" height="68" name="Tokenize" width="90" x="179" y="34"/>
      <operator activated="true" class="multiply" compatibility="8.2.000" expanded="true" height="103" name="Multiply" width="90" x="313" y="34"/>
      <operator activated="true" class="text:generate_n_grams_terms" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Terms)" width="90" x="514" y="136"/>
      <operator activated="true" class="text:generate_n_grams_characters" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Characters)" width="90" x="514" y="34"/>
      <connect from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
      <connect from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
      <connect from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
      <connect from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="105"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Does that help?

 

Scott