Generate n-grams characters or terms operator?

ikayunida123
ikayunida123 New Altair Community Member
edited November 5 in Community Q&A

Hello everyone!

I'm still new to rapidminer and today I'm going to do a pre-processing for sentiment analysis using rapidminer :catvery-happy:

But I'm confused with the generate n-grams operator. There are two generate n-grams files in rapidminer, generate n-grams (character) and generate n-grams (terms). What's the difference between those two actually? I'm trying to figure it out with read the help section, but I'm still don't understand the difference.

I need your help. Thank you!

 

Best Answer

  • sgenzer
    sgenzer
    Altair Employee
    Answer ✓

    hello @ikayunida123 - that is a good question. Have you read the KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.

     

    I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).

     

    <?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="Create Document" width="90" x="45" y="34">
    <parameter key="text" value="This is a sentence to explain n-grams. This is a sentence. This is a character."/>
    </operator>
    <operator activated="true" class="text:tokenize" compatibility="7.5.000" expanded="true" height="68" name="Tokenize" width="90" x="179" y="34"/>
    <operator activated="true" class="multiply" compatibility="8.2.000" expanded="true" height="103" name="Multiply" width="90" x="313" y="34"/>
    <operator activated="true" class="text:generate_n_grams_terms" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Terms)" width="90" x="514" y="136"/>
    <operator activated="true" class="text:generate_n_grams_characters" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Characters)" width="90" x="514" y="34"/>
    <connect from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
    <connect from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
    <connect from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
    <connect from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
    <connect from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
    <connect from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="105"/>
    <portSpacing port="sink_result 3" spacing="0"/>
    </process>
    </operator>
    </process>

    Does that help?

     

    Scott

     

Answers

  • sgenzer
    sgenzer
    Altair Employee
    Answer ✓

    hello @ikayunida123 - that is a good question. Have you read the KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.

     

    I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).

     

    <?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
    <context>
    <input/>
    <output/>
    <macros/>
    </context>
    <operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
    <process expanded="true">
    <operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="Create Document" width="90" x="45" y="34">
    <parameter key="text" value="This is a sentence to explain n-grams. This is a sentence. This is a character."/>
    </operator>
    <operator activated="true" class="text:tokenize" compatibility="7.5.000" expanded="true" height="68" name="Tokenize" width="90" x="179" y="34"/>
    <operator activated="true" class="multiply" compatibility="8.2.000" expanded="true" height="103" name="Multiply" width="90" x="313" y="34"/>
    <operator activated="true" class="text:generate_n_grams_terms" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Terms)" width="90" x="514" y="136"/>
    <operator activated="true" class="text:generate_n_grams_characters" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Characters)" width="90" x="514" y="34"/>
    <connect from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
    <connect from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
    <connect from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
    <connect from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
    <connect from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
    <connect from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>
    <portSpacing port="source_input 1" spacing="0"/>
    <portSpacing port="sink_result 1" spacing="0"/>
    <portSpacing port="sink_result 2" spacing="105"/>
    <portSpacing port="sink_result 3" spacing="0"/>
    </process>
    </operator>
    </process>

    Does that help?

     

    Scott

     

  • ikayunida123
    ikayunida123 New Altair Community Member

    It works. Thanks a lot!