Generate n-grams characters or terms operator?
Hello everyone!
I'm still new to rapidminer and today I'm going to do a pre-processing for sentiment analysis using rapidminer :catvery-happy:
But I'm confused with the generate n-grams operator. There are two generate n-grams files in rapidminer, generate n-grams (character) and generate n-grams (terms). What's the difference between those two actually? I'm trying to figure it out with read the help section, but I'm still don't understand the difference.
I need your help. Thank you!
Best Answer
-
hello @ikayunida123 - that is a good question. Have you read the KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.
I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).
<?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="Create Document" width="90" x="45" y="34">
<parameter key="text" value="This is a sentence to explain n-grams. This is a sentence. This is a character."/>
</operator>
<operator activated="true" class="text:tokenize" compatibility="7.5.000" expanded="true" height="68" name="Tokenize" width="90" x="179" y="34"/>
<operator activated="true" class="multiply" compatibility="8.2.000" expanded="true" height="103" name="Multiply" width="90" x="313" y="34"/>
<operator activated="true" class="text:generate_n_grams_terms" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Terms)" width="90" x="514" y="136"/>
<operator activated="true" class="text:generate_n_grams_characters" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Characters)" width="90" x="514" y="34"/>
<connect from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
<connect from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
<connect from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
<connect from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
<connect from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
<connect from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="105"/>
<portSpacing port="sink_result 3" spacing="0"/>
</process>
</operator>
</process>Does that help?
Scott
1
Altair EmployeeAnswers
-
hello @ikayunida123 - that is a good question. Have you read the KnowledgeBase article here on the community on n-grams? It only refers to "term" n-grams, not character, because those are by far the most common.
I built a quick demo process for you that takes one small document, tokenizes it, and then uses the two different n-gram operators to illustrate the difference. Basically one will generate permutations by character; the other generates permutations by token/term (assuming you tokenize by word).
<?xml version="1.0" encoding="UTF-8"?><process version="8.2.000">
<context>
<input/>
<output/>
<macros/>
</context>
<operator activated="true" class="process" compatibility="8.2.000" expanded="true" name="Process">
<process expanded="true">
<operator activated="true" class="text:create_document" compatibility="7.5.000" expanded="true" height="68" name="Create Document" width="90" x="45" y="34">
<parameter key="text" value="This is a sentence to explain n-grams. This is a sentence. This is a character."/>
</operator>
<operator activated="true" class="text:tokenize" compatibility="7.5.000" expanded="true" height="68" name="Tokenize" width="90" x="179" y="34"/>
<operator activated="true" class="multiply" compatibility="8.2.000" expanded="true" height="103" name="Multiply" width="90" x="313" y="34"/>
<operator activated="true" class="text:generate_n_grams_terms" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Terms)" width="90" x="514" y="136"/>
<operator activated="true" class="text:generate_n_grams_characters" compatibility="7.5.000" expanded="true" height="68" name="Generate n-Grams (Characters)" width="90" x="514" y="34"/>
<connect from_op="Create Document" from_port="output" to_op="Tokenize" to_port="document"/>
<connect from_op="Tokenize" from_port="document" to_op="Multiply" to_port="input"/>
<connect from_op="Multiply" from_port="output 1" to_op="Generate n-Grams (Characters)" to_port="document"/>
<connect from_op="Multiply" from_port="output 2" to_op="Generate n-Grams (Terms)" to_port="document"/>
<connect from_op="Generate n-Grams (Terms)" from_port="document" to_port="result 2"/>
<connect from_op="Generate n-Grams (Characters)" from_port="document" to_port="result 1"/>
<portSpacing port="source_input 1" spacing="0"/>
<portSpacing port="sink_result 1" spacing="0"/>
<portSpacing port="sink_result 2" spacing="105"/>
<portSpacing port="sink_result 3" spacing="0"/>
</process>
</operator>
</process>Does that help?
Scott
1 -
It works. Thanks a lot!
0