Can anyone help how Rapid Miner calculates TF-IDF

subhasisdasgupt
subhasisdasgupt New Altair Community Member
edited November 5 in Community Q&A
I know how TF-IDF is to be calculated but when I tried to calculate TF-IDF manually with simple data and tried to match the same with Rapid Miner, I got different results. TF for each word is 0.5 which I also got. But how IDF value calculated is not clear to me. For example, TF of "Cat" is 0.5 but how the TF-IDF value is coming out to be 0.707 is not clear. For TF-IDF to be 0.707, the IDF value should be 0.707/0.5 = 1.414 and how this value is being calculated is the problem that I am facing. Please help.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.3.013">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.3.013" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document" width="90" x="45" y="30">
        <parameter key="text" value="Ram has a dog"/>
      </operator>
      <operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (2)" width="90" x="45" y="300">
        <parameter key="text" value="He has a Rabbit"/>
      </operator>
      <operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (3)" width="90" x="45" y="210">
        <parameter key="text" value="Shivam has a Parrot"/>
      </operator>
      <operator activated="true" class="text:create_document" compatibility="5.3.001" expanded="true" height="60" name="Create Document (4)" width="90" x="45" y="120">
        <parameter key="text" value="Ravi has a Cat"/>
      </operator>
      <operator activated="true" class="text:process_documents" compatibility="5.3.001" expanded="true" height="148" name="Process Documents" width="90" x="179" y="30">
        <process expanded="true">
          <operator activated="true" class="text:tokenize" compatibility="5.3.001" expanded="true" height="60" name="Tokenize" width="90" x="45" y="30"/>
          <connect from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Create Document" from_port="output" to_op="Process Documents" to_port="documents 1"/>
      <connect from_op="Create Document (2)" from_port="output" to_op="Process Documents" to_port="documents 4"/>
      <connect from_op="Create Document (3)" from_port="output" to_op="Process Documents" to_port="documents 3"/>
      <connect from_op="Create Document (4)" from_port="output" to_op="Process Documents" to_port="documents 2"/>
      <connect from_op="Process Documents" from_port="example set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>
Tagged:

Answers

Categories