[SOLVED] SVM and polynomial attributes from word vector

Vaas

Hi guys,

sorry for coming up with a question as my first post, but I am currently working on my bachelor's thesis and I experience major trouble applying the SVM learner on my dataset. I googled a lot and found 2 related posts already submitted to this forum:
http://rapid-i.com/rapidforum/index.php?topic=3845.0
http://rapid-i.com/rapidforum/index.php?topic=524.0
So I obviously did my job but still can't help myself on this one. I'll make it quick:

I basically pull out two values from my DB:

• 'text' - which is just a field containing some text
• 'pol' - which is a polarity label with the possible values 'pro' and 'con'

The text goes out of the DB, into a text preprocessor which tokenizes and filters the usual way. From there, it should be fed into the x-Validator and the SVM should try to learn polarity classification. The label is binominal, so that shouldn't cause any trouble.
What I understand is that the text gets converted into a word vecor (in my case TF-IDF) and the word vector becomes the items attribute - of course a word vector is polynomial so SVM can't handle it. Correct me if I am wrong on any of this. But heres the point: How can I make SVM handle the word vector correctly? If I convert it with the nominal2numerical (which is what most ppl advise, but I actually doubt that step to be useful in my case), SVM brings crappy results (< default, so basically impossible).

Has anyone ever had that before? Any help would be gratefully appreciated! Thanks in advance

Vaas

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
    <process expanded="true" height="670" width="547">
      <operator activated="true" class="read_database" compatibility="5.2.008" expanded="true" height="60" name="ReadDB" width="90" x="45" y="30">
        <parameter key="connection" value="myDB"/>
        <parameter key="query" value="SELECT `text`, `pol`&#38;#10;FROM `myTable`"/>
        <enumeration key="parameters"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="5.2.008" expanded="true" height="76" name="Set Role" width="90" x="45" y="120">
        <parameter key="name" value="pol"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="text:process_document_from_data" compatibility="5.2.004" expanded="true" height="76" name="Process Documents from Data" width="90" x="45" y="255">
        <parameter key="prune_below_absolute" value="2"/>
        <parameter key="prune_above_absolute" value="999"/>
        <list key="specify_weights"/>
        <process expanded="true" height="499" width="675">
          <operator activated="true" class="web:extract_html_text_content" compatibility="5.2.003" expanded="true" height="60" name="Extract Content" width="90" x="45" y="30">
            <parameter key="minimum_text_block_length" value="1"/>
          </operator>
          <operator activated="true" class="text:transform_cases" compatibility="5.2.004" expanded="true" height="60" name="Transform Cases" width="90" x="45" y="120"/>
          <operator activated="true" class="text:tokenize" compatibility="5.2.004" expanded="true" height="60" name="Tokenize" width="90" x="45" y="210"/>
          <operator activated="true" class="text:filter_stopwords_english" compatibility="5.2.004" expanded="true" height="60" name="Filter Stopwords (English)" width="90" x="45" y="300"/>
          <connect from_port="document" to_op="Extract Content" to_port="document"/>
          <connect from_op="Extract Content" from_port="document" to_op="Transform Cases" to_port="document"/>
          <connect from_op="Transform Cases" from_port="document" to_op="Tokenize" to_port="document"/>
          <connect from_op="Tokenize" from_port="document" to_op="Filter Stopwords (English)" to_port="document"/>
          <connect from_op="Filter Stopwords (English)" from_port="document" to_port="document 1"/>
          <portSpacing port="source_document" spacing="0"/>
          <portSpacing port="sink_document 1" spacing="0"/>
          <portSpacing port="sink_document 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="nominal_to_numerical" compatibility="5.2.008" expanded="true" height="94" name="Nominal to Numerical" width="90" x="246" y="210">
        <list key="comparison_groups"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.2.008" expanded="true" height="112" name="Validation" width="90" x="380" y="120">
        <process expanded="true" height="494" width="452">
          <operator activated="true" class="support_vector_machine_libsvm" compatibility="5.2.008" expanded="true" height="76" name="SVM" width="90" x="179" y="300">
            <parameter key="kernel_type" value="linear"/>
            <list key="class_weights"/>
          </operator>
          <connect from_port="training" to_op="SVM" to_port="training set"/>
          <connect from_op="SVM" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="389" width="346">
          <operator activated="true" class="apply_model" compatibility="5.2.008" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance" compatibility="5.2.008" expanded="true" height="76" name="Performance" width="90" x="179" y="120"/>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Mixed" from_port="output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Process Documents from Data" to_port="example set"/>
      <connect from_op="Process Documents from Data" from_port="example set" to_op="Nominal to Numerical" to_port="example set input"/>
      <connect from_op="Nominal to Numerical" from_port="example set output" to_op="Validation" to_port="training"/>
      <connect from_op="Validation" from_port="model" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

MariusHelf

Hi,

as you'll note if you set a breakpoint after the Process Documents, the TF-IDF values generated by Process Documents are numerical - you get one numerical attribute for each token in your input data. So you can apply the SVM directly without further preprocessing. Btw, an SVM with a linear kernel as you have chosen is usually a good means for text classification.

Best regards,
Marius

Vaas

Hi Marius, thanks for helping me out! I aggree with you on the numerical TF-IDF values. Thats exactly why I was confused to see the operation stop before training the SVM with the message

The operator SVM does not have sufficient capabilities for the given data set: polynomial attributes not supported [...]

So why does he refuse the dataset? Sould that be related to the column types I defined in my sql database? I doubt that because the word vector attribute is created by rapidminer and not fetched from the DB... I tried any SVM available with any parameter setup possible. It still refuses to learn (k-NN and naive bayes are working flawlessly btw, neural networks give the same error upon operation as SVM)

Thanks again for the help, I appreciate your efforts!!

Vaas

I found the error. For the record:

The "Extract content"-Operator designed to strip input data from html-tags adds several attributes he tries to extract from so-called "meta-tags" (e.g. <meta name="author" content="John" />, found in almost all documents online), which are - of course - nominal.

And that's the point where my SVM got confused. Setting the breakpoint and looking at what attributes are added was the right hint in this case. Marius, I have to take a bow: Instructions on how to help oneself are almost always the best way to go! Thanks again for everything and have a nice day

vaas

MariusHelf

Thanks for the kudos! I am glad that I could help you