"SVM (libSVM) Missing Function Value"

B_Miner

Hi guys,

Is it possible to get the decision function value from your implementation of libSVM?
There is a field called 'function value' in the support vector table of the model but in the case I show below, it is all zero. Maybe this is not it though as we'd expect the function to be defined for all cases, not just the support vectors.


The decision function is:

f(x)= sgn(sum_over_all_i {a*yk(x_i,x) } ) where a* is the optimized Lagrangian multipliers, y is the observed target value and k() is the kernel).

Thanks!

Data and Code are below:


I used a small data set:

x1	x2	class
0	0.7	One
0.7	0	One
0	-0.7	One
-0.7	0	One
0.5	0.5	One
-0.5	0.5	One
-0.5	-0.5	One
0.5	-0.5	One
0	2.8	Two
-2.8	0	Two
0	-2.8	Two
2.8	0	Two
2	2	Two
-2	2	Two
-2	-2	Two
2	-2	Two

Here is the code I ran:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.0">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" expanded="true" name="Process">
    <process expanded="true" height="341" width="547">
      <operator activated="true" class="retrieve" expanded="true" height="60" name="Retrieve" width="90" x="45" y="75">
        <parameter key="repository_entry" value="//Default/svmTest"/>
      </operator>
      <operator activated="true" class="nominal_to_binominal" expanded="true" height="94" name="Nominal to Binominal" width="90" x="246" y="30">
        <parameter key="attribute_filter_type" value="single"/>
        <parameter key="attribute" value="class"/>
        <parameter key="include_special_attributes" value="true"/>
      </operator>
      <operator activated="true" class="support_vector_machine_libsvm" expanded="true" height="76" name="SVM (2)" width="90" x="45" y="210">
        <parameter key="kernel_type" value="poly"/>
        <parameter key="degree" value="2"/>
        <parameter key="gamma" value="1.0"/>
        <parameter key="C" value="10.0"/>
        <list key="class_weights"/>
        <parameter key="confidence_for_multiclass" value="false"/>
      </operator>
      <operator activated="true" class="apply_model" expanded="true" height="76" name="Apply Model" width="90" x="246" y="255">
        <list key="application_parameters"/>
      </operator>
      <operator activated="true" class="performance_classification" expanded="true" height="76" name="Performance" width="90" x="380" y="165">
        <list key="class_weights"/>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Nominal to Binominal" to_port="example set input"/>
      <connect from_op="Nominal to Binominal" from_port="example set output" to_op="SVM (2)" to_port="training set"/>
      <connect from_op="SVM (2)" from_port="model" to_op="Apply Model" to_port="model"/>
      <connect from_op="SVM (2)" from_port="exampleSet" to_op="Apply Model" to_port="unlabelled data"/>
      <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
      <connect from_op="Apply Model" from_port="model" to_port="result 1"/>
      <connect from_op="Performance" from_port="performance" to_port="result 2"/>
      <connect from_op="Performance" from_port="example set" to_port="result 3"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
    </process>
  </operator>

land

Hi,
isn't that exactly the outcome the model applier will calculate if you apply the model?

Greetings,
  Sebastian

B_Miner

Oops! Yes, that is the decision isnt it. I meant to leave off the sgn and just ask about getting access to what is inside:

sum_over_all_i {a*yk(x_i,x) }

I know that libsvm will give this information.

IngoRM

Hi,

isn't this exactly the thing the "Create Formula" operator performs?

Cheers,
Ingo

B_Miner

Perhaps, I was not famiiliar with this operator. Trying it out, it gives a strange result for the sample I posted:

which is not correct (the result is close but does not match libsvm or hand calculations).

The "correct answer" for the vector [2.8,0] is -1.0001512 where the below code will give 1.145649441

So, I'm not sure....

  0.0 * (pow((1.0 * (-0.5 * x1 - 0.5 * x2) + 0.0), 2)) + 0.0 * (pow((1.0 * (0.5 * x1 - 0.5 * x2) + 0.0), 2)) + 0.037111961591350995 * (pow((1.0 * (-2.8 * x2) + 0.0), 2)) + 0.037121838508423564 * (pow((1.0 * (2.8 * x1) + 0.0), 2)) - 1.136066635903084

IngoRM

Hi,

the flipped sign could simply derive from the fact which class (One or Two) is internally mapped to -1 and +1. And the derivation between 1.000... and 1.145... could derive from different parameters, rescaling, shuffled data, ...  I would have to check the source code of LibSVM, of RapidMiner, your hand calculations and compare everything to come up with a definite answer.

Cheers,
Ingo