Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

"How are L pos and L neg set in an SVM?"

Noob here. I am working on a basic classifcation problem (responders vs non-responders), where the cost of missing a responder is much higher than the cost of including a non-responder. Specifically, the average sale is $115 and the average cost to mail somebody is $0.68. We want to bias in favor of maximizing sales over minimizing cost to mail.

I can't find any documentation anywhere that shows how the "L pos" and "L neg" settings in an SVM are actually set. Are they numbers between 0 and 1? Are they relative weights for mis-classifying a positive "L pos" and a negative "L neg"? In my case, I would interpret that as setting "L pos" much higher than "L neg", since I want to ensure that we avoid as many false negatives as possible.

Can somebody help me understand how this works?

Thanks in advance...

Eric

Find more posts tagged with

AI Studio

SVMs

Accepted answers

All comments

lel5287

I know this is a few months old, but I'm working on a very similar problem, so if anyone has any insight on this, that would be greatly appreciated.

Additionally, I'm still exploring classification methods for what I'm working on. Is it possible to set similar loss criteria in other classification methods? (False positives aren't too big of a deal, but false negatives are very costly--also, the test set includes very few responders compared to the number of non-responders, so a lot of methods that just look to improve accuracy without weighing false positives against false negatives classify almost everyone as a non-responder.)

MartinLiebig

Hi,

i don't know what it is but i got eager to find it out.

So apprently it is used in /src/main/java/com/rapidminer/operator/learner/functions/kernel/jmysvm/svm/SVM.java

I figured out there that the balance cost option sets the same values.

				double generalCpos = paramOperator.getParameterAsDouble(JMySVMLearner.PARAMETER_L_POS);
				double generalCneg = paramOperator.getParameterAsDouble(JMySVMLearner.PARAMETER_L_NEG);
				if (paramOperator.getParameterAsBoolean(JMySVMLearner.PARAMETER_BALANCE_COST)) {
					generalCpos *= the_examples.count_examples()
							/ (2.0d * (the_examples.count_examples() - the_examples.count_pos_examples()));
					generalCneg *= ((the_examples.count_examples()) / (2.0d * the_examples.count_pos_examples()));
				}
				for (int i = 0; i < cPos.length; i++) {
					cPos = generalCpos;
					cNeg = generalCneg;

So i did a quick test (process XML below). I classified golf with and without balance cost. Further i added weights to ensure class balance this way.

Weights w/o balance cost and w/o class balancing


Temperature	-0.32830007919309123
Outlook = sunny	-0.2990901852444362
Humidity	-0.2759359389650189
Wind = true	-0.23094010767584963
Outlook = rain	0.0
Wind = false	0.17320508075688723
Outlook = overcast	0.35248116362113585

w balance cost but w/o class balancing


Outlook = sunny	-0.33391948463996746
Humidity	-0.2776748983378322
Wind = true	-0.25278262899833226
Outlook = rain	-0.12521980673998773
Temperature	-0.060810818269701084
Wind = false	0.18958697174874925
Outlook = overcast	0.5411008440732543

and finallyw/ scaling but w/o balancing


Outlook = sunny	-0.33391948463996746
Humidity	-0.2776748983378322
Wind = true	-0.25278262899833226
Outlook = rain	-0.12521980673998773
Temperature	-0.060810818269701084
Wind = false	0.18958697174874925
Outlook = overcast	0.5411008440732543

So apprently the balancing here does the same thing than balancing via weights. Thus I expect LPos and LNeg to be class_weights.

Cheers,
Martin


<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="6.5.002">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="6.5.002" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="6.5.002" expanded="true" height="60" name="Retrieve Golf" width="90" x="45" y="120">
        <parameter key="repository_entry" value="//Samples/data/Golf"/>
      </operator>
      <operator activated="true" class="nominal_to_numerical" compatibility="6.5.002" expanded="true" height="94" name="Nominal to Numerical" width="90" x="179" y="120">
        <list key="comparison_groups"/>
      </operator>
      <operator activated="true" class="generate_weight_stratification" compatibility="6.5.002" expanded="true" height="76" name="Generate Weight (Stratification)" width="90" x="313" y="120">
        <parameter key="total_weight" value="14.0"/>
      </operator>
      <operator activated="true" class="support_vector_machine" compatibility="6.5.002" expanded="true" height="112" name="SVM" width="90" x="514" y="120"/>
      <connect from_op="Retrieve Golf" from_port="output" to_op="Nominal to Numerical" to_port="example set input"/>
      <connect from_op="Nominal to Numerical" from_port="example set output" to_op="Generate Weight (Stratification)" to_port="example set input"/>
      <connect from_op="Generate Weight (Stratification)" from_port="example set output" to_op="SVM" to_port="training set"/>
      <connect from_op="SVM" from_port="weights" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

JEdward

For other classifiers (& SVM too) another way to tackle this with different costs is to use cost based modelling.
Here's an example on how it works. Please note, I chose to use Naive Bayes because it demonstrated the extemes well chosing to send to the entire dataset to 'maximise profit' (Have the least cost)

You can chose to either use the Meta Cost operator or use the Performance (Costs) operators separately.
Meta cost builds several models to favour the cost.
Performance (Costs) tells you the expected cost of the model. (I tend to combine this one with an Optimise operator and then build my model parameters with the ones that decrease the cost the most).

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="7.0.000">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_direct_mailing_data" compatibility="7.0.000" expanded="true" height="68" name="Generate Direct Mailing Data" width="90" x="45" y="34">
        <parameter key="number_examples" value="100000"/>
      </operator>
      <operator activated="true" class="multiply" compatibility="7.0.000" expanded="true" height="124" name="Multiply" width="90" x="112" y="136"/>
      <operator activated="true" class="split_validation" compatibility="7.0.000" expanded="true" height="124" name="Validation" width="90" x="246" y="34">
        <process expanded="true">
          <operator activated="true" class="metacost" compatibility="7.0.000" expanded="true" height="82" name="MetaCost" width="90" x="45" y="34">
            <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
            <process expanded="true">
              <operator activated="true" class="naive_bayes" compatibility="7.0.000" expanded="true" height="82" name="Naive Bayes" width="90" x="179" y="136"/>
              <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
              <connect from_op="Naive Bayes" from_port="model" to_port="model"/>
              <portSpacing port="source_training set" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
            </process>
          </operator>
          <connect from_port="training" to_op="MetaCost" to_port="training set"/>
          <connect from_op="MetaCost" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.0.000" expanded="true" height="82" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance_costs" compatibility="7.0.000" expanded="true" height="82" name="Performance (Cost Meta)" width="90" x="112" y="187">
            <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
            <enumeration key="class_order_definition"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="7.0.000" expanded="true" height="82" name="Performance" width="90" x="179" y="30">
            <list key="class_weights"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance (Cost Meta)" to_port="example set"/>
          <connect from_op="Performance (Cost Meta)" from_port="example set" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance (Cost Meta)" from_port="performance" to_op="Performance" to_port="performance"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="create_lift_chart" compatibility="7.0.000" expanded="true" height="103" name="Create Lift Chart" width="90" x="447" y="289">
        <parameter key="target_class" value="response"/>
      </operator>
      <operator activated="true" class="split_validation" compatibility="7.0.000" expanded="true" height="124" name="Validation (2)" width="90" x="246" y="187">
        <process expanded="true">
          <operator activated="true" class="naive_bayes" compatibility="7.0.000" expanded="true" height="82" name="Naive Bayes (2)" width="90" x="45" y="34"/>
          <connect from_port="training" to_op="Naive Bayes (2)" to_port="training set"/>
          <connect from_op="Naive Bayes (2)" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true">
          <operator activated="true" class="apply_model" compatibility="7.0.000" expanded="true" height="82" name="Apply Model (2)" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance_costs" compatibility="7.0.000" expanded="true" height="82" name="Performance (Cost Without Meta)" width="90" x="45" y="238">
            <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
            <enumeration key="class_order_definition"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="7.0.000" expanded="true" height="82" name="Performance (2)" width="90" x="179" y="30">
            <list key="class_weights"/>
          </operator>
          <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
          <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (Cost Without Meta)" to_port="example set"/>
          <connect from_op="Performance (Cost Without Meta)" from_port="example set" to_op="Performance (2)" to_port="labelled data"/>
          <connect from_op="Performance (Cost Without Meta)" from_port="performance" to_op="Performance (2)" to_port="performance"/>
          <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <operator activated="true" class="subprocess" compatibility="7.0.000" expanded="true" height="124" name="Just playing around" width="90" x="380" y="85">
        <process expanded="true">
          <operator activated="true" class="performance_to_data" compatibility="7.0.000" expanded="true" height="82" name="Performance to Data" width="90" x="45" y="34"/>
          <operator activated="true" class="performance_to_data" compatibility="7.0.000" expanded="true" height="82" name="Performance to Data (2)" width="90" x="45" y="238"/>
          <operator activated="true" class="extract_macro" compatibility="7.0.000" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="136">
            <parameter key="macro" value="CostWith"/>
            <parameter key="macro_type" value="data_value"/>
            <parameter key="attribute_name" value="Value"/>
            <parameter key="example_index" value="1"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="extract_macro" compatibility="7.0.000" expanded="true" height="68" name="Extract Macro (2)" width="90" x="179" y="238">
            <parameter key="macro" value="CostWithout"/>
            <parameter key="macro_type" value="data_value"/>
            <parameter key="attribute_name" value="Value"/>
            <parameter key="example_index" value="1"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="generate_data_user_specification" compatibility="7.0.000" expanded="true" height="68" name="Gen Data" width="90" x="313" y="289">
            <list key="attribute_values">
              <parameter key="CostDifference" value="parse(%{CostWith})-parse(%{CostWithout})"/>
            </list>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="format_numbers" compatibility="7.0.000" expanded="true" height="82" name="Saving by using meta model" width="90" x="447" y="238">
            <parameter key="format_type" value="currency"/>
            <parameter key="locale" value="English (United Kingdom)"/>
            <parameter key="use_grouping" value="true"/>
          </operator>
          <connect from_port="in 1" to_op="Performance to Data" to_port="performance vector"/>
          <connect from_port="in 2" to_op="Performance to Data (2)" to_port="performance vector"/>
          <connect from_op="Performance to Data" from_port="example set" to_op="Extract Macro" to_port="example set"/>
          <connect from_op="Performance to Data" from_port="performance vector" to_port="out 2"/>
          <connect from_op="Performance to Data (2)" from_port="example set" to_op="Extract Macro (2)" to_port="example set"/>
          <connect from_op="Performance to Data (2)" from_port="performance vector" to_port="out 3"/>
          <connect from_op="Gen Data" from_port="output" to_op="Saving by using meta model" to_port="example set input"/>
          <connect from_op="Saving by using meta model" from_port="example set output" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="source_in 3" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
          <portSpacing port="sink_out 3" spacing="0"/>
          <portSpacing port="sink_out 4" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Generate Direct Mailing Data" from_port="output" to_op="Multiply" to_port="input"/>
      <connect from_op="Multiply" from_port="output 1" to_op="Validation" to_port="training"/>
      <connect from_op="Multiply" from_port="output 2" to_op="Validation (2)" to_port="training"/>
      <connect from_op="Multiply" from_port="output 3" to_op="Create Lift Chart" to_port="example set"/>
      <connect from_op="Validation" from_port="model" to_op="Create Lift Chart" to_port="model"/>
      <connect from_op="Validation" from_port="averagable 1" to_op="Just playing around" to_port="in 1"/>
      <connect from_op="Create Lift Chart" from_port="model" to_port="result 5"/>
      <connect from_op="Create Lift Chart" from_port="lift pareto chart" to_port="result 6"/>
      <connect from_op="Validation (2)" from_port="model" to_port="result 1"/>
      <connect from_op="Validation (2)" from_port="averagable 1" to_op="Just playing around" to_port="in 2"/>
      <connect from_op="Just playing around" from_port="out 1" to_port="result 2"/>
      <connect from_op="Just playing around" from_port="out 2" to_port="result 3"/>
      <connect from_op="Just playing around" from_port="out 3" to_port="result 4"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="84"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
      <portSpacing port="sink_result 5" spacing="0"/>
      <portSpacing port="sink_result 6" spacing="0"/>
      <portSpacing port="sink_result 7" spacing="0"/>
    </process>
  </operator>
</process>

MartinLiebig

While i totally support you i would like to add, that in a best case scenario you can give each example (e.g. customer) a cost (or revenue). Than you can aggregate on this and optimize on the costs. This is very similar but even more detailed. I think this is a better way - if the data is available.

~Martin