How can I insert a Lift Chart into a Validation

marcopo

Hello Community,
I saw this both interesting videos about ROC and LIFT charts. In this videos the LIFT charts appears all the time after a apply model.

http://www.youtube.com/watch?v=bsiG-xCoKP8

http://www.youtube.com/watch?v=j6Hxf5UtDKU

Nevertheless, in both videos they are not using a validation. I am trying to use the validation and a Lift Chart but I am getting all the time errors? I tried to insert the lift chart after the apply model in the validation but in the end I get this errors. What can I do?

Expected AverageVector but received ExampleSet.
Expected AverageVector but received DistributionModel.
Expected AverageVector but received LiftParetoChart.

In addition I am looking for a way or setting, that in the results example set an additionally row is inserted which shows the predicted class. So I can see which value is true and which one is false.

Best Regards and thank you

Marco

marcopo

No one an idea?

MariusHelf

Hi,

as always, it would be a good idea to attach the process you have so far to your post as explained in my signature.

Concerning the Lift Chart, what do you expect from a lift chart in a validation? What do you want to achieve?

Best regards,
Marius

marcopo

Hi,

actually I wanna see the original class and the class which is predicted through the model. Additionally, I wanna use the lift-chart as performance indicator.

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.008">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.008" expanded="true" name="Process">
    <process expanded="true" height="755" width="1015">
      <operator activated="true" class="retrieve" compatibility="5.2.008" expanded="true" height="60" name="Retrieve" width="90" x="45" y="390">
        <parameter key="repository_entry" value="../../Daten/TextMiningAusgang0MinBinominaNeu"/>
      </operator>
      <operator activated="true" class="split_validation" compatibility="5.2.008" expanded="true" height="148" name="Validation (2)" width="90" x="410" y="293">
        <parameter key="parallelize_training" value="true"/>
        <parameter key="parallelize_testing" value="true"/>
        <process expanded="true" height="759" width="483">
          <operator activated="true" class="nominal_to_numerical" compatibility="5.2.008" expanded="true" height="94" name="Nominal to Numerical (2)" width="90" x="45" y="30">
            <list key="comparison_groups"/>
          </operator>
          <operator activated="true" class="support_vector_machine_libsvm" compatibility="5.2.008" expanded="true" height="76" name="SVM" width="90" x="264" y="30">
            <parameter key="svm_type" value="nu-SVC"/>
            <parameter key="C" value="2.0"/>
            <list key="class_weights"/>
          </operator>
          <connect from_port="training" to_op="Nominal to Numerical (2)" to_port="example set input"/>
          <connect from_op="Nominal to Numerical (2)" from_port="example set output" to_op="SVM" to_port="training set"/>
          <connect from_op="SVM" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="759" width="483">
          <operator activated="true" class="apply_model" compatibility="5.2.008" expanded="true" height="76" name="Apply Model (2)" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="create_lift_chart" compatibility="5.2.008" expanded="true" height="94" name="Create Lift Chart" width="90" x="112" y="390">
            <parameter key="target_class" value="UP"/>
          </operator>
          <operator activated="true" class="performance_binominal_classification" compatibility="5.2.008" expanded="true" height="76" name="Performance (2)" width="90" x="246" y="30">
            <parameter key="classification_error" value="true"/>
            <parameter key="precision" value="true"/>
            <parameter key="recall" value="true"/>
            <parameter key="f_measure" value="true"/>
            <parameter key="sensitivity" value="true"/>
            <parameter key="specificity" value="true"/>
          </operator>
          <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
          <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
          <connect from_op="Apply Model (2)" from_port="model" to_op="Create Lift Chart" to_port="model"/>
          <connect from_op="Create Lift Chart" from_port="lift pareto chart" to_port="averagable 2"/>
          <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
          <connect from_op="Performance (2)" from_port="example set" to_port="averagable 3"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
          <portSpacing port="sink_averagable 3" spacing="0"/>
          <portSpacing port="sink_averagable 4" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Validation (2)" to_port="training"/>
      <connect from_op="Validation (2)" from_port="averagable 1" to_port="result 1"/>
      <connect from_op="Validation (2)" from_port="averagable 2" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

Best Regards

Marco

MariusHelf

Hi,

you can't use the Lift Chart as output of the X-Validation. As you know, the cross validation executes it's inner processes 10 times and then builds the average of the performances of all 10 iterations. But averaging a Lift Chart is not possible.
You would have to create a model outside of the validation and then create the Lift Chart based on that model.

I hope this helps!

Best regards,
Marius