WEKA learners + xvalidation

Unknown

Hi
is there a way to 'mix' weka learners with validation and optimization operators?

So far it seems Weka operators dont really work with validation,performance or optimization operators.
thx

wessel

What you mean it does not work?

Attached below is a process that uses a Bayesian Network inside x-validation.
It loads the famous play tennis data and builds a Bayesian Network (using -- -P 22 to get a real network).
I double checked that the results look the same in Weka as they do in Rapid Miner.
The only problem I can find is that you can not turn off the "use AD Tree" in Rapid Miner.

Correctly Classified Instances          9              64.2857 %
Incorrectly Classified Instances        5              35.7143 %

accuracy: 60.00% +/- 43.59% (mikro: 64.29%)

Best regards,

Wessel



<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<process version="5.2.006">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="5.2.006" expanded="true" name="Process">
    <parameter key="logverbosity" value="status"/>
    <process expanded="true" height="392" width="300">
      <operator activated="true" class="retrieve" compatibility="5.2.006" expanded="true" height="60" name="Retrieve" width="90" x="45" y="30">
        <parameter key="repository_entry" value="//Samples/data/Golf"/>
      </operator>
      <operator activated="true" class="x_validation" compatibility="5.2.006" expanded="true" height="112" name="Validation" width="90" x="180" y="30">
        <process expanded="true" height="392" width="165">
          <operator activated="true" class="weka:W-BayesNet" compatibility="5.1.001" expanded="true" height="76" name="W-BayesNet" width="90" x="45" y="30">
            <parameter key="Q" value="weka.classifiers.bayes.net.search.local.K2 -- -P 22 -S BAYES"/>
            <parameter key="E" value="weka.classifiers.bayes.net.estimate.SimpleEstimator -- -A 0.5"/>
          </operator>
          <connect from_port="training" to_op="W-BayesNet" to_port="training set"/>
          <connect from_op="W-BayesNet" from_port="model" to_port="model"/>
          <portSpacing port="source_training" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <portSpacing port="sink_through 1" spacing="0"/>
        </process>
        <process expanded="true" height="392" width="165">
          <operator activated="true" class="apply_model" compatibility="5.2.006" expanded="true" height="76" name="Apply Model" width="90" x="45" y="30">
            <list key="application_parameters"/>
          </operator>
          <operator activated="true" class="performance_classification" compatibility="5.2.006" expanded="true" height="76" name="Performance" width="90" x="45" y="120">
            <list key="class_weights"/>
          </operator>
          <connect from_port="model" to_op="Apply Model" to_port="model"/>
          <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
          <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
          <portSpacing port="source_model" spacing="0"/>
          <portSpacing port="source_test set" spacing="0"/>
          <portSpacing port="source_through 1" spacing="0"/>
          <portSpacing port="sink_averagable 1" spacing="0"/>
          <portSpacing port="sink_averagable 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Retrieve" from_port="output" to_op="Validation" to_port="training"/>
      <connect from_op="Validation" from_port="model" to_port="result 1"/>
      <connect from_op="Validation" from_port="training" to_port="result 2"/>
      <connect from_op="Validation" from_port="averagable 1" to_port="result 3"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
      <portSpacing port="sink_result 4" spacing="0"/>
    </process>
  </operator>
</process>

Unknown

i wish i knew...
was getting butt load of wild errors with any weka learner in validation wrapped in param opt(worked fine with e.g. NN).

will look into it...

wessel

Send the process XML?

Weka learners can sometimes be a bit counter intuitive to get working, but they should work.

Unknown

Standard param optimization design as in Sample RM folder.

Now am testing some simple flows using xval,boostin... and all is fine.

maybe tomorrow ill try optimization again and will see.

MariusHelf

As Wessel said: without the process XML we can't help you much. Often it's the small things which cause the problems and are not caught in "Standard param optimization as in Sample folder".

Best, Marius