compare ROCs with different subsets of attributes

Mela

Hi,

is it possible to compare ROCs of models with different subsets of attributes? I'm trying to compare the ROCs fo 2 Naive Bayes models, one has the full set of attributes and the other only has a subset. I tried including "Remove Correlated Attributes" / "Select Attributes" in the "Compare ROCs" process but I keep getting errors. "Select Attributes" displays the following error:

Exception: java.lang.ArrayIndexOutOfBoundsException
Message: 1
Stack trace:

  com.rapidminer.operator.learner.bayes.SimpleDistributionModel.performPrediction(SimpleDistributionModel.java:447)
  com.rapidminer.operator.learner.PredictionModel.apply(PredictionModel.java:76)
  com.rapidminer.operator.visualization.ROCBasedComparisonOperator.doWork(ROCBasedComparisonOperator.java:173)
  com.rapidminer.operator.Operator.execute(Operator.java:831)
  com.rapidminer.operator.execution.SimpleUnitExecutor.execute(SimpleUnitExecutor.java:51)
  com.rapidminer.operator.ExecutionUnit.execute(ExecutionUnit.java:709)
  com.rapidminer.operator.OperatorChain.doWork(OperatorChain.java:369)
  com.rapidminer.operator.Operator.execute(Operator.java:831)
  com.rapidminer.Process.run(Process.java:902)
  com.rapidminer.Process.run(Process.java:798)
  com.rapidminer.Process.run(Process.java:793)
  com.rapidminer.Process.run(Process.java:783)
  com.rapidminer.gui.ProcessThread.run(ProcessThread.java:63)


Anyone know what I'm doing wrong? Your help would be greatly appreciated!

MariusHelf

Hi Mela,

without knowing the exact process setup I can only make a guess: is it possible that you are trying to apply a model which has been trained on the full set of attributes to the example set with less attributes? That is not supposed to work - please train the model on the same example set with the same attributes as the one you apply it to. I admit the the error message could be a bit more expressive, though.

If this does not help you, please post your process here.

Cheers,
Marius

cyckuan

Biweekly task question! 

Come see me in class for the answer.