"Compare model performance graphs"

dupolas

Hi there,

I was wondering if it is possible to run multiple classification models on the same dataset and compare the different performance curves.

For example I run a decision tree, neural net and logistic regression on a training and test set and would like to compare training and test performance.(eg. sq. avg error)

is this possible with Rapidminer?  I know it is with SAS and it is pretty usefull for me

Thanks a lot...

I googled for this and could not see any tutorials/posts on it..

Regards,

Geoffrey

MariusHelf

Hi,

I suppose you already know how to evaluate a single model in RapidMiner, e.g. with the X-Validation. The X-Validation puts out a performance vector, which should contain the root mean squared error. This value is only a scalar value, so I don't know from which data you want to create curves. Of course you can compare the values from different models, and if you extract the performance values, you also can write them into a single example set for easier comparison.

For classification problems we can create ROC comparison charts.

For the performance curves, please give us a more detailed description of what you want to do. Please read the post linked in my signature on how to write good questions.

Happy Mining!
~Marius

dupolas

Marius

Thanks for the hints.... Still struggling with some of the concepts being new to data mining .

Yes I was looking for ROC curves for classification problems.

However conceptually trying this on the iris data set didn't work out because the label needs to be binomial instead of multiclass. So will have to manipulate my data set a bit to get the 'one vs the rest' labels. Some practical learning to be done there

Regards

Geoffrey

MariusHelf

Fine. if you get stuck anywhere just post a message here!

Aprile

Hi, I have the same question too and I've tried this process. But I'm not sure if it right in theory but it actually present the Comparing ROC for my 3 models. Would you mind helping me discuss about the process, if I did anything wrong.

Aprile

https://community.rapidminer.com/discussion/58946/how-to-compare-classification-models-in-the-same-time-with-compare-their-rocs-this-is-my-process#latest