"Compare the perfomance of various models (accuracy)"

Hi,
I'm new to data mining, and i'm stuck in a project that my teacher gave me to do.
I have a data set and 6 models and for each model i want to generate a report that compare the accuracys between them.
The illustration of my work is here:
Process
Cross Validation
I'm sorry if this is not the correct place for post this doubt, but like i said before, i'm new here.
Thanks for the attention!
Best Answer
-
Brian,
Micro is the weighted one. See: http://rushdishams.blogspot.de/2011/08/micro-and-macro-average-of-precision.html
~martin
0
Answers
-
I am not totally certain what output you are trying to achieve, but you might try the operator "Compare ROCs." You simply place your individual models into that subprocess, which is similar to the Cross Validation process you are using. It will produce an exhibit that shows the performance of all of the models together.
0 -
Hi,
i usually use Performance to Data. This can then be joined/filtered/appended how ever you like it.
~Martin
0 -
@mschmitz I have a related question: when running cross-validation, if performance vectors are output, there are 3 summary values returned, such as the following example:
"accuracy: 77.896% +/- 3.824% (mikro: 77.912%)"
Can you clarify the calculation of these 3 metrics? My assumption is as follows:
- The first is the simple arithmetic mean of the chosen performance metric across the k folds of the cross-validation.
- The +/- margin is simply the standard deviation of the recorded performance vector across the same k folds of the validation and it is computed using the typical sample standard deviation formula.
- I don't know, however, what the third value reported in parentheses represents--some kind of adjusted average performance? (mikro=German for micro?) My obseravtion from experience is that it is often identical or very close to the first average performance reported, but I don't know its meaning or derivation?
Thanks for the help!
0 -
Brian,
you got it right. Mirko is german for micro
. And also the rest. The one value is a simple average. The other value is a average where you use the number of examples on the testing side of the fold as a weight. I always mix it up which one is which.
With enough examples micro=macro because all weights are equal.
~Martin1 -
-
Brian,
Micro is the weighted one. See: http://rushdishams.blogspot.de/2011/08/micro-and-macro-average-of-precision.html
~martin
0 -
Thanks @mschmitz!
0 -
Man, people are too fast here. I never get a chance to answer myself :smileytongue:
Thanks,
Ingo
0