"Compare the perfomance of various models (accuracy)"

a64863
a64863 New Altair Community Member
edited November 2024 in Community Q&A

Hi,

 

I'm new to data mining, and i'm stuck in a project that my teacher gave me to do.

I have a data set and 6 models and for each model i want to generate a report that compare the accuracys between them.

The illustration of my work is here:

 

15102336_1350520011639610_972960018_o.pngProcess15064892_1350518428306435_1241446617_o.pngCross Validation

I'm sorry if this is not the correct place for post this doubt, but like i said before, i'm new here.

 

Thanks for the attention!

Best Answer

Answers

  • Telcontar120
    Telcontar120 New Altair Community Member

    I am not totally certain what output you are trying to achieve, but you might try the operator "Compare ROCs."  You simply place your individual models into that subprocess, which is similar to the Cross Validation process you are using.  It will produce an exhibit that shows the performance of all of the models together. 

     

     

  • MartinLiebig
    MartinLiebig
    Altair Employee

    Hi,

     

    i usually use Performance to Data. This can then be joined/filtered/appended how ever you like it.

     

    ~Martin

  • Telcontar120
    Telcontar120 New Altair Community Member

    @mschmitz I have a related question: when running cross-validation, if performance vectors are output, there are 3 summary values returned, such as the following example:

    "accuracy: 77.896% +/- 3.824% (mikro: 77.912%)"

    Can you clarify the calculation of these 3 metrics? My assumption is as follows:

    1. The first is the simple arithmetic mean of the chosen performance metric across the k folds of the cross-validation.
    2. The +/- margin is simply the standard deviation of the recorded performance vector across the same k folds of the validation and it is computed using the typical sample standard deviation formula.
    3. I don't know, however, what the third value reported in parentheses represents--some kind of adjusted average performance? (mikro=German for micro?) My obseravtion from experience is that it is often identical or very close to the first average performance reported, but I don't know its meaning or derivation?

    Thanks for the help!  

     

  • MartinLiebig
    MartinLiebig
    Altair Employee

    Brian,

     

    you got it right. Mirko is german for micro :). And also the rest. The one value is a simple average. The other value is a average where you use the number of examples on the testing side of the fold as a weight. I always mix it up which one is which.

     

    With enough examples micro=macro because all weights are equal.


    ~Martin

  • Telcontar120
    Telcontar120 New Altair Community Member

    Thanks @mschmitz!  

    @IngoRM do you remember which average is which (i.e., what the mikro version is, weighted or unweighted)?

    Happy Thanksgiving!

     

  • MartinLiebig
    MartinLiebig
    Altair Employee
    Answer ✓

    Brian,

     

    Micro is the weighted one. See: http://rushdishams.blogspot.de/2011/08/micro-and-macro-average-of-precision.html

     

    ~martin

  • Telcontar120
    Telcontar120 New Altair Community Member

    Thanks @mschmitz!  

  • IngoRM
    IngoRM New Altair Community Member

    Man, people are too fast here.  I never get a chance to answer myself :smileytongue:

     

    Thanks,

    Ingo