Compare Models Side by Side

btibert

Is there any way to compare the performance of two or more models side-by-side?  The Compare models in the Model Management Extension appears to auto-select the winner, and Group Models -> Apply appears to process in order, effectively using the output from the last model.

The use-case is showing my students how to compare and think about the output to make a smart decision on what is best for their business problem.  While I could have them export the results individually, which requires tabbing across the results, I was hoping for something that aggregated and displayed the performance metrics side by side for each.

Telcontar120

There doesn't appear to be a single operator to product a nice table like that shown in AutoModel.  However there are a number of ways of doing this manually in RapidMiner.  You can create a table manually using the Performance to Data operator from multiple models and then merging the resulting performances together.
Alternatively you could use the Log operator to log performance values from multiple models and then combine those as well.

lionelderkrikor

Hi @btibert,

Why not use the AutoModel tool ?. In the final screen (results panels) , the performances of all selected models are compared...

Regards,

Lionel

varunm1

Hi @lionelderkrikor

Any idea, if it is possible to get a table just like the one auto model creates in our own process that we create? I looked for it but I didn't find any luck in getting, except the reporting tools that create documents. I think in auto model they are fetching values from the operators and displaying in the UI, not sure we have some operators to do the same.

lionelderkrikor

Hi @varunm1,

Yes, in practice, you're right, Varun : To my knowledge there is no operator that generates a table like the one presented by AutoModel...
But as a partial solution, you have the Compare ROCs operator. This operator can be useful to compare the performances of classifiers (binary classification).

Regards,

Lionel

lionelderkrikor

@varunm1, @btibert,

It is not exactly what you are looking for, but I found a process  that I used one time. It can be a good starting point.
This process is called "Automatic Model Selection and Optimization"
You can find it in the Training Ressources repository.

Concretely, in this first release of the process, you can compare the performance (here the accuracy) of 3 models in the results panels.
I think that this process, can be easily improved to compare N models and/or to compare other performance metrics than accuracy.

Here a screenshot of the table of performances comparaison : 



Regards,

Lionel

btibert

Thanks @lionelderkrikor, I will give your process a look.  I am aware of Automodel, but because I am teaching a class where the application of these techniques is new for them, I want the group to wrestle with the mental model of what is going on; the fitting of a model, the application of it, and the assessment.  I want the class to understand the mechanics of what is happening first.

btibert

And @lionelderkrikor , silly question, but in your process, you are able to use a note/comment and include a hyperlink.  How can I do that for my own students, as it would be great to link to external resources.  I must be missing something obvious.

Telcontar120

There doesn't appear to be a single operator to product a nice table like that shown in AutoModel.  However there are a number of ways of doing this manually in RapidMiner.  You can create a table manually using the Performance to Data operator from multiple models and then merging the resulting performances together.
Alternatively you could use the Log operator to log performance values from multiple models and then combine those as well.

btibert

Thanks @Telcontar120 , I wasn't aware of the performance to data, and that did the trick.  Obviously it's a manual approach and complexity grows linearly with the number of models we want to compare, but this does what I need.  And just for sake of argument, screenshot below on what it looks like on my side, as there was a need to add a column to identify the model type for sorting in the results.

lionelderkrikor

Hi @btibert,

OK, I understand your point of view concerning the use of AutoModel.
Concerning the link to external ressources in the note, .............it's a good question !!!
Me too, I don't see any easy method via the UI to have such links in a note...

I just see one method : manually modify the XML of the process to add the link (See the last lines of the XML of the process, tag "description" to understand).

Regards,

Lionel

btibert

@lionelderkrikor
Ahhh, I didn't think about modifying the XML, good to know!