Model Performance

k_vishnu772

Hi All,

 

I want to see how the model performcane improves as i add data to the model.I want to add one row at a time and see the performance ? is there any way how i can i achive it and represent in graph? how to do it properly?

 

 

Thanks in advance.

 

Regards,

Vishnu

BalazsBaranyRM

Hi SGolbert!

 

Yes, the operator is deprecated but still usable (just look at the Log output).

 

If you want to do the same manually, create a list of sample ratios (e. g. 0.25, 0.50, 0.75, 1) or absolute sample sizes (e. g. 100, 150, 200, ...) and use Loop Values to sample the example set with this parameter, determine the performance and log it. 

 

Regards,

Balázs 

lionelderkrikor

Hi @k_vishnu772,

 

I think that Loop and Filter Example Range operators can be a beginning of solution...

 

In order we give you more precise elements of answer, can you share your dataset(s) and your process ?

 

Regards,

 

Lionel

BalazsBaranyRM

Hi!

 

What you describe is called a "learning curve".

There's even a sample process for this in RapidMiner:

//Samples/processes/06_Meta/03_LearningCurve

 

Regards,

Balázs

SGolbert

Hi @BalazsBarany,

 

The LearningCurve operator is unfortunately deprecated :S

 

To extend the topic a bit: What do you use for model diagnostics in RapidMiner?

BalazsBaranyRM

Hi SGolbert!

 

Yes, the operator is deprecated but still usable (just look at the Log output).

 

If you want to do the same manually, create a list of sample ratios (e. g. 0.25, 0.50, 0.75, 1) or absolute sample sizes (e. g. 100, 150, 200, ...) and use Loop Values to sample the example set with this parameter, determine the performance and log it. 

 

Regards,

Balázs 

kypexin

Hi @BalazsBarany

 

Little offtopic if you kindly let me... and just pure curiosity.

I was always wondering what is the reason for deprecating some operators without providing a replacement?  

I remember the same thing happened to 'Stream Database' operator (maybe the name is wrong but you know it, a versatile alternative to 'Read Database'). I am sure there are more examples. So -- why?

 

Thanks. 

BalazsBaranyRM

Hi @kypexin,

 

I'm not the right person to answer this, as I'm not in RapidMiner Development. There's probably one answer per operator, not one common one for all the operators.

In my experience RapidMiner tries to keep backward compatibility as far as it's possible, but sometimes keeping a deprecated operator would hinder other developments.

 

Regards,

Balázs

MartinLiebig

By the way @kypexin,

 

deprecated operators are not visible but still executeable. You can even get them back into processes if you know the key for it in the xml. Keys can be extracted from this xml: https://github.com/rapidminer/rapidminer-studio/blob/master/src/main/resources/com/rapidminer/resources/OperatorsCore.xml

 

W.r.t deprecation: Most deprecates are because we got a better replacement. Others have very special reasons.

 

BR,

Martin

Marco_Boeck

Hi,

 

Operators that disappear without a replacement should be very rare. They are still there, just hidden, to keep backwards compatibilty for existing processes. I cannot remember any other than those two examples in the past couple of years.

Stream Database was "soft-killed" because it was broken beyond repair, and you can have the same functionality (just working) via Loops + Read Database and using macros.

Create Learning Curve was removed because it did not do anything useful for 99% of the few people who did find it (and were then confused by it), so we figured it's better to remove it than continue getting people confused by it.

 

To sum it up, not getting a direct replacement either means there are other ways you can achieve the same result, or we consider the operator to be used so rarely, that we rather invest our time in things that make the product better for everyone than a very small minority. Unfortunately, that's a call you sometimes have to make..

 

Regards,

Marco