"Performance estimation"
Hi,
I'm using supervised machine learning to classify my data. The
approach I use as classifier is a decision tree (but could by any
other)- After constructing an appropriate decision tree, I would like
to measure the model's performance. What are standard measures in the
domain of statistics and artificial intelligence domain to estimate
performance of a classification algorithm?
So far, I've used a leave-one-out cross validation (due to the small
number of examples in the learning set which is about 400) to evaluate
the accuracy (classification error), i.e. how many examples in the test set
were incorrectly predicted. However, I don't think that this is sufficient
for a reliable performance evaluation. What else should I measure?
I'm not sure if a significance test would provide helpful information.
In my text book, they use the significance test to compare two
different classification algorithm w.r.t. to their absolute error
(they determine by a cross validation). Also in the one RapidMiner sample where
the T-Test operator is used, two models are compared. Can a significance test be
also exploited to make performance assumption about a single classifier?
If so, what hypothesis should be tested? And how can this be achieved
in RapidMiner which for T-Test expects two PerformanceVectors?
Thank you.
Regards,
tim
I'm using supervised machine learning to classify my data. The
approach I use as classifier is a decision tree (but could by any
other)- After constructing an appropriate decision tree, I would like
to measure the model's performance. What are standard measures in the
domain of statistics and artificial intelligence domain to estimate
performance of a classification algorithm?
So far, I've used a leave-one-out cross validation (due to the small
number of examples in the learning set which is about 400) to evaluate
the accuracy (classification error), i.e. how many examples in the test set
were incorrectly predicted. However, I don't think that this is sufficient
for a reliable performance evaluation. What else should I measure?
I'm not sure if a significance test would provide helpful information.
In my text book, they use the significance test to compare two
different classification algorithm w.r.t. to their absolute error
(they determine by a cross validation). Also in the one RapidMiner sample where
the T-Test operator is used, two models are compared. Can a significance test be
also exploited to make performance assumption about a single classifier?
If so, what hypothesis should be tested? And how can this be achieved
in RapidMiner which for T-Test expects two PerformanceVectors?
Thank you.
Regards,
tim