Text Categorization and BinomialClassificationPerformance

soras

Hi!

I am experimenting with the text categorization on the Reuters dataset. In this dataset, each text might belong to several categories or not belong to a category at all. So I made a binary classifier for each category which predicts whether a text belongs to the given category (positive cases) or not (negative cases).

Problem is that some categories contain very few positive cases. So, if I measure performance of a classifier, there might be a few or even none positive examples in the testing set. Currently I use BinominalClassificationPerformance to measure performance and for some categories I get "unknown" values as a measure, for example:

precision: unknown (positive class: barley_pos)
ConfusionMatrix:
True: barley_neg barley_pos
barley_neg: 458 1
barley_pos: 0 0
recall: 0.00% (positive class: barley_pos)
ConfusionMatrix:
True: barley_neg barley_pos
barley_neg: 458 1
barley_pos: 0 0

If I look at the confusion matrix above, I see that all the negative cases were predicted correctly as negative, so I am not sure that I can agree with the results showing poor or unknown performance. The question is, how to correctly measure performance in such cases?

RalfKlinkenberg

Hi Soras,

accuracy is not the best measure for very unbalanced class distributions. Therefore the measures precision and recall try to capture additional information about the quality of text classifiers. In your example case, the recall of the minority class was 0%, i.e. no positive example of the rare class was recognized, while the recall of the negative cases was 100% and the accuracy was also nearly 100%. Often rare classes or events are important and then the recall on these classes is very important.

To avoid test sets without cases of the rare class, you can use cross-validation (XValdiation operator in RapidMiner) with stratified sampling. Of course you need to make sure that you provide at least as many examples of the rare class as the number of folds (iterations) in the cross-validation.

To put more emphasis on rare classes, you can use MetaCost to assign higher costs (importance) to minority classes. MetaCost is a meta-learner and can be wrapped around any classification learner. Alternatively you can use cost-sensitive learners. JMySVM for example supports different weights for the positive and the negative class in binary classification settings. Or you could use learning techniques able to handle weights, like DecisionTree and NaiveBayes, and assign individual weights to all examples.

I hope this helps.

Best regards,
Ralf