"seemingly inconsistent result in prediction with Decision Tree MetaCost"
I've built a process that intends to predict customers that are likely to churn (i.e. leave service provided by a company). I used two DT (decision tree) algorithm implementations based on C4.5 - the one of RM, and J48 of Weka. In particular DTs are useful in profiling the potential churners here, such that you learn about their characteristics. Meta learning via using the MetaCost operator was included to encourage the two algorithms to detect more possible churners. Just playing around with the parameter tuning, I discovered some abnormality:
RM's C4.5 implementation generated a tree formed by the root only: decision churn=No (expected because most customers do not churn). This is not a problem in itself and can be changed easily if you retune parameters, in particular the minimal gain. However what is a problem is that although the prediction with this tree is to be No for all instances, few instances are predicted Yes ...
Since the tree has just one node, the confidence for No is the same for all instances, and is equal to 0.726. I hardly see under these circumstances why a few instances, with the same confidence of 0.726 for No as any other instance, are predicted Yes.
Another inconsistency: the confidences for the classes No and Yes for one instance do not add up to 1.
The dataset is not publicly available for the process to be tested, but if one wants to check this likely inconsistency, I could make available the image files of the instances' scores that sufficiently illustrate what said above, the confusion matrix, the tree (however I'm not sure if the insert image button works for the posting).
Cheers,
Dan
