How important is the confidence interval? Should I worry if the confidence is low?

Ras94

Hi,

I have made two final models, both are performing equally good when cross validating. When it comes to deployment in a business setting, one of them shows very low confidence in its prediction (the most confident prediction is around .6 and even when its confidence is 0.25 or more, it decides to predict it as true.) How come? When it is predicting on the training and test set, it has much higher confidence in its predictions.

My other model is equally good in terms of cross validation, but when deploying it, it has much higher confidence in its prediction (most ranging from .6 to .9

Should I ignore this or can I conclude based on this, that the latter model is better to use?

IngoRM

Hi,

Confidences in general are not comparable between different models.  They are calculated in different ways for each model type and you can not conclude that one model is better than another one just based on them (at least not without any additional postprocessing).  In general I would recommend to pick the model based on accuracy, scoring speed, understandability etc. and then adapt your business rules to the typical range of confidences produced by the selected model.

Hope this helps,
Ingo