how i can describe the quality of my model using confidence level value?
bniyaseen
New Altair Community Member
if I have a classification model to detect X from dataset contains X and Y labels.
1: what I can say about the model if the confidence level of Y prediction= 100% and for X=0 (most likely = Y?
2: what I can say about the model if the confidence level of X prediction=70% and for Y=30 (most likely = X?
1: what I can say about the model if the confidence level of Y prediction= 100% and for X=0 (most likely = Y?
2: what I can say about the model if the confidence level of X prediction=70% and for Y=30 (most likely = X?
Tagged:
0
Best Answer
-
The confidence level is essentially the score. In general, the higher the confidence, the more likely that the model is going to be correct in its prediction. However, this is something that is true globally but not necessarily for any individual observation, in which case the prediction is always either correct or not correct.
You probably don't want to measure the quality of your model based on the confidence scores alone, but rather by using one of the performance metrics that are included. If you run the cross-validation tutorial, for example, you can see that for a classification problem, you can look at the confusion matrix and measure the quality of the score using measures like accuracy, precision, and recall. Or you could use the AUC (a personal favorite) which is slightly more complicated but tells you how good a job your score does overall separating and rank ordering the two classes.5
Answers
-
The confidence level is essentially the score. In general, the higher the confidence, the more likely that the model is going to be correct in its prediction. However, this is something that is true globally but not necessarily for any individual observation, in which case the prediction is always either correct or not correct.
You probably don't want to measure the quality of your model based on the confidence scores alone, but rather by using one of the performance metrics that are included. If you run the cross-validation tutorial, for example, you can see that for a classification problem, you can look at the confusion matrix and measure the quality of the score using measures like accuracy, precision, and recall. Or you could use the AUC (a personal favorite) which is slightly more complicated but tells you how good a job your score does overall separating and rank ordering the two classes.5