"[SOLVED] Using user set confidence for AUC performance, not working.."
mafern76
New Altair Community Member
Hi!
What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.
What I did was:
1. Rename confidence(class) to confidence(1).
2. Generate attribute confidence(0) = 0.
3. Set roles, confidence(1) to confidence_1 and 0 to 0.
4. Rename prediction(multiclass) to prediction(singleclass).
5. Map classofinterest to 1 and default to 0.
6. Transform the polynominal prediction to binominal.
This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.
Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.
On the other hand, the prediction label works because I can get accuracy.
I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?
Thanks a lot!
What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.
What I did was:
1. Rename confidence(class) to confidence(1).
2. Generate attribute confidence(0) = 0.
3. Set roles, confidence(1) to confidence_1 and 0 to 0.
4. Rename prediction(multiclass) to prediction(singleclass).
5. Map classofinterest to 1 and default to 0.
6. Transform the polynominal prediction to binominal.
This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.
Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.
On the other hand, the prediction label works because I can get accuracy.
I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?
Thanks a lot!
Tagged:
0
Answers
-
Oh well an apple fell upon my head and I realized the problem was I was assuming AUC would use confidence_1, when it could perfectly use confidence_0.
Solution: instead of (2) generate attribute confidence(0) = 0, generate it equal to 1- confidence_1...
0