Different cost of misclassification for every single example – how to implement?
Hello all!
Please advise on the following problem.
[Dataset description]
Dataset contains about 10 thn examples with about 15-20 numeric attributes (not sure that all of them are useful, many are correlated, so this amount can be diminished to 7-10 attributes). Target (label) for these is binominal, let’s say it’s ‘One’ and ‘Two’.
By nature the data is quite noisy which means that it’s normal to talk about probability that case X belongs to class ‘One’ or ‘Two’. The normal expected probability is not expected to be higher than 0.75 (lower than 0.25) for 95% of cases.
The above is pretty typical. Now what I believe is special: for every case in dataset there is a confidence threshold.
Let’s say we have case X1, it’s actual class is ‘One’, threshold is T1. If we learn a model that predicts case X1 to be class ‘One’ with confidence P1, and P1 > T1 then Cost = 0. But if P1 < T1 then Cost = 1/T1.
For case X2 actual class is ‘Two’, threshold is T2. If a model predicts case X2 to be class ‘One’ with confidence P2, and P2 > T2 then Cost = 1/(1-T2). But if P2 < T2 then Cost = 0.
[Problem]
Normally I use some standard classifier like kNN which provides me with confidence output. Then I compare those confidences computed with my thresholds for every case (I do it with Excel). And this gives me final estimation for training set and for testing set. Then I compare performances and decide whether model is good or bad.
What I want is to make a model minimize the Cost function, which is sum of Costs for every case in set.
Does anyone know how It can be done with Rapidminer? I will appreciate any feedback.
Please advise on the following problem.
[Dataset description]
Dataset contains about 10 thn examples with about 15-20 numeric attributes (not sure that all of them are useful, many are correlated, so this amount can be diminished to 7-10 attributes). Target (label) for these is binominal, let’s say it’s ‘One’ and ‘Two’.
By nature the data is quite noisy which means that it’s normal to talk about probability that case X belongs to class ‘One’ or ‘Two’. The normal expected probability is not expected to be higher than 0.75 (lower than 0.25) for 95% of cases.
The above is pretty typical. Now what I believe is special: for every case in dataset there is a confidence threshold.
Let’s say we have case X1, it’s actual class is ‘One’, threshold is T1. If we learn a model that predicts case X1 to be class ‘One’ with confidence P1, and P1 > T1 then Cost = 0. But if P1 < T1 then Cost = 1/T1.
For case X2 actual class is ‘Two’, threshold is T2. If a model predicts case X2 to be class ‘One’ with confidence P2, and P2 > T2 then Cost = 1/(1-T2). But if P2 < T2 then Cost = 0.
[Problem]
Normally I use some standard classifier like kNN which provides me with confidence output. Then I compare those confidences computed with my thresholds for every case (I do it with Excel). And this gives me final estimation for training set and for testing set. Then I compare performances and decide whether model is good or bad.
What I want is to make a model minimize the Cost function, which is sum of Costs for every case in set.
Does anyone know how It can be done with Rapidminer? I will appreciate any feedback.
Find more posts tagged with
Sort by:
1 - 3 of
31
Gabor, thanks for your answer. It was useful as I'm new to rapidminer and I didn't know anything about meta-operators, now I do. Still the problem is that what MetaCost and couple of WEKA operators do is that they use the cost matrix *by classes*. And I need different cost per every single case, not class.
Performance doesn't really help as I want to learn using these costs, not just to estimate the result.
Or maybe I have to think of how to reformulate my task to make it solvable by existing algorythmes.
Performance doesn't really help as I want to learn using these costs, not just to estimate the result.
Or maybe I have to think of how to reformulate my task to make it solvable by existing algorythmes.
You might try something with the weight role. Some of the operators use that information. From your description it seems a column can be generated, and its role can be set to weight.
Maybe the custom cost function for the labels would be a good option for you? The Performance (Costs) operator does that (you can specify the cost for each misclassification in a matrix), and there is also a meta-modelling operator called MetaCost.
Hope this helps, gabor