Regarding Kappa Value in Cross Validation

varunm1

Hello,

Kappa values can be positive or negative depending on the type (level of agreement +ve, level of disagreement -ve). But in cross-validation how the final value of kappa is calculated? Are the negative values converted to positive and then the average kappa of all the folds are taken into consideration or are they are averaged directly?

Thanks,
Varun

MartinLiebig

Hi @varunm1 ,

this is the code we use:

double pa = accuracy;
double pe = 0.0d;
for (int i = 0; i < counter.length; i++) {
   double row = 0.0d;
   double column = 0.0d;
   for (int j = 0; j < counter[i].length; j++) {
      row += counter[i][j];
      column += counter[j][i];
   }
// pe += ((row * column) / Math.pow(total, counter.length));
   pe += row * column / (total * total);
}
return (pa - pe) / (1.0d - pe);

Where counter has the confusion matrix values.

Does this make sense to you?

BR,

Martin

MartinLiebig

Hi @varunm1 ,

this is the code we use:

double pa = accuracy;
double pe = 0.0d;
for (int i = 0; i < counter.length; i++) {
   double row = 0.0d;
   double column = 0.0d;
   for (int j = 0; j < counter[i].length; j++) {
      row += counter[i][j];
      column += counter[j][i];
   }
// pe += ((row * column) / Math.pow(total, counter.length));
   pe += row * column / (total * total);
}
return (pa - pe) / (1.0d - pe);

Where counter has the confusion matrix values.

Does this make sense to you?

BR,

Martin

varunm1

Hello @mschmitz

Thanks for your response. I see that this kappa (Cohen) formula is combining all confusion matrices into one to get final kappa value in cross-validation. Is my understanding correct? Generally, we take the arithmetic mean of all individual folds, so I was confused how it's working here.

Thanks,
Varun

MartinLiebig

Hi,
nono. This is basically the formula to get one fold's Kappa. In X-val it is again aggregated.
BR,
Martin

varunm1

@mschmitz Thanks. I got it, so the negative kappa will remain negative while aggregating. 

Thanks,
Varun