Altair RISE

A program to recognize and reward our most engaged community members

Nominate Yourself Now!

Can I see the individual results of each X-Validation fold (before they are averaged)?

Dear community,

Is there a way to view the individual performance results of each fold of X-Validation or only the overall average?

I am applying a 5-fold X-Validation and would be interested in understanding the how different the results of each fold are, e.g., in terms of accuracy (or even better F1 score)?

If this is not possible with the X-Validation operator, what alternatives would I have? (As a work-around I created three 80/20 splits with the Split Validation operator, using the three different "sampling types", but I am not really satisfied with this appraoch.)

Thanks in advance!

Find more posts tagged with

AI Studio

Accepted answers

MartinLiebig

Dear Jochen,

it is possible, but just with a little trick. You can use performance to data on your performance vector and put this on the inner tes port. If you do this, you get a table with performances back at the outer tes port.

The reason that you need to do this trick is, that it often is not wanted to look at the individual performances. Keep in mind: The average of the perfomance of a cross validation is the best unbiased estimator for the true performance of the method. (with an emphesis on method).

I saw time and time again that scientist report the best fold of their x-val in papers. This is simply wrong. Only the average is to be reported.

Best,

Martin

MartinLiebig

Hi,

i personally go for a simple t-test on the two performances. It's very tough to do more tests with a sample size of 5-10. In your case both will be fine and you usually take the "easier explainable" one. For SVM and k-NN you might take SVM because it is WAY faster in application (if you need this speed).

Best,

Martin

All comments

MartinLiebig

Dear Jochen,

I saw time and time again that scientist report the best fold of their x-val in papers. This is simply wrong. Only the average is to be reported.

Best,

Martin

jochen_hartmann

Thanks so much for your swift and helpful reply, Martin!

Understood. Then, my follow-up question is rather a statistical one:

How can I tell if the X-Validation results of two different classifiers, e.g., kNN and SVM, are in fact statistically distinguishable if they are very close to each other (accuracies of 44.5 and 45.2, respectively, for example). I thought it may be a good idea to look at the underlying distributions of accuracies from each fold, but I may also be completely off.

In other words, is there a way to determine a threshold to be able to tell if two results are "different enough"?

Thanks!

MartinLiebig

Hi,

Best,

Martin

jochen_hartmann

Thank you, Martin!