Is my model good enough?

onep

Hello.

I'm trying to make a predictive regression model, but having a hard time telling if my model is good enough?

I'm using X-validation and I've read somewhere that you can tell if it's a good fit based on the difference between the training error and the validation error? But how do I get the X-validation to tell me the training error?

 

Currently my model has a RMSE of about 1,000 and my label has a range from 0 to about 32,000. Out from this I can't really tell if it is any good? Is there another way I can measure if it's a good model?

 

Oh, and one more thing - I can manage to make my model better if I use a k-NN global anomaly score and remove some of the outliers comming from noise - but i'm afraid that I remove too much information. How can I decide how many outliers I can remove?

 

Thanks in advance!

MartinLiebig

Hi mathias,

it really depends on the use case wether this is good or not. Hard to judge. But i would of course look at the testing error, not the training.

 

What might help you to get a better feeling is to get a plot of the scored set returned by the new Cross Validation in 7.3. Just plot label against prediction(label) in a scatter plot. You can extract information like "if the truth is 5, my prediction is between 3 and 7".

 

Best,

Martin