How to rate / score created models

qwertz

Dear community,

I've got the following puzzle to solve but my problem is that I currently dont't know how to get started.

I have 10 models that provide sales prediction for different countries. These models have each been built on several training and testing iterations so that I have quality figures for each model like "prediction trend accuracy" or "standard deviation" (based on ratio prediction to label).

Now I feed all the models with the latest market data and get a prediction value from every applied model. But which predicted value has the best potential?

Example:
Model 1 has high quality figures, e.g. "prediction trend accuracy" is 85% and "standard deviation" is 1,5. The predicted value however is rather low, e.g. 15.
Model 2 has low quality figures, e.g. "prediction trend accuracy" is 60% and "standard deviation" is 8.  The predicted value in this case is rather high, e.g. 30.
How to find out which scenario is the better one - the one that predicts a higher market value but with less accuracy or the other one?

I thought of maybe applying a decision tree and train it with data from the past but not sure whether this is a valid way at all.
Another idea is simply to multiply "prediction trend accuracy" with the predicted value increase / decrease.

It would be glad if you could help me to find the right approach.


Kind regards
Sachs