Using AI to Predict Machine Learning Model Accuracy

Joseph Pajot

Machine learning can predict physics, but can it predict its own errors?  Learn how to better understand these errors to get the most out of your data models.

Machine Learning with simulation data to build predictive models has proven to be a valuable tool for engineers.  These data models produce results in a fraction of the time of a full computational physics simulation.  Today’s cutting-edge companies quickly design and evolve their next generation of products using this combination of data analytics and CAE technologies seamlessly integrated into software with intuitive and easy to use interfaces such as Altair HyperWorks and HyperStudy, as seen in the videos below.

 

 

 

Almost like magic, the predictions learn the cause and effect relationships in complex physical systems.   But it isn’t magic, its just math, and as with any mathematical model there are errors associated with the predictions.  Because these models are used for engineering design, predictive accuracy must be quantified and understood.  At some point, every engineering data scientist ponders: “When machine learning predicts the physics, can it also predict its own errors?”.  This information would be useful, for example, to know that a data model for vibrations is accurate for low frequency but less reliable at high frequencies.  So, is it possible to train a metalevel predictive model to anticipate the error from another data model?   It turns out that while it is not possible to predict the error itself, it is possible to predict a useful error bound.  At first glance, this may seem contradictory, but with some explanation it becomes evident why this happens.

For discussion, consider two simple datasets, b and s, with a single independent variable, as shown in the image below along with their associated regression curves, in blue.

It is insightful to visualize the prediction errors for each model as a function of the independent variable.

Each dataset has a distinct error pattern.  The errors from the b dataset show a dependency on the independent variable while the s dataset errors appear to be completely random.   Both error patterns have a mean value of approximately zero.  This zero-mean is an expected consequence of how regression works, but its effect is that any attempt to predict this error with another regression will result in a model that predicts the trivial mean value of zero, as show in the blue lines in the image here.

To get something useful, we will need to build a model on a modified form of this error that has a non-zero mean.  For example, taking the absolute value of the errors and regressing that quantity results in a model with non-trivial prediction curves, as seen here.

The predictions of the absolute errors for the s dataset are constant while the b dataset errors vary, and this is expected given the patterns in the starting data.  This result shows it is possible to predict and error measure associated with a predictive regression model and leads to three thoughts.

1. If the error itself were easily predicted with regression, the first regression itself would have already accounted for with a more accurate prediction. This seeming paradox is why only a modified form of the error can be predicted, such as the absolute value or squared error.
2. Biased predictions (read more here) can be used to envelope the modified error bounds, but for ideal cases such as the s dataset, the envelope is only a confirmation of the standard conventional training metrics already available from the initial trainings, such as the max or mean absolute errors.
3. When the predicted error has a pattern that correlates to with an independent variable, such as the b dataset, it is known as endogeneity.  This is not a desirable property, and preferably avoidable, if possible.

In conclusion, it is possible to train a secondary model to predict an error bound such as absolute error, but any non-trivial outcome indicates issues with the model, such as insufficient variables or an indication of the need for more training data in that area.  This information can be useful to identify improvements to a model or dataset, even if it can’t be used to directly self-correct the primary predictive model.  This error information needs to be presented alongside any predictions, for example with error bounds alongside a predicted curve in the image below.

It is possible to do better than assessing predictive data model quality based on abstract textbook criteria.  To establish confidence and reliability, engineers need to be informed of different aspects of the predictive model’s behavior, such as the estimated error alongside the predictions. Altair is focused and committed to bringing these types of data visualizations to our software to augment engineering decision making and maximize the returns of data driven models.