Generating Conservative Predictive Models

All prediction errors are not equivalent in engineering.  Learn how to ensure that your predictive models do not underestimate failure. 

I have spent the majority my career as an engineer creating numerical results with finite element analyses.  The remainder of my career has been spent on other numerical methods, and more recently machine learning, specifically.  Virtually simulating the real world is a perfect solution to synthesize data for supervised learning.  Personally, I have much more experience in this area with regression as opposed to classification.  However, a recent classification collaboration with other Altair colleagues led me to critically rethink how engineers build predictive models for physics.

To fully explain my train of thought, I need to start with the idea of a binary classifier.  For these problems, the machine leaning model learns to predict one of two outcomes, such as true/false, cat/dog, or yes/no.   These models could be used to identify cancer cells from medical imaging or predicting maintenance needs on vehicle fleets.  Regardless of the application, the model’s predictive ability is often reported in a confusion matrix, where the model’s performance is scored against known results. A confusion matrix from Altair’s KnowledgeStudio is shown below.

The diagonal entries in the table represent correctly predicted values, while the off diagonal entries represent incorrect predictions.  Taking the case of true/false class labels, the upper right entry is a false negative and the lower left is a false positive.  Obviously, correct predictions are most desirable, but in many industry specific applications, false positives and false negatives are not equally undesirable.  Considering the maintenance example mentioned previously, it is much more costly to misdiagnose a vehicle that should have come for a fix and then consequently breaks down than it is to send a healthy vehicle in for an unnecessary service check.  To satisfy this business objective, the predictive models are tuned to exhibit a bias against the unfavorable inaccuracies.

In a previous article, I demonstrated the effect of quality metric selection on a regression model.  In that case, the difference between a mean square error and a mean absolute error metric could have an observable impact.  Both of those metrics are inherently unbiased; overprediction and underprediction are treated equivalently.   But engineering is a conservative practice, by design.  I think we all feel safer flying through the skies knowing that engineers considered a factor of safety.   Should we expect the same conservatism from our artificial intelligence?  For example, it feels preferable for our models to overpredict the stress rather than underpredict the stress.   The confluence of these thoughts had me revisit the work from my previously mentioned article to create a biased metric that would result in conservative predictions.   The image below shows both a conventional least squares regression and another regression resulting from a biased absolute error metric where underpredictions are weighted ten times more than overpredictions.

The conventional black regression curve weaves through the data point, mixing both overpredictions and underpredictions.  In contrast, the green curve exhibits a conservative bias and consistently avoids underpredictions.  This illustrates the potential to better align physics simulation and machine learning with business objectives.