Evaluate model prediciton quality with PhysicsAI

Filippo1993_G

Hello, 

I am trying to evaluate the quality of predictions made using a trained model in PhysiscsAI. 

It seems to me that the model, trained with more than 250 result files, reaches a good quality.

This can be stated considering 2 aspects (as reported in the figure herunder): 

• Minimal difference in MAE values between the model trained with TRAIN-DATA vs the model trained with TEST-DATA
• Low value of MAE with respect to the maximum contour value obtained in the corresponding simulations: the MAE is 2 order of magnitude smaller compared to the contour value. 

Additionally, when I try to predict the contour value on new geometry, I obtain very high values of confidence (close or equal to 1) (see the image herunder)

Nevertheless, when I try to compare the prediction with the ''conventional'' simulation obtained from Optistruct (applied on the same geometry) I notice a  similar stress distribution, but different values of the contour (comparing the max values of the scales reported in the image above and hereunder, there is a difference of almost 2x in the stress value)

 

So the questions are: 

1. Is my AI model of good quality?
2. How can I compare the results obtained with Physics AI with the ones obtained with Optistruct in order to obtain useful information concerning the structural repsonse?

Thank you in advance.

Community-Pasted-Attachment-1723110769652.png

Adriano_Koga

Hi,

your MAE is related to the same physical attribute you're training, so if you have a model showing stresses around 100 MPa, then your MAE ideally should be as small as possible, but at least around 5MPa or less.

 

The comparison that you're performing with a new geometry will give you the confidence level of that geometry. Confidence level is related to how similar the new geometry is to the training set. Confidence of 1.0 means your new geometry is really similar to the original training set, so results should match quite well, assuming you have a nice model trained.

 

When comparing the prediction to the real solver result, you shoukld always adjust the legends to be the same, and check if the difference isn't related to a hotspot, or singularity.