Versatile and No Code Simulation Data Extraction with Monarch

Text data files come in many formats yet have few standards.  Monarch saves time and effort extracting data from tricky simulation outputs.

One of the great synergies between engineering and data science comes from multiple runs of physics simulations.  Whether powered by HPC, cloud, or hybrid computing, modern systems can efficiently run all the virtual experiments required to train machine learning models.  As an engineering data scientist, I feel like a kid in a candy store with all the data that gets generated and all the possibilities to gain insights from it.  However, almost as unlimited as the volume of data is the variety.  Simulation codes span a range of physics (check out the list you get with just Altair!) with each code often capable of generating a corresponding range of output formats, from binary to human readable.  Binary formats are often supported with associated reader APIs that provide a straightforward solution to collecting data, but in my experiences, it is the seemingly simple human readable text files that cause some of the biggest headaches. Altair Monarch is a versatile data prep tool that can streamline data collection from nearly any text output.

To illustrate its use, let’s take a very simple example of an Altair OptiStruct out file generated by a topology optimization.  The out file has a repeating block of information that contains the status for each iteration of the optimization process.  For demonstration, we will focus on collecting the mass information within each block.  Monarch is used to author a data trap template that can be saved and applied to any new instance of this file to automate data collection.  Once the file is open in Monarch, the steps are illustrated in the image below : (1) add a “detail” to our trap definition, (2) highlight the one of the lines of interest, (3) add the line to the sample text, (4) highlight the character fields of interest, and (5) define a template filter (“Mass    =”) that ensures that only the lines matching the filter are captured.  The data collected by the trap are previewed at the bottom of the interface (6).

Monarch data traps can be saved and applied to any new instance of this file to automate data collection, providing a no-code solution to collect data efficiently from large sets of simulation data.  As an introduction, this demonstration focused on a very simple use case with a single field record.  In future posts, I will explore more complicated traps to capture data records with multiple fields extending across more than one line.