Breaking the Billion-Particle Simulation Barrier with Altair EDEM
Altair EmployeeSimulating how bulk and granular materials interact with equipment, containers, and one another is a critical capability for industrial, manufacturing, and life science organizations. The larger these simulations, the more accurate they become, decreasing the time and expense companies must spend iterating designs and prototypes.
Altair and Google Cloud recently collaborated to see how large a simulation they could produce using Altair® EDEM™ on a single Google Cloud virtual machine.
The challenge
Physical testing of equipment and materials can be both costly and time-consuming. Engineering companies often build virtual testing environments to help mitigate these costs using tools like Altair’s EDEM software application. It provides engineers with detailed insights into the dynamics of systems that are difficult to study experimentally, especially when dealing with large numbers of particles and complex interactions. By testing and modifying designs virtually, engineers can reduce the need for extensive physical prototyping, which can save both time and money.
Discrete Element Method (DEM) simulations can also be computationally demanding. The more particles that are simulated, the higher the computational resources required. More particles can result in higher resolutions and more accurate representations of the physical behaviour of the material being simulated, but they will require immense amounts of raw computing and memory provided by the underlying infrastructure.
EDEM's GPU-based Solver
In recent years, EDEM has benefited from GPU and multi-GPU technology developments based on NVIDIA technology, which have increased performance and, therefore, allowed larger and more complex simulations. The example below shows the speed-up achieved using GPU and multi-GPU with different NVIDIA cards for a real-scale simulation of iron ore sinter in a transfer chute.
Altair and Google Cloud collaboration
In May 2023, Google Cloud announced the availability of its A3 virtual machines (VMs) with NVIDIA H100 GPUs. The A3 VMs combine NVIDIA H100 Tensor Core GPUs with modern CPUs, offering improved host memory and major network upgrades, which made large simulations possible.
Through the partnership between Altair and Google Cloud, we collaborated on two main goals: to use the new A3 VMs to simulate the largest DEM simulation possible, containing one billion particles, and to gather data to build estimates for mapping a given hardware type to a possible simulation scale.
Because the main cost associated with DEM simulations is the computation of the particle interactions (contacts), two cases were defined to better understand memory consumption with and without the presence of interactions between particles.
The first scenario considered placing particles randomly in a 30m x 30m x 30m space with no gravity or particle movement. The second scenario, designed to represent a realistic application for particle storage, considered dropping particles from a moving plate with gravity in a container of 6m x 6m x 26m.
Each case was simulated using two different particle shapes to analyse the influence of shape in performance and memory consumption. The shapes used were a spherical particle of 2mm of diameter and a multi-sphere particle composed of three spheres in a triangular configuration, with an average size of 3mm.
The first scenario successfully allowed a DEM simulation of 1 billion particles to run for the first time, including the possibility of using more complex particle shapes by using multi-spheres.
It is noted that no difference in memory usage was observed between particle shapes, with both using less than 30GB of memory per GPU, which proves that there is no extra memory usage due to the particle shape.
Analysing the computational time required with each particle shape, it is seen that 60% more time is required to calculate a second of simulation with multi-spheres versus the spherical particles.
With the second scenario, representing a more realistic case including particle interactions (contacts), it was observed that the simulation was able to add over 550 million particles before the system’s GPU memory became fully utilized, with an impressive record of more than 2.2 billion interactions computed per timestep when using multi-sphere particles.
This scenario is essential to better understand the memory consumption based on the number of contacts per particle, which is representative of the system density. The multi-sphere particles present a denser system with an average of 7.9 contacts per particle, while the spherical particles show 4.2 contacts per particle. This is highly valuable information for EDEM users, allowing them to estimate better the maximum size of their simulations that can be allocated in this system.
Because of the higher contact density, the computational cost required for each simulation shows an important increment of over three times for the multi-sphere particles compared to the single sphere, which is expected.
Compared with the results from scenario 1, it is estimated at the extra computational cost of 2.7 times due to the presence of contacts in the system.
Visualising results
With simulations of this size, a major challenge is the postprocessing and visualisation of this amount of data. Combining new visualisation tools developed in Altair and specialised hardware, it is possible to load and visualise the 1 billion particle case (scenario 1). This by itself is also a major achievement and shows the capabilities of the latest postprocessing technologies developed by Altair. The video below shows scenario 1 using Altair HyperMesh CFD for post-processing.
Conclusion
This study has shown that Altair EDEM, powered by A3 instances on Google Cloud, can solve extremely challenging problems at scale. By simulating more particles, engineers can build more accurate and representative simulations of real-world systems and materials by leveraging NVIDIA GPU-accelerated instances.
With this new scale for particle simulation breakthrough, manufacturing industries can better understand and predict granular material behaviors, evaluate equipment performance, and optimize processes on an unprecedented scale, allowing higher fidelity and larger simulations in the future.
Check on the Google Cloud blog post on this collaboration: here
Apply today for an Altair EDEM trial in Google Cloud: here