Material Transport in Water
“Developing a numerical simulation provides a deeper insight into the complex process of pollution transport in rivers, offering a means to enhance control and prevention strategies”
The fifth use case of HiDALGO2 is titled “Sediment and Pollution Transport” (SPT), and more specifically “Material Transport in Water” (MTW)- Direct numerical simulation of environmental multiphase flows including phase transitions”, and is led by our new partner, the Friedrich–Alexander University of Erlangen–Nuremberg (FAU), who joined officially the consortium in early March 2024.
River pollution poses significant risks to the environment, with contaminants such as microplastics and biochemical substances adversely affecting aquatic ecosystems. To better understand and mitigate this environmental threat, researchers are turning to advanced numerical simulations. Developing a numerical simulation provides a deeper insight into the complex process of pollution transport in rivers, offering a means to enhance control and prevention strategies.
Simulating river pollution transport, however, is no easy feat due to the involvement of intricate physics that must be coupled. This includes fluid simulation with a free surface to model river flow, particle simulation for the river bed (sediment transport), and the coupling between the fluid and particles. Additionally, the simulation of pollution is described by a partial differential equation, adding another layer of complexity. Addressing these challenges is essential for creating an accurate and comprehensive model that reflects real-world scenarios.
The heart of this use case lies in coupling the High-Performance Computing (HPC) multiphysics framework waLBerla with the C++ framework for large-scale, high-performance finite element simulations, HyTeG. In this collaboration, waLBerla is responsible for simulating the river flow, including the sediment, while HyTeG handles the simulation of pollution transport. Given the computationally expansive nature of this task, the application is set to run on the top European Union supercomputers, emphasizing the need for significant computing power to tackle the complexities involved in studying and mitigating river pollution.
The workflow of MTW use case
More specifically, we will use fast scalable solvers that are capable of solving nonlinear discrete systems with billions of unknowns in each time step of the simulation. We will employ state-of-the-art matrix-free multigrid as solvers or preconditioners for Krylov methods, using adaptive meshing techniques, efficient error indicators, and implicit extrapolation to obtain the best possible tradeoff between accuracy and time-to-solution. Overall, we aim initially at simulations with Lattice Boltzmann grids of a billion cells and several hundred thousand time steps. The finite element models (for temperature, species transport, etc.) will have a compatible resolution but will require one or two orders fewer time steps due to the implicit time stepping. For the particulate phase, we will begin with several million individual particles.
Their time-stepping must be synchronous with the lattice Boltzmann algorithm due to the necessary tight coupling. In the second half of the project, we will focus on scaling up with the target of increasing the number of cells by two orders of magnitude (reaching 100 billion or more) and the number of time steps by one (resulting in more than a million-time step of the LBM). This together will constitute a scale-up factor of at least 1000 during the runtime of the project.
Control and data flow of MTW Use Case
Main Algorithms, Tools, and Data in the Material Transport in Water Use Case
This use case will extend the two open-source program packages waLBerla and HyTeG in such a way that they can be used to model and predict sediment and pollutant transport. More specifically, we will simulate the hydrodynamics with a highly parallel Lattice Boltzmann algorithm, as available within the waLBerla framework, using a submodule named Mesa-PD. Also, we will simulate heat and species transport with the multigrid finite element solver HyTeG, originally developed as part of the TerraNeo project. Coupling between the subsystems will be achieved using the communication routines provided by the frameworks. A discriminating feature of waLBerla and HyTeG is the use of automatic code generation. This approach to software development relies on an in-house toolchain that can produce highly optimised code based on a sequence of transformation steps.
The transformations start from an abstract, user-friendly mathematical problem description all the way to level machine instructions (e.g. for vectorization with intrinsics). The transformations use symbolic mathematical manipulations combined with techniques from compiler construction. The power of our approach results from using customised, application-dependent transformations that cannot be realised by general purpose compilers, but only in the context of our specifically developed domain-specific languages. As for the data, this use case will mostly rely on synthetical data, as is typical for such direct numerical simulations. Physical and geometric parameters are available from precursor projects and the literature.