Introducing the feelpp.benchmarking framework

A Comprehensive Benchmarking Solution for HPC

In the context of High Performance Computing, it is crucial to understand how applications perform in different systems and architectures. However, finding a benchmarking tool that combines flexibility, robustness, reproducibility of results, and a complete and easy to use pipeline that goes from benchmarking to generating comprehensive reports, can be quite challenging.

Why feelpp.benchmarking?

This is where the feelpp.benchmarking tool comes in hand. Built on top of ReFrame, this application was conceived to facilitate and automate benchmarking tasks for HPC users, as well as to provide users and organizations a centralized platform to aggregate and display results in the form of a dashboard.

With a focus on flexibility, the feelpp.benchmarking tool ensures reproducibility of results, and is equipped with a continuous benchmarking pipeline that empowers CI/CD workflows for any application.

Although the framework is still under development, feelpp.benchmarking is already functional and is becoming increasingly robust. By focusing on flexibility, modularity and adaptability, it has been set to become a valuable resource for any developer or organization looking to be fully aware of their application’s performance over the course of its development.

A Key Tool for HiDALGO2

In the context of the HiDALGO2 project, this open source application is extremely useful for monitoring the pilots’ performance over time and to automatically be able to launch new benchmarks for a given task.

For example, the Urban Buildings pilot can be configured to compare its parallel code performance on multiple EuroHPC systems each time a new feature is introduced to the main codebase. That way, it can be guaranteed that the application’s scaling has not been compromised.

Key Features

The most notable features of the feelpp.benchmarking are:

General dashboard creation
It is possible and easily feasible to use the render component of the feelpp.benchmarking tool to generate not only a benchmarking dashboard, but also any kind of gallery-like static website. This can be done by simply modifying the input templates and configuring the Model-View-Controller (MVC) components. This way, users can adapt the tool to suit their needs.
Flexible configuration files
Machine, benchmark and figure configuration files support complex parametrization and include a placeholder syntax for easy refactoring and reuse.
Container support and benchmark reproducibility
The feelpp.benchmarking tool seamlessly integrates Apptainer containers, ensuring compatibility across many HPC systems. The framework ensures, via input configuration, that results are consistent and reproducible. Additionally, the tool guarantees dashboard persistence, allowing users to track performance over time and across different views.
HPC system integration
At the moment, the Discover supercomputer is supported, and many more machines are planned to be integrated into the framework. Integrating a new HPC system can be done easily by describing the hardware, following ReFrame’s configuration reference, and configuring access through the CI runners.
A Continuous Benchmarking (CB) workflow The benchmarking framework of Feel++ provides a pipeline that can be directly executed by any application via a REST request, enabling continuous benchmarking for any application through their CI/CD pipelines.

Get Started

We encourage all developers, researchers, and HPC enthusiasts to explore the feelpp.benchmarking framework. By using this tool, you can gain actionable insights into your application’s performance and contribute to its growth and enhancement.