Training Session: Containerization, CI/CD, and Benchmarking Solutions for HPC

Overview

In today’s fast-changing world of high-performance computing (HPC) and urban energy modeling, staying current is essential for researchers, developers, and policymakers. As part of CoE HiDALGO2, the Urban Building Pilot team is pleased to offer a three-day training session on Containerization, CI/CD, and Benchmarking Solutions for HPC. In collaboration with the Exa-MA project (via NumPEx), HLRS, and NCC Czechia, this session is designed to provide practical tools and methodologies for working with advanced HPC platforms in urban energy simulation.

> Training event official site.

> Registration form

During the session, you will learn how to create containers for HiDALGO2 codes and scientific HPC applications and implement CI/CD workflows using GitHub Actions, GitLab Runners, and JacamarCI. The course also covers setting up benchmarking pipelines with tools such as feelpp.benchmarking and reframe-hpc to evaluate and improve performance.

Through a mix of lectures and hands-on workshops, you’ll gain insights into managing HPC environments—integrating job schedulers, environment modules, and efficient data handling techniques—to optimize and maintain your computational workflows. With the Urban Building Pilot team’s extensive experience in HPC and CI/CD on EuroHPC supercomputers, this training aims to equip you with practical skills for large-scale simulations.

Objectives

Containerize HPC Applications: Build, manage, and optimize container images using Docker and Singularity/Apptainer.
Implement CI/CD Workflows: Automate builds, tests, and deployments with GitHub Actions, GitLab Runners, and JacamarCI.
Design Benchmarking Pipelines: Use feelpp.benchmarking and reframe-hpc to assess performance and scalability.
Optimize HPC Workloads: Leverage HPC environment modules, job schedulers, and best practices for scalable, reproducible workflows.

Format & Technical Setup

Dates: March 24–26, 2025
Delivery Format:
- Online via Zoom: For video conferencing and live presentations.
- Slack: For real-time chat, Q&A, and troubleshooting.
- GitHub Classroom: For hands-on code distribution and assignment management.
- Recommended IDE: Visual Studio Code (alternatives such as Emacs, PyCharm, or vim are acceptable).
HPC Resources & Tools:
- HPC System: Karolina (NCC Czechia)
- Account & Access: Ensure you have a Karolina HPC account with SSH access.
- Containerization Tools: Docker (for local experiments, optional) and Singularity/Apptainer (installed on Karolina).
- CI/CD Platforms: GitHub (with GitHub Actions) and GitLab (with GitLab Runner); JacamarCI is also used.
- Benchmarking Frameworks: feelpp.benchmarking and reframe-hpc.

Target Audience

Researchers, scientists, and engineers focused on HPC workflows.
HPC administrators and DevOps professionals seeking automation strategies for cluster environments.
Software developers aiming to improve reproducibility and performance for scientific applications.

Prerequisites

Essential:
- Basic scripting skills in Bash and Python.
- Familiarity with Git and GitHub or GitLab and YAML.
Recommended:
- Experience with Docker and/or Singularity/Apptainer.
- Basic knowledge of HPC schedulers (e.g., Slurm) and CI/CD concepts.

Detailed Timetable

Day 1: Foundations of Containerization and CI/CD for HPC

09:00 – 09:30: Welcome, course objectives, and participant introductions.
09:30 – 11:00: Introduction to containerization and CI/CD workflows using GitHub Actions.
11:00 – 11:15: Coffee Break.
11:15 – 12:30: Hands-on: Building a simple CI pipeline with GitHub Actions.
12:30 – 13:30: Lunch Break.
13:30 – 15:00: Building and pushing container images for HPC applications (Docker vs. Singularity/Apptainer).
15:00 – 15:15: Coffee Break.
15:15 – 16:30: Hands-on: Containerizing an HPC code (testing locally with Docker and deploying on Karolina with Singularity).
16:30 – 17:00: Q&A and wrap-up.

Day 2: Advanced CI/CD, Benchmarking, and Performance Optimization

09:00 – 09:30: Recap of Day 1 and Q&A.
09:30 – 11:00: Advanced CI/CD techniques using GitLab Runner and JacamarCI.
11:00 – 11:15: Coffee Break.
11:15 – 12:30: Hands-on: Setting up a multi-stage HPC pipeline integrating with a job scheduler.
12:30 – 13:30: Lunch Break.
13:30 – 15:00: Introducing benchmarking strategies using feelpp.benchmarking and reframe-hpc.
15:00 – 15:15: Coffee Break.
15:15 – 16:30: Hands-on: Designing a benchmarking pipeline and collecting performance data.
16:30 – 17:00: Q&A and discussion.

Day 3: Integrating Advanced Benchmarking and Performance Tuning

09:00 – 09:30: Recap of Day 2 and Q&A.
09:30 – 11:00: Building complex benchmarking pipelines integrating reframe-hpc and feelpp.benchmarking.
11:00 – 11:15: Coffee Break.
11:15 – 12:30: Hands-on: Multi-run benchmarking setup and automated report generation.
12:30 – 13:30: Lunch Break.
13:30 – 15:00: Best practices for scalable, reproducible HPC workflows (environment management and performance tuning).
15:00 – 15:15: Coffee Break.
15:15 – 16:30: Final hands-on session and open discussion on HPC container orchestration.
16:30 – 17:00: Closing remarks and feedback.

Trainer Information

Christophe Prud’homme: Professor in Applied Mathematics; Founder of Cemosis; main developer of Feel++; expert in HPC and CI/CD pipeline design.
Javier Cladellas: Junior Research Software Engineer; lead developer of feelpp.benchmarking.
Vincent Chabannes: Research Engineer specializing in HPC infrastructure and software development.
Sameer Haroon (HLRS): Research Engineer with expertise in systems engineering, HPC provisioning, and domain-specific languages.

Registration & Additional Information

Maximum number of participants: 30