HiDALGO2 brings together teams and institutions across different countries and time zones, all working together to improve the scalability of several applications from the domain of Global Challenges on Europe’s HPC systems. Here, the concept of DevOps provides the cultural push, and the methodology of continuous integration and deployment (CI/CD) provides the unified framework needed for our teams to seamlessly integrate and advance the development of simulation codes and software infrastructure.
Read on if you are curious and want to discover what these buzzwords mean and how they help HiDALGO2 achieve its ultimate goal: overcoming environmental challenges such as increased flooding risks, wildfires, and air pollution.
DevOps, or Development Operations, is an umbrella term that describes the collection of changes that have taken place in the philosophy and methodology of building, shipping, and operating software systems. Before DevOps, operational teams were afraid to change production systems, and development teams often developed in a vacuum. DevOps paved the way for Development and Operations to work together and have each other’s backs, as it were.
One of the critical pillars of the DevOps movement is CI/CD, or Continuous Integration and Continuous Delivery/Deployment, a practice in software development that aims to make collaborating on code much easier across multiple and disjointed teams and components, simultaneously speeding up the pace of development, and dramatically reducing the number of bugs and errors that might make their way into the final product.
CI, or Continuous Integration, fundamentally means that all developers push code changes frequently and that multiple code changes are integrated often and tested automatically against a suite of regression tests. It ensures that every component of the code-base is always in sync with the rest of the development and that any potential bugs are caught as quickly as possible.
On the other hand, CD, or Continuous Delivery/Deployment focuses on upgrading and maintaining software, either to test or production platforms. Without solid CD practices in place, failed or buggy deployments and maintenance can result in frustrating, limited, or a complete lack of access to the desired services of the software system.
In HiDALGO2, we implement CI/CD across three fundamental levels:
1) CI/CD for Web Services: Here, we utilise Ansible, an automation platform, and the core concepts of integrating quickly and regularly to maintain our configuration and deployment scripts. It ensures that whenever there comes a time for an update in the main software site, e.g. if there is an official update by the developers of Wiki-js (the software HiDALGO2 uses for its internal and later also its public Wiki), our scripts will pull these changes and redeploy the software, including the many constituent modules and dependencies, all at the push of a button (or key press). Equally importantly, all deployments in HiDALGO2 aim to be idempotent, minimising the hard-to-trace, specific bugs relating to particular servers and hardware.
2) Internal CI/CD for Pilot Simulations: Here, we focus on building up a pipeline that takes code changes from the repositories of the individual Pilot simulations, merges them to build a release artifact, and couples this artifact together with our Workflow Orchestrators to deploy the simulations on HiDALGO2 HPC infrastructure, such as the Altair from PSNC. We use established tools like Bitbucket and Gitlab Runners to realise the CI/CD pipelines.
3) External CI/CD for Pilot Simulations: Finally, the third phase of the CI/CD will extend this unifying framework across Europe by connecting HiDALGO2 Use cases and Simulation Codes, such as the Urban Air Pollution pilot to the major European Supercomputing sites, e.g. LUMI in Finland.
In summary, HiDALGO2 demonstrates international collaboration focused on addressing global challenges through DevOps and Continuous Integration/Continuous Delivery (CI/CD) methods, which have optimised the project’s development procedures. The three-tiered implementation of CI/CD has enabled HiDALGO2 to make notable advancements in maintaining web services, orchestrating pilot simulations, and expanding its framework across Europe.
