Lay Summary

Matrix multiplication is the foundational mathematical operation behind modern artificial intelligence and scientific computing. When distributing these computations across systems with multiple GPUs, transferring data between chips creates significant performance bottlenecks. This paper introduces “Uncut-GEMMs,” a communication-aware algorithm designed to optimize matrix multiplication on multi-GPU nodes. By minimizing unnecessary data movement across interconnected hardware, the approach maximizes computational throughput. The result is a faster, more hardware-efficient execution framework for data-intensive AI workloads and high-performance numerical simulations. (77 words)

Full Abstract

General Matrix Multiplication (GEMM) is one of the most common kernels in high-performance computing (HPC) and machine-learning (ML) applications, frequently dominating their execution time, rendering its performance vital. Modern multi-GPU compute nodes feature high computational capacity, but communication bottlenecks between GPUs prevent standard distributed GEMM implementations from achieving theoretical peak performance. In this paper, we present Uncut-GEMMs, a static-scheduling GEMM library optimized for multi-GPU compute nodes. We propose a communication-aware scheduling framework that minimizes communication volume, maximizes interconnect utilization via fast routing, and overlaps data transfers with computation. Integrated into our PARALiA framework, Uncut-GEMMs eliminates runtime scheduling overheads and delivers near-optimal performance across arbitrary matrix sizes, shapes, and layouts.

Metadata

• Publication Date: September 2024
• Author: Petros Anastasiadis, Nikela Papadopoulou, Nectarios Koziris, Georgios Goumas
• DOI: 10.1109/CLUSTER59578.2024.00020