Lay Summary

Training machine learning models on multi-GPU systems often fails to achieve expected speedups due to hidden hardware bottlenecks, such as data transfer delays and sub-optimal memory usage. This paper focuses on profiling and optimizing multicard GPU workflows. The authors analyze how resource allocation, batch sizing, and communication strategies impact overall training speed. By identifying key execution bottlenecks and presenting tailored optimization techniques, the study provides practical recommendations for maximizing hardware efficiency and reducing training times for complex AI workloads. (82 words)

Full Abstract

Training state-of-the-art machine learning architectures across nodes with multiple GPU accelerators frequently encounters performance degradation due to interconnect congestion, thread synchronization delays, and memory bandwidth bounds. This paper conducts a comprehensive profiling study of multicard GPU workloads under PyTorch and TensorFlow execution environments. By utilizing hardware performance counters and profiling utilities, we isolate key overhead drivers during model execution. We propose targeted optimization strategies including gradient accumulation tuning, communication overlap, and custom memory management, demonstrating significant speed improvements on multi-GPU computing systems.

Metadata

• Publication Date: 2024
• Author: Marcin Lawenda, Kyrylo Khloponin, Krzesimir Samborski, Łukasz Szustak
• Journal: IEEE Transactions on Parallel and Distributed Systems / HPC Workshops