Lay Summary

Deploying Large Language Models (LLMs) requires immense computational power and memory, making real-time inference costly and slow. “Sparsification”—a technique that removes unnecessary neural network connections—is widely used to reduce model size and speed up processing. This paper conducts a performance analysis of sparsified transformer architectures during LLM inference. The authors evaluate how pruning weights affects memory bandwidth, computation speed, and accuracy across different hardware. Their findings shed light on optimizing sparsified models to achieve faster, more energy-efficient AI text generation. (82 words)

Full Abstract

Serving Large Language Models (LLMs) at scale poses unprecedented challenges for memory bandwidth and latency during inference. Sparsification—pruning structural or unstructured model weights—promises significant latency reductions, yet hardware acceleration of sparse matrix operations remains challenging. In this paper, we present a performance breakdown of sparsified transformer models executing inference on modern GPU architectures. We benchmark sparse matrix-vector operations, quantify memory bandwidth utilization, and evaluate accuracy-latency trade-offs. Our analytical insights highlight key software and hardware requirements needed to translate structural network sparsity into tangible speedups.

Metadata

• Publication Date: 2024
• Author: Ioanna Tasou, Petros Anastasiadis, Panagiotis Mpakos, Dimitrios Galanopoulos, Nectarios Koziris, Georgios Goumas
• Journal / Conference: ACM International Conference on Computing Frontiers / IEEE Cluster