Understanding the Memory Wall: Arithmetic Intensity and the Roofline Model for AI Workloads

The presentation explains why increasing compute power alone does not guarantee faster AI workloads, focusing on the memory wall and the widening gap between compute and memory bandwidth. It introduces arithmetic intensity and the roofline model as frameworks for determining whether a workload is compute bound or memory bound. The discussion uses large language model inference stages and serving frameworks to illustrate how batching can improve performance.

■ The Memory Wall
- Compute capabilities and memory bandwidth growth from 2006 to 2026
- The widening gap and its effect on workload speed

■ Arithmetic Intensity and the Roofline Model
- Definition of arithmetic intensity as FLOPs per byte
- Compute-bound versus memory-bound regimes
- Ridge point calculation from peak compute and memory bandwidth

■ Workload Examples
- Decode stage: low arithmetic intensity, memory bound
- Prefill stage: high arithmetic intensity, compute bound
- Batching and continuous batching in LLM serving

■ Practical Takeaways
- Estimate workload arithmetic intensity before benchmarking
- Compare workload characteristics with hardware specifications
- Use roofline modeling to identify optimization priorities

It is aimed at engineers, ML practitioners, and technical decision-makers working with model training, serving, or AI infrastructure. Viewers can gain a conceptual framework for diagnosing performance bottlenecks and deciding whether to focus on compute or memory improvements.