How KV Cache Data Movement Shapes Accelerator Efficiency During LLM Generation

This video explains how the KV cache works in large language models and why efficiently moving that cached data between memory levels matters as much as raw compute. It traces the capacity-versus-locality problem across three accelerators and highlights which software or hardware layer orchestrates the movement in each case. Through concrete architecture examples, it connects memory hierarchy decisions to decode efficiency and throughput.

■ Understanding the KV Cache
- Why each generated token must review the full conversation, and how reused keys and values avoid redundant computation
- The growth of cache notes as context expands, and the trade-off between large slow memory and small fast working sets

■ Orchestration Across Accelerators
- GPU (data center): HBM and on-chip memory, with serving runtimes such as vLLM allocating KV storage in fixed-size blocks and attention kernels shaping access patterns
- TPU: HBM and VMEM, with the compiler and kernels handling tiling and pipelining
- IBM Spyre accelerator: LPDDR5 and per-core SRAM scratchpads, with compiler-scheduled loads and stores

■ Why It Matters
- The same attention state behaves differently depending on where it lives and which layer controls its movement

This is aimed at engineers, researchers, and practitioners working on inference serving and accelerator design. Viewers will gain a framework for comparing accelerators beyond arithmetic throughput, focusing on how each system moves model memory states. The next time you evaluate hardware for generation workloads, consider asking how it handles memory movement rather than only how much math it can perform.