LLM Inference Explained: Serving Models from Prompt to Response

LLM inference is the process that turns a trained model's static weights into real-time responses for chatbots, code assistants, and agentic applications. This overview follows a request from tokenization and authentication through routing, batching, prefill, and decode, and explains why serving cost and latency matter over time. It focuses on serving mechanics rather than training, model architecture, or hardware procurement.

■ Inference engines and request flow
- Role of inference engines such as vLLM, Ollama, and llama.cpp in scheduling, GPU management, and request handling
- OpenAI-compatible endpoints and model swapping
- Tokenization and per-token prompt cost

■ Data center routing and batching
- Authentication and routing to suitable hardware
- Continuous batching and scheduling to maintain GPU utilization

■ Prefill and decode phases
- Prefill: compute-bound prompt reading and KV cache creation
- Decode: memory-bandwidth-bound token generation and probability selection
- TTFT and ITL as latency measures

The explanation is aimed at developers, ML engineers, and platform teams who need a clear picture of the LLM serving pipeline. Viewers can use it to identify where latency and cost arise and to evaluate batching, routing, and prefill/decode optimization choices in their own inference stack.