How 2-bit Quantization Really Works: Inside the DeepSeek Anti-Rax Recipe

2-bit quantization of DeepSeek models consistently draws skepticism, so this session breaks down why heavily quantized models can still perform well. Recorded during a walk in Greenwich, it explains the mechanics behind modern quantization recipes and reports on ongoing experiments comparing the original and quantized models on agentic tasks.

■ Why Quantization Works Better Than Expected
- Why not every weight is compressed to 2 bits: attention projections, shared experts and router gates kept at 8-bit or 16-bit
- The role of model scale and built-in redundancy
- Quantization-aware training in DeepSeek V4 and V4 1.1

■ Inside the Quantization Recipe
- Grouped quantization: how small weight groups avoid needing to cover the full BF16 range
- Magnitude pattern tables (IQ2 XXS / IQ2_XXS): mapping eight weights to a shared magnitude value plus sign storage
- Origin of the magnitude patterns and their approximation role

■ Importance Matrix and Practical Testing
- Building an importance matrix with a calibration dataset to prioritize which weights to preserve
- Difficulties of isolating quantization differences on Terminal Bench agent tasks
- Notes on sampling behavior at inference time

If you work with local LLMs, run open-weight models, or want a clearer mental model of what happens when a 16-bit model becomes a 2-bit one, this gives you the vocabulary and intuition to judge quantization claims. It also shows what is involved in benchmarking the real-world cost of quantization on agent-style workloads.