How Model Distillation Shrinks Large AI Models for Phones and Laptops

This video explains model distillation, the technique used to transfer knowledge from a large AI model into a smaller one so it can run on devices with limited memory and compute. It covers why smaller parameter counts matter, how teacher-student training works, and why soft labels help a student model learn efficiently.

■ Model Distillation Basics
- What "smaller model" means: parameter counts, memory, latency, and cost
- The teacher and student roles in the distillation setup

■ How LLMs Predict Tokens
- Generating text one token at a time
- Assigning probabilities across the full vocabulary
- Hard labels versus ground truth when training the teacher

■ Soft Labels and Dark Knowledge
- Training the student on the teacher's output distribution instead of hard labels
- A breakfast example showing reasonable versus off-target alternatives

■ Trade-offs and Takeaways
- Balancing capability against efficiency
- What the student gains and gives up compared to the teacher

Those who want to understand how capable AI models are compressed for practical deployment will come away with a clear picture of the workflow, the reasoning behind soft labels, and the trade-offs involved.