Model distillation
Last verified: June 2026· definition
The recipe is straightforward: run a strong, expensive model over a representative set of inputs, keep its outputs as training data, and fine-tune a much smaller model on those pairs. The small model never learns to be generally brilliant — it learns to do your one task the way the big model did it.
For a bounded task with a stable definition of “good” — classification, extraction, routing, a fixed-format summary — a distilled small model often lands within a point or two of the teacher at a small multiple of the cost, and much lower latency.
It is a poor fit where the task keeps changing or needs fresh knowledge; retraining is slower to iterate than editing a prompt or an index. Distillation is a late-stage cost lever: prove the task is stable, prove quality with evals, then distill.
It only works on bounded tasks
Distillation transfers behaviour on a specific distribution of inputs, not general capability. That makes it a strong fit for classification, extraction, routing, and fixed-format generation, where the task is stable and success is checkable. It is a poor fit for open-ended work, anything requiring broad world knowledge, or tasks whose definition shifts, because the student has only ever learned to imitate the teacher on the inputs you showed it.
The training data is the whole project
Quality depends almost entirely on whether your input set represents real traffic, including the awkward cases. A student trained only on clean, typical examples performs well in testing and fails on the messy edges that make up a meaningful share of production. Sampling actual production inputs rather than constructing a synthetic set is the single most important decision, and it is where most disappointing results originate.
Distillation freezes a moment
The student learns the teacher as it was on the day you generated the data. When the task drifts, when new categories appear, or when the underlying model improves, the student does not follow — it needs regenerating and retraining. That maintenance obligation is real and recurring, and it should be weighed against simply routing to a smaller off-the-shelf model, which requires no retraining at all.
Common misconceptions
MythA distilled model is a smaller general-purpose model.
RealityIt is a specialist. It may perform well on your task and poorly on anything adjacent, because it was optimised for one distribution rather than trained for breadth.
MythDistillation is how you make a model cheaper.
RealityIt is one way, and rarely the first. Routing to an existing smaller model requires no training, no data pipeline, and no ongoing maintenance — worth exhausting before committing to distillation.
MythOnce distilled, you are done.
RealityThe student encodes the teacher at one point in time. Task drift, new input types, and improvements in available models all require regenerating data and retraining to keep the advantage.
Frequently asked questions
When is distillation worth it over just using a smaller model?
When you have tried a smaller off-the-shelf model and it does not meet your quality bar, but the task is narrow enough that imitation could close the gap. Routing to an existing small model costs nothing to try and has no maintenance burden, so it is the correct first experiment. Distillation earns its complexity when that fails and the volume is high enough that the saving justifies building and maintaining a training pipeline.
How much training data does distillation need?
Less than training from scratch, since the student is learning to imitate a specific behaviour rather than acquire general capability, but the amount varies with task complexity. What matters more than volume is coverage: the set must include the unusual and malformed inputs that occur in production, because those are where an under-covered student fails and where the failures are least visible in testing.
Does distillation reduce quality?
Usually somewhat, and on a bounded task the gap can be small enough not to matter. The right way to decide is to evaluate the student against the teacher on your own held-out production inputs and see whether the difference is acceptable for that route. A small drop on a low-stakes classification is a good trade; the same drop on a high-stakes route may not be.