Model distillation

The recipe is straightforward: run a strong, expensive model over a representative set of inputs, keep its outputs as training data, and fine-tune a much smaller model on those pairs. The small model never learns to be generally brilliant — it learns to do your one task the way the big model did it.

For a bounded task with a stable definition of “good” — classification, extraction, routing, a fixed-format summary — a distilled small model often lands within a point or two of the teacher at a small multiple of the cost, and much lower latency.

It is a poor fit where the task keeps changing or needs fresh knowledge; retraining is slower to iterate than editing a prompt or an index. Distillation is a late-stage cost lever: prove the task is stable, prove quality with evals, then distill.