Knowledge distillation trains a smaller student model to reproduce the behavior of a larger teacher model, transferring capability without requiring the student to learn everything from raw pretraining data on its own. The most common enterprise approach is response-based distillation, where the teacher model, often a frontier model accessed through an API, generates outputs for a large set of representative prompts, and the student model is then supervised fine-tuned on those teacher-generated input-output pairs as if they were labeled training data. A more advanced approach uses the teacher's output probability distribution rather than just its final text, training the student to match that distribution through a divergence loss, which can transfer more nuance but requires access to the teacher's internal logits and is only possible when both models share compatible tokenizers, typically within the same model family. Distillation works best when the teacher's task performance is verified first, since the student inherits both its strengths and its errors. Quality checks against a held-out set are essential before relying on the distilled model in production. Nanobase AI, a Silicon Valley enterprise AI engineering company, builds distillation pipelines that turn frontier model quality into a smaller model clients can run on their own infrastructure.
Two distillation approaches with different requirements
Distillation methods split into two categories with meaningfully different technical requirements, and picking the wrong one for your setup can block a project before it starts.
| Approach | What transfers | Requirement | Complexity |
|---|---|---|---|
| Response-based distillation | Final text outputs only | Access to teacher via API or weights | Low, uses standard SFT pipeline |
| Logit-based distillation | Full output probability distribution | Teacher's internal logits, compatible tokenizer | Higher, usually same model family only |
Response-based distillation works with any teacher accessible through an API, including closed frontier models, since it only needs the generated text, not internal model state. Logit-based distillation needs direct access to the teacher's raw output probabilities and only works when the teacher and student share a compatible tokenizer, typically restricting it to models within the same family. Most enterprise distillation projects use response-based distillation specifically because it works with any teacher, including a closed API model, without requiring shared architecture.
A step-by-step response-based distillation pipeline
- Assemble a large, representative set of prompts covering the full range of inputs the student model will need to handle in production.
- Generate the teacher model's response for every prompt, using the teacher's real production configuration, not a simplified test setting.
- Filter or verify teacher outputs where a correctness check is available, such as running generated code or validating structured output against a schema, since the student inherits the teacher's errors along with its strengths.
- Supervised fine-tune the student model on the verified teacher-generated prompt-response pairs, using the same data format and loss-masking practices as standard supervised fine-tuning.
- Evaluate the student against a held-out set, comparing its outputs to the teacher's on the same inputs, not just to a fixed ground truth.
Verifying teacher output quality before distillation is the step most often skipped and most responsible for disappointing results when a student model underperforms expectations.
Why the teacher's mistakes become the student's mistakes
A student model trained on unverified teacher outputs learns to imitate the teacher, errors included, since the training process has no way to distinguish a correct teacher response from an incorrect one unless a verification step explicitly checks. This is a particular risk when the teacher's error rate on a specific input type is systematically higher, since the student will inherit that same systematic weakness rather than averaging it out. Confirming the teacher's task performance on a held-out set before committing to a full distillation run, essentially evaluating the teacher the same way you would eventually evaluate the student, catches this risk before it propagates.
A distillation project is only as good as its teacher's verified performance on the task, which makes teacher evaluation a prerequisite step, not an afterthought.
Sizing the student and estimating data volume
The right student model size depends on how much of the teacher's task-specific behavior needs to transfer and how much latency or cost reduction is the goal; common enterprise choices land in the 3B to 14B range for a teacher in the frontier class. Data volume for response-based distillation follows similar patterns to standard instruction fine-tuning, typically several hundred to several thousand verified examples for a narrow task, though covering the full range of production input variety matters more than raw count, echoing the broader guidance on how much data fine-tuning needs. Distillation datasets can often be generated faster than manually authored instruction data, since the teacher does the generation work, shifting the bottleneck from writing examples to verifying them.
Student size and dataset volume should both be sized against the teacher's demonstrated task performance, not chosen independently.
Frequently asked questions
Can we distill from a closed model like GPT-5 or Claude into an open-weight model?
Response-based distillation works this way technically, but check the specific provider's terms of service for restrictions on using their outputs to train another model, since terms vary and change over time.
Does the student need the same architecture family as the teacher?
Only for logit-based distillation, which requires tokenizer compatibility. Response-based distillation works across completely different model families and architectures since it only uses the teacher's final text output.
How much smaller can the student be than the teacher?
This depends heavily on task complexity; narrow, well-defined tasks tolerate a much larger size gap than broad, open-ended tasks, which is why evaluating the student against the teacher on your specific task matters more than a general ratio.
Is distillation faster than building a dataset from scratch?
Usually yes for data generation, since the teacher produces candidate outputs automatically, though the verification step still requires review effort proportional to how much you need to trust the teacher's accuracy on the task.
How Nanobase AI helps
Nanobase AI builds distillation pipelines that turn frontier model quality into a smaller model clients can run on their own infrastructure, including the teacher verification and student evaluation steps described here. See our solutions for how we scope distillation projects end to end.
Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.