Knowledge Distillation

Knowledge distillation trains a student model to match a teacher model. The teacher can be a larger model, an ensemble, a stronger checkpoint, or a model with access to a richer training signal.

Given teacher logits $z^{(T)}$ and student logits $z^{(S)}$, temperature-scaled probabilities are:

$$p_T(k\mid x)=\mathrm{softmax}\left(\frac{z_k^{(T)}}{\tau }\right),p_S(k\mid x)=\mathrm{softmax}\left(\frac{z_k^{(S)}}{\tau }\right)$$

where $\tau >0$ is the distillation temperature. Higher $\tau$ softens the teacher distribution and exposes similarity between classes.

The distillation loss is often a KL divergence or cross-entropy between teacher and student probabilities:

$${\mathcal{L}}_{\mathrm{KD}}={\tau }^2\mathrm{KL}\left(p_T(\cdot \mid x)\Vert p_S(\cdot \mid x)\right)$$

The ${\tau }^2$ factor is commonly used to keep gradient scale comparable across temperatures.

When hard labels $y$ are available, training can combine supervised and distillation losses:

$$\mathcal{L}=\alpha {\mathcal{L}}_{\sup }(p_S,y)+(1-\alpha ){\mathcal{L}}_{\mathrm{KD}}$$

where $\alpha \in [0,1]$ controls the hard-label versus teacher-signal tradeoff.

Distillation Targets

Distillation is broader than class-probability matching.

Target	Example	Main check
logits/probabilities	classifier compression	calibration and class slices
embeddings	representation transfer	retrieval or downstream task quality
sequences	language/model output imitation	exactness, diversity, hallucination
rankings	reranker or recommender distillation	ranking metrics and hard negatives
trajectories	agent/tool traces	completion and side-effect correctness
preferences	policy optimization	reward hacking and held-out preference data

The target determines the loss and verifier.

Teacher Boundary

The teacher is a data source. Its provenance matters.

$$\text{student claim}\le \text{teacher evidence}+\text{student evaluation evidence}$$

Teacher property	Why it matters
training data	may include benchmark or private data
calibration	soft labels may be overconfident
domain coverage	student inherits teacher blind spots
generation policy	synthetic outputs depend on decoding
hidden tools/retrieval	teacher answer may use unavailable evidence

Public notes should not imply ground truth if the teacher signal is only a model-generated label.

Evaluation Boundary

Metric	What it proves
teacher agreement	student imitates teacher
ground-truth metric	student solves task
calibration metric	probabilities remain meaningful
slice/OOD metric	rare cases and transfer behavior
cost/latency metric	compression or deployment benefit

Distillation is successful only if it improves the intended tradeoff, not merely agreement.

Uses

• Compress a large model into a smaller deployable model.
• Transfer behavior from an ensemble or stronger checkpoint.
• Train a specialized model from weak labels generated by a teacher.
• Distill generation, ranking, retrieval, or policy behavior into a cheaper model.

Risks

• The student can inherit teacher errors and biases.
• Teacher confidence can hide uncertainty if probabilities are poorly calibrated.
• Distillation may improve average performance while hurting rare cases.
• Synthetic teacher labels can leak benchmark or private information if the data source is not controlled.

Checks

• What is the teacher, and what data did it see?
• Are teacher outputs logits, probabilities, labels, rationales, trajectories, or preferences?
• Is the student evaluated against ground truth, not only teacher agreement?
• Are temperature, loss weights, and teacher checkpoint documented?
• Does distillation preserve calibration and out-of-distribution behavior?
• Are private teacher prompts, hidden datasets, or unpublished results omitted from public notes?
• Is teacher agreement separated from task correctness?
• Does the student improve the claimed cost, latency, or deployment constraint?

Related

• Supervised learning
• Fine-tuning
• Transfer learning
• Instruction tuning
• Preference optimization
• Cross-entropy loss
• Probability metrics
• Calibration