Rethinking the Role of Temperature in Large Language Model Distillation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/lhchau/rethinking-kd-temperature-llm
Area: LLM Reasoning
Keywords: Large Language Model Distillation, Knowledge Distillation Temperature, Forward KL Divergence, Reverse KL Divergence, Soft Label Distribution
TL;DR¶
This paper revisits the neglected role of distillation temperature in LLMs, revealing that temperature inherently changes the supervision targets for Forward KL while primarily rescaling optimization gradients for Reverse KL, thereby overturning the conventional belief that RKL intrinsically outperforms FKL and demonstrating that simple KL objectives with tuned temperature match or beat recent SOTA distillation frameworks.
Background & Motivation¶
In the knowledge distillation (KD) of large language models, Reverse Kullback–Leibler (RKL) divergence has become the standard objective of choice, underpinning prominent frameworks such as MiniLLM and DistilLM. The prevailing narrative in the literature asserts that RKL's mode-seeking behavior prevents student models from generating low-probability hallucinated tokens in low-density teacher regions. Conversely, Forward Kullback–Leibler (FKL) divergence has often been treated as intrinsically inferior due to its zero-avoiding, mean-seeking propensity, which supposedly forces resource-constrained student models to over-generalize across vast token vocabularies and degrade output generation quality.
However, this empirical consensus rests upon an overlooked experimental convention: almost all recent LLM distillation studies set the distillation temperature to a rigid default of \(\tau = 1\). Influenced by early sequence-level KD work in machine translation that reported optimal performance at \(\tau = 1\), subsequent research took the default softmax output as granted and completely eliminated temperature from the distillation objective design space. Unlike RKL, which samples transitions weighted by the student's own distribution, FKL penalizes student errors directly scaled by teacher probabilities. Consequently, under \(\tau = 1\), the extremely sharp, low-entropy distribution of an autoregressive LLM teacher suppresses non-dominant token probabilities, starving FKL of the inter-class "dark knowledge" that makes soft-target distillation effective in the first place. This raises a fundamental question: Is RKL truly superior for LLM distillation, or has FKL simply been systematically evaluated under an unfair, insufficiently softened distribution?
To address this foundational issue, this paper reintroduces distillation temperature into the theoretical and empirical analysis of language model compression, examining how temperature fundamentally reshapes distributional alignment. Core idea: theoretically prove that both FKL and RKL asymptotically converge to identical teacher-student logit matching as temperature approaches infinity, whereas in practical low-to-medium temperature regimes, temperature enriches FKL with crucial non-target supervision signals while merely rescaling RKL gradients, an asymmetric mechanism that enables temperature-tuned FKL to consistently overturn and surpass RKL across benchmarks.
Method¶
Overall Architecture¶
The paper investigates the impact of temperature from both theoretical formulation and broad empirical benchmarks. The overall methodology progresses through three interconnected phases: first, asymptotic gradient expansion at high temperatures to prove mathematical equivalence with mean squared logit matching; second, low-temperature gradient decomposition to uncover the asymmetric mechanics between probability-matching targets and gradient-variance conditioning; and third, generalized temperature scaling across diverse KL-based distillation families under standardized evaluation protocols.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Inputs: Teacher Logits zt and Student Logits zs"] --> B["High-Temperature Asymptotic Analysis: FKL-RKL Logit Matching Equivalence<br/>Proving both gradients reduce to 1/(Vτ²)(zs−zt) as τ→∞"]
B --> C["Low-Temperature Mechanistic Dissection: Asymmetric Target and Gradient Dynamics<br/>FKL gains non-dominant dark knowledge vs RKL receives gradient rescaling"]
C --> D["Generalized Temperature Scaling on KL Families: Broad Re-evaluation<br/>Encompassing FKL, RKL, Sym-KL, JS, SFKL, SRKL, and AKL"]
D --> E["Outputs: Optimal Temperature-Tuned Student and Fair Objective Benchmarks"]
Key Designs¶
1. High-Temperature Asymptotic Analysis: FKL-RKL Logit Matching Equivalence
Prior theoretical explanations of the divergence between FKL and RKL frequently relied on idealized assumptions such as unconstrained model capacity or exact global convergence. This work provides an analytical resolution by deriving first-order Taylor expansions of both objectives directly in the finite logit space. Assuming centered teacher and student logits over vocabulary \(V\) (such that \(\sum_i z_{ti} = \sum_i z_{si} = 0\)), temperature-scaled distributions \(p_\tau = \text{softmax}(z_t/\tau)\) and \(q_\tau = \text{softmax}(z_s/\tau)\) expand as \(\exp(z_i/\tau) = 1 + z_i/\tau + O(\tau^{-2})\). Substituting these expansions into the gradient formulations yields: $$ \nabla_{z_{si}} \mathcal{L}{\text{FKL}} = \frac{1}{\tau} (q) $$ $$ \nabla_{z_{si}} \mathcal{L}} - p_{\tau i}) = \frac{1}{V \tau^2}(z_{si} - z_{ti}) + O(\tau^{-3{\text{RKL}} = \frac{q}}{\tau} \left(\log \frac{q_{\tau i}}{p_{\tau i}} - \mathcal{L{\text{RKL}}\right) = \frac{1}{V \tau^2}(z) $$ This proposition demonstrates that high distillation temperatures eliminate any distinction between mean-seeking and mode-seeking behaviors, reducing both objectives to identical teacher-student logit matching. Consequently, any empirical divergence between FKL and RKL observed in practice must stem strictly from their differing dynamics in the finite, low-temperature regime rather than an intractable theoretical conflict.} - z_{ti}) + O(\tau^{-3
2. Low-Temperature Mechanistic Dissection: Asymmetric Target and Gradient Dynamics
In practical temperature regimes (e.g., \(\tau \in [1, 4]\)), temperature exerts fundamentally distinct effects on the two objectives. For FKL, the target distribution \(p_\tau\) explicitly defines the optimization ground truth. The derivative \(\frac{\partial p_{\tau i}}{\partial \tau} = \frac{p_{\tau i}}{\tau} (\mathbb{E}_{j \sim p_\tau}[z_{tj}] - z_{ti})\) indicates that raising \(\tau\) flattens the distribution by transferring substantial probability mass from dominant top tokens to plausible alternative tokens, thereby directly enriching the semantic supervisory signal. In contrast, for RKL, defining the logit gap as \(\Delta_i = z_{si} - z_{ti}\), its gradient reduces to: $$ \nabla_{z_{si}} \mathcal{L}{\text{RKL}} = \frac{q}}{\tau^2} (\Delta_i - \mathbb{E{j \sim q\tau}[\Delta_j]) $$ This reveals that RKL is driven by student-weighted logit gaps rather than fitting a target probability distribution. Temperature does not alter the underlying teacher target for RKL; instead, it primarily rescales the gradient magnitude by \(1/\tau^2\) while mildly smoothing the student weights \(q_{\tau i}\). Hence, temperature changes what FKL learns by altering the supervision distribution, but mainly changes how RKL learns by reconditioning optimization scales. This fundamental asymmetry explains why FKL experiences dramatic performance improvements when temperature reveals non-target tokens, whereas RKL exhibits marginal gains or even optimization instability.
3. Generalized Temperature Scaling on KL Families: Broad Re-evaluation
Recognizing that modern LLM distillation proposals frequently innovate by designing hybrid or skewed divergences (such as Jensen-Shannon JS, skewed KL forms SFKL/SRKL, Adaptive KL AKL, \(\alpha\)-\(\beta\) divergence ABKD, and diversity-aware DRKL), this work expands temperature scaling across the entire spectrum of KL objectives. Under a systematic grid search of \(\tau \in \{1.5, 2.0, 2.5, 3.0, 3.5, 4.0\}\), the authors demonstrate that all objectives containing forward probability matching components benefit substantially from higher temperatures. This re-evaluation shifts the performance rankings across benchmarks, showing that sophisticated objective modifications often appear superior at \(\tau = 1\) simply because their baselines were artificially handicapped by cold distributions.
Loss & Training¶
In the standard off-policy instruction-following setting, teacher models are fine-tuned on the 14k training subset of Databricks-Dolly-15k and generate fixed response sequences for distillation: - For vocabulary size \(V\), temperature-scaled probabilities \(p_\tau = \text{softmax}(z_t / \tau)\) and \(q_\tau = \text{softmax}(z_s / \tau)\) are computed; - Forward objectives optimize \(\mathcal{L}_{\text{FKL}} = D_{\text{KL}}(p_\tau \parallel q_\tau)\) while reverse objectives minimize \(\mathcal{L}_{\text{RKL}} = D_{\text{KL}}(q_\tau \parallel p_\tau)\); - Symmetric KL is balanced as \(0.5 \mathcal{L}_{\text{FKL}} + 0.5 \mathcal{L}_{\text{RKL}}\); skewed objectives SFKL/SRKL adopt skew parameter \(\alpha = 0.1\); - Consistent training hyper-parameters are applied: batch size 32, 20 training epochs, max sequence length 512, with learning rate \(5 \times 10^{-4}\) for GPT-2 Base and \(5 \times 10^{-5}\) for OPT-1.3B; evaluation uses greedy decoding with decoding temperature set to 1.0.
Key Experimental Results¶
Main Results¶
The framework was evaluated across five instruction-following benchmarks (Dolly, Super-NI, Unnatural Instructions, Self-Instruct, and Vicuna) using GPT-2 (GPT-2 XL 1.5B \(\to\) GPT-2 Base 120M) and OPT (OPT 6.7B \(\to\) OPT 1.3B) setups. The table below presents the ROUGE-L scores (mean ± std over five random seeds) comparing default \(\tau=1.0\) against temperature-tuned variants:
| Teacher \(\to\) Student | Objective | Dolly | Super-NI | UnNI | Avg. | \(\Delta\)Avg. |
|---|---|---|---|---|---|---|
| GPT-2 XL \(\to\) Base | Teacher Upper Bound (GPT-2 XL) | 27.00±0.19 | 26.46±0.41 | 31.10±0.06 | 28.19 | – |
| FKL (\(\tau=1\)) | 23.80±0.55 | 16.27±0.24 | 19.03±0.09 | 19.70 | – | |
| FKL + \(\tau\) | 25.83±0.24 | 23.78±0.26 | 26.97±0.12 | 25.53 | +5.83 | |
| RKL (\(\tau=1\)) | 24.67±0.13 | 21.03±0.21 | 23.94±0.12 | 23.21 | – | |
| RKL + \(\tau\) | 25.00±0.07 | 21.41±0.24 | 24.68±0.34 | 23.70 | +0.49 | |
| Sym-KL (\(\tau=1\)) | 24.38±0.14 | 19.92±0.25 | 22.21±0.03 | 22.17 | – | |
| Sym-KL + \(\tau\) | 25.54±0.26 | 21.69±0.23 | 25.29±0.21 | 24.17 | +2.00 | |
| JS (\(\tau=1\)) | 24.08±0.14 | 19.66±0.10 | 22.17±0.14 | 21.97 | – | |
| JS + \(\tau\) | 25.51±0.11 | 24.34±0.40 | 27.03±0.16 | 25.63 | +3.66 | |
| AKL (\(\tau=1\)) | 21.83±0.27 | 15.40±0.16 | 18.06±0.16 | 18.43 | – | |
| AKL + \(\tau\) | 25.35±0.19 | 23.63±0.20 | 26.18±0.13 | 25.05 | +6.62 | |
| SOTA: AB (\(\tau=1\)) | 24.32±0.29 | 23.08±0.19 | 24.32±0.14 | 23.91 | – | |
| SOTA: DRKL (\(\tau=1\)) | 25.51±0.42 | 23.39±0.11 | 26.57±0.11 | 25.16 | – | |
| OPT 6.7B \(\to\) 1.3B | Teacher Upper Bound (OPT 6.7B) | 27.52±0.29 | 30.41±0.46 | 31.39±0.20 | 29.77 | – |
| FKL (\(\tau=1\)) | 26.07±0.65 | 22.11±0.38 | 26.82±0.10 | 25.00 | – | |
| FKL + \(\tau\) | 28.20±0.11 | 27.62±0.16 | 31.43±0.10 | 29.08 | +4.08 | |
| RKL (\(\tau=1\)) | 26.58±0.48 | 22.64±0.45 | 26.29±0.17 | 25.17 | – | |
| RKL + \(\tau\) | 27.64±0.13 | 26.96±0.48 | 31.29±0.27 | 28.63 | +3.46 | |
| AKL + \(\tau\) | 28.35±0.27 | 28.10±0.18 | 32.40±0.23 | 29.62 | +4.88 | |
| SOTA: DRKL (\(\tau=1\)) | 27.72±0.36 | 25.56±0.36 | 29.84±0.15 | 27.71 | – |
Ablation Study¶
To verify whether the asymmetric effect generalizes across distillation dynamics and modern model families, the authors conducted on-policy distillation experiments on GPT-2 and evaluated contemporary open-source architectures (Llama-3.2 and Qwen3):
| Protocol & Setting | Objective & Temperature | Dolly | SelfInst | Super-NI | UnNI | Vicuna | Avg. ROUGE-L |
|---|---|---|---|---|---|---|---|
| On-Policy KD | FKL (\(\tau=1.0\)) | 22.03±0.86 | 10.29±0.40 | 17.42±0.18 | 20.11±0.10 | 14.89±0.43 | 16.95 |
| (GPT-2 Base) | FKL (\(\tau=2.0\)) | 24.16±0.11 | 11.81±0.62 | 20.96±0.05 | 23.64±0.14 | 15.20±0.43 | 19.15 |
| FKL (\(\tau=3.0\)) | 25.30±0.30 | 12.11±0.25 | 23.70±0.20 | 26.79±0.26 | 16.62±0.58 | 20.90 | |
| FKL (\(\tau=3.5\)) | 25.32±0.07 | 12.40±0.22 | 24.92±0.44 | 27.20±0.02 | 16.35±0.29 | 21.24 | |
| RKL (\(\tau=1.0\)) | 24.12±0.25 | 12.20±0.38 | 22.43±0.09 | 24.93±0.06 | 16.14±0.37 | 19.97 | |
| RKL (\(\tau=1.5\)) | 23.96±0.13 | 12.00±0.62 | 23.16±0.26 | 26.78±0.16 | 16.14±0.37 | 20.41 | |
| RKL (\(\tau=3.5\)) | 24.00±0.18 | 10.66±0.14 | 20.41±0.38 | 23.48±0.22 | 15.78±0.30 | 18.87 | |
| Modern Architecture | FKL (\(\tau=1\)) | 25.86±0.54 | 15.46±1.03 | 28.14±0.25 | 29.52±0.13 | 15.53±0.04 | 22.90 |
| (Llama 3.2 3B \(\to\) 1B) | FKL + \(\tau\) | 27.34±0.23 | 18.73±0.24 | 33.04±0.29 | 34.07±0.14 | 17.11±0.66 | 26.06 (+3.16) |
| RKL (\(\tau=1\)) | 28.85±0.23 | 18.90±0.10 | 34.94±0.15 | 35.90±0.09 | 17.44±0.90 | 27.21 | |
| RKL + \(\tau\) | 27.61±0.89 | 17.44±0.93 | 33.57±0.42 | 34.37±0.11 | 17.10±0.53 | 26.02 (-1.19) |
Key Findings¶
- Complete Reversal of FKL vs RKL: At default \(\tau=1\), RKL consistently outperforms FKL (e.g., 19.97 vs 16.95 on on-policy GPT-2 Base, and 23.21 vs 19.70 on off-policy). However, tuning \(\tau\) to \(2.5-3.5\) turns FKL into the dominant objective, soaring to 21.24 and 25.53 respectively, completely inverting the comparative ranking.
- RKL Fails to Benefit from Temperature: Across both scale and model family, RKL shows marginal sensitivity to \(\tau\) (+0.44 to +0.49 on GPT-2 Base) and degrades sharply on modern architectures (dropping by -1.19 on Llama-3.2 and -1.50 on Qwen3), confirming the theoretical prediction that temperature does not inject new supervision targets into RKL.
- Simple KL Objectives Outperform Recent SOTA: Basic FKL+\(\tau\) (25.53) and JS+\(\tau\) (25.63) outperform specialized objectives like ABKD (23.91) and DRKL (25.16) on GPT-2 Base. On OPT-1.3B, AKL+\(\tau\) reaches 29.62, beating DRKL (27.71).
- LLM-as-a-Judge Corroboration: Evaluated by Claude Haiku 4.5 using MT-Bench protocols on OPT, FKL+\(\tau\) achieved a 57.7% pairwise win rate against \(\tau=1\) (+4.3 pointwise score), whereas RKL+\(\tau\) won only 42.3%, demonstrating that temperature's benefits translate directly into qualitative instruction-following coherence.
Highlights & Insights¶
- Disproved inherent theoretical incompatibility: Through asymptotic expansion as \(\tau \to \infty\), the work proves both FKL and RKL converge to identical mean squared logit matching, confirming that the divide between mode-seeking and mean-seeking is strictly a localized phenomenon of low-temperature optimization.
- "What is learned vs how it is learned" duality: The paper provides a clear intuition: temperature feeds FKL richer non-dominant token supervision by smoothing teacher targets, while it merely acts as a gradient scale dampener for RKL.
- Critical wake-up call for LLM benchmarking: Researchers proposing novel distillation loss functions must include temperature in their hyperparameter search space, as evaluating at \(\tau=1\) artificially depresses baseline performance and yields misleading comparisons.
Limitations & Future Work¶
- Computational limits restricted full-model distillation experiments to models up to 7B parameters (GPT-2 XL, OPT-6.7B); evaluating ultra-large teachers (70B+) remains to be validated.
- Modern architecture experiments (Llama-3.2 and Qwen3) relied on LoRA parameter-efficient fine-tuning rather than full fine-tuning, leaving open the interaction between very large token vocabularies (e.g., 128k+) and temperature scaling during full pre-training/distillation.
- Benchmarks focused primarily on English instruction-following tasks; assessing whether elevated temperatures risk introducing semantic drift in exact-reasoning domains (formal logic, competitive coding, math proofs) represents a natural next investigation.
Related Work & Insights¶
- vs Seq-KD (Kim & Rush, 2016): Seq-KD historically established \(\tau=1\) as the de facto standard for sequence distillation in machine translation; this paper demonstrates that modern instruction-tuned LLMs possess vastly richer output distributions where \(\tau > 1\) is indispensable.
- vs MiniLLM (Gu et al., 2024): MiniLLM championed RKL to alleviate generation degeneration under the assumption that FKL is fundamentally flawed; this paper shows that FKL's apparent flaws were an artifact of evaluated temperatures rather than intrinsic mathematical limitations.
- vs DistilLM (Ko et al., 2024): DistilLM introduced skewed divergences (SFKL/SRKL) to manually interpolate between mode-seeking and mode-covering; this work proves that simply tuning temperature on vanilla FKL or JS achieves superior performance without adding heuristic skew hyperparameters.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates a precise theoretical and empirical challenge to entrenched community consensus by rigorously examining the distillation temperature mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across off-policy and on-policy KD, multiple model generations (GPT-2, OPT, Llama-3.2, Qwen3), ROUGE-L, and LLM-as-a-Judge protocols.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear progression from mathematical asymptotic proofs to empirical reversals, accompanied by insightful conceptual taxonomy.
- Value: ⭐⭐⭐⭐⭐ Provides an indispensable benchmarking guideline that will re-anchor future LLM knowledge distillation research.