Rethinking the Role of Temperature in Large Language Model Distillation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/lhchau/rethinking-kd-temperature-llm
Area: Multimodal VLM
Keywords: Knowledge Distillation, Distillation Temperature, Forward vs Reverse KL, LLM Alignment, Soft Target Calibration
TL;DR¶
This paper revisits the neglected role of temperature in LLM distillation, theoretically and empirically demonstrating that temperature exhibits an asymmetric effect: it fundamentally reshapes forward KL (FKL) supervision by activating dark knowledge across non-dominant tokens while primarily rescaling reverse KL (RKL) gradients, thereby completely overturning the widespread convention that RKL is intrinsically superior to FKL.
Background & Motivation¶
Knowledge distillation (KD) has long treated the distillation temperature \(\tau > 1\) as its cornerstone mechanism, flattening the teacher's prediction logits to redistribute probability mass from dominant classes onto semantically plausible alternative categories, thereby transferring rich inter-class relational dark knowledge. In computer vision and classical pre-LLM natural language processing (e.g., DistilBERT and TinyBERT), temperature scaling is an indispensable default. However, with the rise of instruction tuning and autoregressive sequence generation in large language models, early sequence-level KD empirical observations that \(\tau=1\) was optimal led subsequent landmark frameworks (such as MiniLLM, DistilLM, and their variants) to implicitly standardize on the default softmax temperature \(\tau=1\), virtually exiling temperature tuning from the design space of modern LLM distillation objectives.
This persistent benchmarking convention has severely biased the comparative understanding of forward Kullback-Leibler divergence (FKL, \(D_{KL}(p \parallel q)\)) versus reverse Kullback-Leibler divergence (RKL, \(D_{KL}(q \parallel p)\)). A prevailing consensus across modern LLM literature asserts that RKL consistently outperforms FKL due to its "mode-seeking" behavior, which purportedly prevents smaller student models from over-allocating probability mass to low-probability tokens. Yet, this empirical conclusion rests entirely upon evaluations conducted under \(\tau=1\). At default temperature, autoregressive teacher models produce near-degenerate, extremely peaked distributions with minimal entropy. Because FKL weights token updates directly by teacher probabilities, such sharp soft targets choke off the non-target probability mass that FKL requires to transfer meaningful relational knowledge. In stark contrast, RKL gradients are naturally weighted by student probabilities, allowing optimization to progress via log-odds ratios even under sharp teacher targets. This disparity raises a fundamental question: Is RKL truly intrinsically superior for LLM distillation, or has FKL simply been systematically underestimated due to omitted temperature calibration?
The authors address this gap by dissecting the distinct analytical mechanisms governing distribution softening and gradient scaling across the temperature spectrum. Through rigorous mathematical derivation, they establish that both FKL and RKL asymptotically converge to identical logit matching in the infinite-temperature limit, but diverge sharply at finite operational temperatures: temperature actively alters the target probability distribution that FKL is trained to match, whereas it predominantly conditions the optimization dynamics of RKL via gradient magnitude rescaling. Core idea: Temperature exhibits a fundamental asymmetric effect in LLM distillation—it fundamentally reshapes FKL supervision by unlocking non-dominant token dark knowledge while primarily rescaling RKL gradients; this asymmetry completely overturns the standard empirical ranking, demonstrating that FKL's historical deficiency was merely an artifact of \(\tau=1\) under-softening, and allowing temperature-calibrated FKL and generalized KL objectives to decisively surpass RKL.
Method¶
Overall Architecture¶
The framework systematically investigates the optimization dynamics of temperature-scaled KL distillation across off-policy and on-policy instruction-following regimes. Given input prompt contexts and teacher-generated response sequences, the teacher and student models output vocabulary-wide logit vectors \(z_t, z_s \in \mathbb{R}^V\). Under temperature scaling parameter \(\tau \ge 1\), the softened probability targets are computed as \(p_\tau = \text{softmax}(z_t / \tau)\) and \(q_\tau = \text{softmax}(z_s / \tau)\). By decoupling gradient expressions of \(L_{FKL} = D_{KL}(p_\tau \parallel q_\tau)\) and \(L_{RKL} = D_{KL}(q_\tau \parallel p_\tau)\) with respect to student logits \(z_{s, i}\), the framework formalizes how temperature shapes target semantics versus optimization step conditioning across low-temperature and high-temperature asymptotic regimes, extending these insights to symmetric KL (Sym-KL), Jensen-Shannon (JS), skewed KL (SFKL/SRKL), and adaptive KL (AKL) objectives.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Teacher Logits z_t & Student Logits z_s"] --> B["Target Distribution Softening & Dark Knowledge Redistribution<br/>Scale τ flattens probability mass and exposes non-dominant token signals"]
B --> C["Gradient Rescaling & Asymmetric Optimization Decoupling<br/>Prove asymptotic logit matching and contrast finite-temperature dynamics"]
C --> D["Generalized KL-Family Temperature Calibration<br/>Extend temperature calibration across Sym-KL, JS, SFKL, and AKL objectives"]
D --> E["Calibrated High-Performance Student LLM<br/>FKL and generalized KL decisively outperform RKL on instruction benchmarks"]
Key Designs¶
1. Target Distribution Softening and Dark Knowledge Redistribution: Sharp distributions suppress non-target mass -> Temperature scaling reallocates teacher probability -> Unlocking forward KL supervisory richness
Under the standard setting of \(\tau=1\), autoregressive LLM teacher outputs exhibit sharp probability peaks where top-1 tokens dominate nearly the entirety of the distribution. For FKL minimizing \(D_{KL}(p_\tau \parallel q_\tau)\), the gradient with respect to student logit \(z_{s, i}\) evaluates directly to \(\nabla_{z_{s, i}} L_{FKL} = \frac{1}{\tau} (q_{\tau, i} - p_{\tau, i})\). Because each token's contribution to the update direction is weighted by \(p_{\tau, i}\), collapsing the teacher distribution to near one-hot certainty completely suppresses informative inter-token semantic structures, reducing FKL supervision to little more than hard-label cross-entropy. The paper analyzes the derivative of the teacher target distribution with respect to temperature:
This expression shows that increasing \(\tau\) monotonically increases probability mass for any token whose logit lies below the teacher distribution's expectation \(\mathbb{E}_{j \sim p_\tau}[z_{t, j}]\), while suppressing overconfident dominant tokens. Temperature scaling directly changes "what FKL learns" by explicitly expanding the supervisory support of plausible alternative tokens, transforming FKL from an under-informative objective into an expressive distributor of teacher dark knowledge.
2. Gradient Rescaling and Asymmetric Optimization Decoupling: Masked optimization mechanics -> Dual gradient decomposition and asymptotic equivalence -> Clarifying RKL's muted temperature responsiveness
To theoretically ground why previous studies consistently observed RKL outperforming FKL at \(\tau=1\), the authors derive an explicit representation of the RKL gradient. Let \(\Delta_i = z_{s, i} - z_{t, i}\) denote the student-teacher logit discrepancy at token index \(i\). Proposition 2.2 proves that the RKL gradient can be written as a centered logit discrepancy weighted by the student's own softened distribution:
This analytical form highlights the core operational divergence: RKL does not optimize against a softened probability target directly; rather, it penalizes logit differences \(\Delta\) under student-weighted expectation. Consequently, increasing temperature \(\tau\) primarily scales down the RKL gradient magnitude by \(1/\tau^2\) and modestly smooths student exploration weights \(q_{\tau, i}\), leaving the fundamental supervisory target unshifted. Furthermore, under centered logit assumptions (\(\sum_i z_{t, i} = \sum_i z_{s, i} = 0\)), Proposition 2.1 establishes via first-order Taylor expansion that in the asymptotic limit as \(\tau \to \infty\):
Both FKL and RKL converge identically to teacher-student logit matching. Hence, the empirical distinction between the two objectives is strictly confined to the low-temperature regime, where temperature genuinely enriches FKL's learning targets while offering RKL only cosmetic gradient rescaling. This mathematically elucidates why RKL plateaus or degrades at higher \(\tau\) while FKL undergoes dramatic quality improvements.
3. Generalized KL-Family Temperature Calibration: Fixed single-temperature evaluation bias -> Objective-adaptive temperature grid calibration -> Reshuffling LLM distillation benchmark hierarchies
Because the impact of temperature scaling is deeply objective-dependent, enforcing a uniform default temperature of \(\tau=1\) across all distillation formulations introduces substantial structural evaluation bias. Emerging objectives in recent literature—including Sym-KL, Jensen-Shannon divergence (JS), skewed KL variants (SFKL, SRKL), and adaptive KL (AKL)—have previously been compared strictly at \(\tau=1\), masking their true potential. The authors institute a systematic grid calibration across \(\tau \in \{1.5, 2.0, 2.5, 3.0, 3.5, 4.0\}\) under identical training protocols. The empirical findings reveal that any objective incorporating forward distribution-matching characteristics (such as JS+\(\tau\) and AKL+\(\tau\)) experiences substantial performance leaps when calibrated with appropriate temperature, allowing simple classical objectives to rival or surpass complex, heavily engineered contemporary methods.
Benchmark & Evaluation Protocol¶
- Evaluation Benchmarks: Five standard instruction-following benchmarks: Databricks Dolly Eval, Self-Instruct, Vicuna Eval, Super-Natural Instructions (Super-NI), and Unnatural Instructions (UnNI). All models are evaluated with greedy decoding (evaluation temperature 1.0) and scored via ROUGE-L across 5 random seeds \(\{10, 20, 30, 40, 50\}\). Model checkpoints are selected based on optimal validation ROUGE-L. In addition, an independent LLM-as-a-judge protocol using Claude Haiku 4.5 applies MT-Bench pointwise scoring (1-10 scale) and position-swap debiased pairwise win rates.
- Model Families & Scale: Multi-tier distillation covering GPT-2 (GPT-2 XL 1.5B teacher \(\to\) GPT-2 Base 120M / Medium 350M students), OPT (OPT 6.7B teacher \(\to\) OPT 1.3B student), and modern open-source architectures including Llama-3.2 (3B \(\to\) 1B) and Qwen3 (4B \(\to\) 1.7B) using parameter-efficient LoRA distillation.
- Controlled Training Details: All models are distilled on 14k training pairs from databricks-dolly-15k (with 500 validation and 500 test instances). Training is standardized across all methods with a batch size of 32, 20 epochs, sequence length 512, and learning rates of \(5 \times 10^{-4}\) (GPT-2 Base) and \(5 \times 10^{-5}\) (OPT 1.3B).
Quantitative Experimental Results¶
Main Results¶
Table 1: Off-policy instruction distillation across KL objectives with and without temperature scaling (GPT-2 XL \(\to\) Base and OPT 6.7B \(\to\) 1.3B; ROUGE-L % mean ± std over 5 seeds)
| Model & Objective | Dolly | Super-NI | UnNI | Avg. (3-Dataset) | \(\Delta\text{Avg.}\) |
|---|---|---|---|---|---|
| GPT-2 XL (1.5B) \(\to\) GPT-2 Base (0.1B) | |||||
| Teacher: GPT-2 XL | \(27.00 \pm 0.19\) | \(26.46 \pm 0.41\) | \(31.10 \pm 0.06\) | 28.19 | – |
| FKL (\(\tau=1\)) | \(23.80 \pm 0.55\) | \(16.27 \pm 0.24\) | \(19.03 \pm 0.09\) | 19.70 | – |
| FKL + \(\tau\) | \(25.83 \pm 0.24\) | \(23.78 \pm 0.26\) | \(26.97 \pm 0.12\) | 25.53 | +5.83 |
| RKL (\(\tau=1\)) | \(24.67 \pm 0.13\) | \(21.03 \pm 0.21\) | \(23.94 \pm 0.12\) | 23.21 | – |
| RKL + \(\tau\) | \(25.00 \pm 0.07\) | \(21.41 \pm 0.24\) | \(24.68 \pm 0.34\) | 23.70 | +0.49 |
| Sym-KL (\(\tau=1\)) | \(24.38 \pm 0.14\) | \(19.92 \pm 0.25\) | \(22.21 \pm 0.03\) | 22.17 | – |
| Sym-KL + \(\tau\) | \(25.54 \pm 0.26\) | \(21.69 \pm 0.23\) | \(25.29 \pm 0.21\) | 24.17 | +2.00 |
| JS (\(\tau=1\)) | \(24.08 \pm 0.14\) | \(19.66 \pm 0.10\) | \(22.17 \pm 0.14\) | 21.97 | – |
| JS + \(\tau\) | \(25.51 \pm 0.11\) | \(24.34 \pm 0.40\) | \(27.03 \pm 0.16\) | 25.63 | +3.66 |
| SFKL (\(\tau=1\)) | \(24.44 \pm 0.42\) | \(22.01 \pm 0.23\) | \(23.47 \pm 0.12\) | 23.31 | – |
| SFKL + \(\tau\) | \(25.33 \pm 0.16\) | \(24.40 \pm 0.28\) | \(27.07 \pm 0.27\) | 25.60 | +2.29 |
| SRKL (\(\tau=1\)) | \(24.48 \pm 0.37\) | \(23.25 \pm 0.36\) | \(24.01 \pm 0.05\) | 23.91 | – |
| SRKL + \(\tau\) | \(25.03 \pm 0.42\) | \(23.95 \pm 0.14\) | \(25.13 \pm 0.08\) | 24.70 | +0.79 |
| AKL (\(\tau=1\)) | \(21.83 \pm 0.27\) | \(15.40 \pm 0.16\) | \(18.06 \pm 0.16\) | 18.43 | – |
| AKL + \(\tau\) | \(25.35 \pm 0.19\) | \(23.63 \pm 0.20\) | \(26.18 \pm 0.13\) | 25.05 | +6.62 |
| ABKD (\(\tau=1\)) | \(24.32 \pm 0.29\) | \(23.08 \pm 0.19\) | \(24.32 \pm 0.14\) | 23.91 | – |
| DRKL (\(\tau=1\)) | \(25.51 \pm 0.42\) | \(23.39 \pm 0.11\) | \(26.57 \pm 0.11\) | 25.16 | – |
| OPT 6.7B \(\to\) OPT 1.3B | |||||
| Teacher: OPT 6.7B | \(27.52 \pm 0.29\) | \(30.41 \pm 0.46\) | \(31.39 \pm 0.20\) | 29.77 | – |
| FKL (\(\tau=1\)) | \(26.07 \pm 0.65\) | \(22.11 \pm 0.38\) | \(26.82 \pm 0.10\) | 25.00 | – |
| FKL + \(\tau\) | \(28.20 \pm 0.11\) | \(27.62 \pm 0.16\) | \(31.43 \pm 0.10\) | 29.08 | +4.08 |
| RKL (\(\tau=1\)) | \(26.58 \pm 0.48\) | \(22.64 \pm 0.45\) | \(26.29 \pm 0.17\) | 25.17 | – |
| RKL + \(\tau\) | \(27.64 \pm 0.13\) | \(26.96 \pm 0.48\) | \(31.29 \pm 0.27\) | 28.63 | +3.46 |
| Sym-KL + \(\tau\) | \(27.74 \pm 0.40\) | \(28.21 \pm 0.16\) | \(31.73 \pm 0.15\) | 29.23 | +4.37 |
| JS + \(\tau\) | \(27.97 \pm 0.44\) | \(28.27 \pm 0.19\) | \(32.20 \pm 0.11\) | 29.48 | +3.81 |
| SFKL + \(\tau\) | \(27.52 \pm 0.64\) | \(27.39 \pm 0.20\) | \(32.03 \pm 0.09\) | 28.98 | +3.73 |
| SRKL + \(\tau\) | \(27.83 \pm 0.38\) | \(28.20 \pm 0.37\) | \(31.68 \pm 0.14\) | 29.24 | +4.67 |
| AKL + \(\tau\) | \(28.35 \pm 0.27\) | \(28.10 \pm 0.18\) | \(32.40 \pm 0.20\) | 29.62 | +4.88 |
| ABKD (\(\tau=1\)) | \(26.86 \pm 0.17\) | \(24.39 \pm 0.11\) | \(26.62 \pm 0.09\) | 25.96 | – |
| DRKL (\(\tau=1\)) | \(27.72 \pm 0.36\) | \(25.56 \pm 0.36\) | \(29.84 \pm 0.15\) | 27.71 | – |
Table 2: On-policy distillation dynamics across temperatures (GPT-2 XL \(\to\) Base; ROUGE-L % mean ± std)
| Loss | Temperature \(\tau\) | Dolly | SelfInst | Super-NI | UnNI | Vicuna | 5-Task Avg. |
|---|---|---|---|---|---|---|---|
| FKL | 1.0 | \(22.03 \pm 0.86\) | \(10.29 \pm 0.40\) | \(17.42 \pm 0.18\) | \(20.11 \pm 0.10\) | \(14.89 \pm 0.43\) | 16.95 |
| FKL | 1.5 | \(23.62 \pm 0.49\) | \(10.57 \pm 0.55\) | \(19.01 \pm 0.08\) | \(21.40 \pm 0.09\) | \(14.27 \pm 0.40\) | 17.77 |
| FKL | 2.0 | \(24.16 \pm 0.11\) | \(11.81 \pm 0.62\) | \(20.96 \pm 0.05\) | \(23.64 \pm 0.14\) | \(15.20 \pm 0.43\) | 19.15 |
| FKL | 2.5 | \(26.00 \pm 0.54\) | \(12.62 \pm 0.54\) | \(22.35 \pm 0.34\) | \(25.83 \pm 0.10\) | \(16.06 \pm 0.56\) | 20.57 |
| FKL | 3.0 | \(25.30 \pm 0.30\) | \(12.11 \pm 0.25\) | \(23.70 \pm 0.20\) | \(26.79 \pm 0.26\) | \(16.62 \pm 0.58\) | 20.90 |
| FKL | 3.5 | \(25.32 \pm 0.07\) | \(12.40 \pm 0.22\) | \(24.92 \pm 0.44\) | \(27.20 \pm 0.02\) | \(16.35 \pm 0.29\) | 21.24 (+4.29) |
| FKL | 4.0 | \(25.60 \pm 0.48\) | \(12.36 \pm 0.44\) | \(24.28 \pm 0.26\) | \(26.63 \pm 0.09\) | \(16.42 \pm 0.50\) | 21.06 |
| RKL | 1.0 | \(24.12 \pm 0.25\) | \(12.20 \pm 0.38\) | \(22.43 \pm 0.09\) | \(24.93 \pm 0.06\) | \(16.14 \pm 0.37\) | 19.97 |
| RKL | 1.5 | \(23.96 \pm 0.13\) | \(12.00 \pm 0.62\) | \(23.16 \pm 0.26\) | \(26.78 \pm 0.16\) | \(16.14 \pm 0.37\) | 20.41 |
| RKL | 2.0 | \(24.56 \pm 0.45\) | \(12.33 \pm 0.09\) | \(22.76 \pm 0.18\) | \(25.86 \pm 0.15\) | \(15.64 \pm 0.72\) | 20.23 |
| RKL | 2.5 | \(24.48 \pm 0.13\) | \(10.96 \pm 0.46\) | \(21.88 \pm 0.22\) | \(24.94 \pm 0.01\) | \(15.90 \pm 0.44\) | 19.63 |
| RKL | 3.0 | \(24.36 \pm 0.37\) | \(11.06 \pm 0.13\) | \(19.92 \pm 0.48\) | \(23.63 \pm 0.13\) | \(16.01 \pm 0.12\) | 19.00 |
| RKL | 3.5 | \(24.00 \pm 0.18\) | \(10.66 \pm 0.14\) | \(20.41 \pm 0.38\) | \(23.48 \pm 0.22\) | \(15.78 \pm 0.30\) | 18.87 |
| RKL | 4.0 | \(23.85 \pm 0.31\) | \(11.26 \pm 0.03\) | \(19.88 \pm 0.43\) | \(23.15 \pm 0.15\) | \(16.07 \pm 0.39\) | 18.84 |
Table 3: Generalization to modern open-source LLM architectures (Llama-3.2 and Qwen3; ROUGE-L % mean)
| Model & Objective | Dolly | Self-Inst | Super-NI | UnNI | Vicuna | 5-Task Avg. | \(\Delta\text{Avg.}\) |
|---|---|---|---|---|---|---|---|
| Llama3.2-3B \(\to\) 1B | |||||||
| FKL (\(\tau=1\)) | 25.86 | 15.46 | 28.14 | 29.52 | 15.53 | 22.90 | – |
| FKL + \(\tau\) | 27.34 | 18.73 | 33.04 | 34.07 | 17.11 | 26.06 | +3.16 |
| RKL (\(\tau=1\)) | 28.85 | 18.90 | 34.94 | 35.90 | 17.44 | 27.21 | – |
| RKL + \(\tau\) | 27.61 | 17.44 | 33.57 | 34.37 | 17.10 | 26.02 | -1.19 |
| AKL + \(\tau\) | 28.31 | 18.39 | 34.31 | 34.83 | 17.00 | 26.57 | +1.45 |
| SFKL + \(\tau\) | 28.08 | 18.14 | 34.41 | 34.68 | 17.11 | 26.48 | +1.32 |
| Qwen3-4B \(\to\) 1.7B | |||||||
| FKL (\(\tau=1\)) | 26.56 | 19.17 | 34.83 | 30.56 | 16.69 | 25.57 | – |
| FKL + \(\tau\) | 27.75 | 19.47 | 35.36 | 32.73 | 17.91 | 26.65 | +1.08 |
| RKL (\(\tau=1\)) | 29.39 | 20.66 | 39.39 | 36.48 | 19.12 | 28.99 | – |
| RKL + \(\tau\) | 29.02 | 19.11 | 35.97 | 34.44 | 18.88 | 27.49 | -1.50 |
| AKL + \(\tau\) | 28.04 | 19.37 | 35.39 | 33.42 | 18.09 | 26.86 | +0.19 |
| SFKL + \(\tau\) | 28.20 | 20.22 | 35.77 | 33.63 | 18.45 | 27.25 | +0.02 |
Ablation & Analytical Insights¶
- Temperature Sensitivity Heatmaps: Across the sweep \(\tau \in [1.5, 4.0]\), FKL-based objectives (FKL, Sym-KL, JS, and AKL) exhibit broad, robust positive gains peaking between \(\tau = 2.5\) and \(\tau = 3.5\). On GPT-2 Base, FKL gains sequentially by \(+0.56, +1.72, +3.03, +3.56, +3.47\), and \(+3.09\) ROUGE-L points. Conversely, RKL-based objectives (RKL, SRKL, SFKL) remain flat or degrade rapidly at higher temperatures (RKL drops by \(-0.39\) at \(\tau=3.0\) and \(-0.57\) at \(\tau=4.0\)).
- LLM-as-a-Judge Evaluation (Table 3 & 4 in paper): On OPT 1.3B, MT-Bench pointwise evaluation with Claude Haiku 4.5 gives FKL+\(\tau\) an average score of 30.1 (vs. 25.8 at \(\tau=1\), a gain of +4.3), whereas RKL+\(\tau\) advances by only +0.9 (26.1 vs. 25.2). In pairwise comparisons with position-bias correction, FKL+\(\tau\) achieves a 57.7% win rate against FKL(\(\tau=1\)), whereas RKL+\(\tau\) secures only 42.3% against its uncalibrated counterpart, confirming that temperature tuning actively harms RKL while substantially upgrading FKL.
- Qualitative Adherence: In strict-constraint tasks (Unnatural Instructions), case studies demonstrate that temperature-calibrated models eliminate entity hallucination and format collapse, faithfully producing exact structured extractions where uncalibrated models fail.
Key Findings¶
- Reversal of the FKL-RKL Hierarchy: The common belief that RKL is superior for LLM distillation is an artifact of zero temperature scaling (\(\tau=1\)). When teacher distributions are properly softened (\(\tau \in [2.5, 3.5]\)), FKL consistently overtakes RKL across model scales and tasks.
- Asymmetric Mechanism of Action: Temperature fundamentally alters "what FKL learns" by distributing supervision across semantically valid non-dominant tokens, whereas it only alters "how RKL learns" through gradient magnitude dampening (\(1/\tau^2\)) and student weight flattening.
- Classical Objectives Outperform Complex SOTA: Simple FKL and JS divergence with temperature scaling outperform sophisticated recent distillation objectives like ABKD and DRKL, demonstrating that hyperparameter calibration is paramount before proposing bespoke divergence variants.
Highlights & Insights¶
- Exposing a Decade-Old Blind Spot: Directly challenges the unquestioned convention originating from early sequence-level KD that \(\tau=1\) is optimal, demonstrating that omission of temperature scaling has skewed comparative distillation research for years.
- Asymptotic Logit Matching Equivalence: Proposition 2.1 mathematically proves that FKL and RKL converge to identical logit-matching gradients as \(\tau \to \infty\), proving that divergence differences are purely low-temperature artifacts.
- Drop-in Engineering Impact: Delivering substantial gains (+3 to +6 ROUGE-L points) without introducing auxiliary loss parameters, architectural modifications, or additional training stages—achieved purely by tuning a single scalar in the loss function.
Limitations & Future Work¶
- Teacher Scale Ceiling: Due to academic lab resource limits, experiments evaluated teachers up to 7B parameters (OPT 6.7B, Qwen3-4B); verifying the impact on 70B+ frontier models remains an open avenue.
- Parameter-Efficient Tuning Constraint: Evaluations on modern models (Llama-3.2, Qwen3) relied on LoRA rather than full parameter fine-tuning, leaving open how full-rank capacity interacts with softened target gradients.
- Task Scope: Primary benchmarks focused on English instruction following; extending temperature calibration to mathematical reasoning, multi-turn dialogue, multilingual corpora, and RLHF exploration distributions represents natural future directions.
Related Work & Insights¶
- vs MiniLLM (Gu et al., 2024): MiniLLM advocated for RKL to avoid mode-averaging hallucinations under the assumption that FKL forces students to cover long-tail invalid tokens. This paper demonstrates that MiniLLM's diagnosis was symptomatic of \(\tau=1\) target starvation; properly softened FKL transfers richer semantic structure with lower hallucination rates on structured generation.
- vs DistilLM (Ko et al., 2024) & ABKD (Wang et al., 2025): Rather than designing complex skewed mixtures or multi-parameter alpha-beta divergences to rebalance probability mass, appropriately chosen temperature scaling on standard KL achieves equal or superior performance with zero additional hyperparameters.
Rating¶
- Novelty: 9.0/10 [Overturns a deeply entrenched foundational assumption with rigorous theoretical proofs]
- Soundness & Rigor: 9.5/10 [Extensive multi-seed cross-architecture benchmarks, on/off-policy evaluations, and LLM-as-a-judge corroboration]
- Clarity & Writing: 9.5/10 [Exceptional theoretical and narrative clarity, seamlessly tying gradient proofs to empirical heatmaps]
- Practical Impact: 9.5/10 [Immediate, frictionless drop-in recipe for substantial performance gains across all LLM distillation pipelines]