LLM Fine-tuning with an Unconstrained Fine-tuned Teacher¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zjh-vinky/UFT
Area: LLM Reasoning
Keywords: LLM Fine-tuning, Knowledge Distillation, Value Discovery, Catastrophic Forgetting, General Capability Retention
TL;DR¶
Addressing the trade-off between target-task adaptation and general capability retention during LLM fine-tuning, UFT reconceptualizes the teacher from a static knowledge transferer into a value discovery mechanism, identifying high-value training samples via source-relative uncertainty budgets and guiding confidence-adaptive dual-teacher distillation to achieve superior Pareto performance on code and math reasoning benchmarks.
Background & Motivation¶
Fine-tuning large language models (LLMs) on specialized downstream tasks such as mathematical reasoning and code generation inevitably triggers a fundamental tension between target-task adaptation and the retention of prior general capabilities. To mitigate catastrophic forgetting, conventional strategies primarily deploy parameter-efficient fine-tuning (PEFT) techniques like LoRA or incorporate old-knowledge regularization teachers. However, low-rank updates restrict parameter adaptation and often underfit task-specific subtleties and nuanced reasoning steps. Meanwhile, standard knowledge distillation frameworks treat teachers as static repositories of external knowledge, transferring supervision uniformly across all target-task examples regardless of whether individual samples are redundant, noisy, or destructive to pre-existing capabilities.
Data weighting and sample selection methodologies attempted to circumvent this dilemma by assigning non-uniform importance based on heuristic proxies, such as training loss, confidence, or representation similarity. However, these generic proxies reflect transient optimization dynamics, superficial sample difficulty, or proxy-specific biases rather than task-specialized expert feedback. Uniform learning from all downstream samples forces models to fit noisy reasoning traces and high-variance examples, while conservative loss-based weighting excessively privileges simple samples where the source model already excels, resulting in suboptimal task adaptation. Consequently, fine-tuning struggles in the absence of an expert signal that gauges both task adaptation utility and retention compatibility.
The core limitation stems from a conceptual gap in the spectrum of teacher roles: existing distillation paradigms enforce blind transfer, lacking any mechanism to assess which target-task samples provide reliable adaptation signals without destabilizing confident source behavior. If an unconstrained expert model can first explore task-specific adaptation without retention constraints, its behavioral divergence relative to the source model can serve as an ideal radar for discovering high-value samples. Core idea: shift the teacher's primary role from "knowledge transfer" to "value discovery" by constructing an Unconstrained Fine-tuned Teacher (UFT), utilizing source-model uncertainty budgets to penalize behavioral divergence and discover high-value samples, followed by confidence-adaptive dual-teacher distillation.
Method¶
Overall Architecture¶
The UFT framework comprises three tightly integrated stages: dual-teacher construction, the value discovery engine, and value-guided dual-teacher distillation. Given a pre-trained source model \(M_{\text{ref}}\) with parameters \(\theta_{\text{ref}}\) and a target-task dataset \(\mathcal{D}_{\text{tgt}} = \{(x_i, y_i)\}_{i=1}^N\), Stage 1 establishes the Source Reference Teacher (SRefT) from the frozen source checkpoint and fine-tunes an unconstrained counterpart \(M_{\text{uft}}\) on \(\mathcal{D}_{\text{tgt}}\) without preservation constraints. In Stage 2, the value discovery engine evaluates UFT reliability alongside source-relative divergence bounded by the source uncertainty budget, yielding continuous sample utilities that map to optimal training weights via entropy regularization. In Stage 3, the student model is fine-tuned under utility-weighted task loss coordinated with dual-teacher distillation, where SRefT stabilizes pre-existing abilities and UFT concentrates adaptation guidance on high-value samples.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Source Model M_ref & Target Dataset D_tgt"] --> B["Constructing Unconstrained Fine-tuned Teacher (UFT)<br/>Expose task-induced drift without preservation constraints"]
B --> C["Sample Utility Assessment<br/>UFT reliability gate & source uncertainty relative divergence"]
C --> D["Entropy-Regularized Weight Optimization<br/>Closed-form Softmax weighting avoiding collapse"]
D --> E["Confidence-Adaptive Dual-Teacher Distillation<br/>Balance SRefT preservation anchor & UFT adaptation guidance"]
E --> F["Output: Pareto-Optimal Student Model M_theta"]
Key Designs¶
1. Constructing the Unconstrained Fine-tuned Teacher (UFT): Exposing Task-Induced Behavioral Drift Existing distillation teachers are restricted by preservation objectives or generalist multi-task scopes, preventing them from exposing extreme task-specific adaptations. UFT breaks free from retention constraints by directly optimizing standard task loss over the target split: \(\theta_{\text{uft}} = \arg\min_{\theta} \sum_{(x,y) \in \mathcal{D}_{\text{tgt}}} \ell(f_{\theta}(x), y)\). By monitoring target-task validation negative log-likelihood (NLL), early stopping selects the checkpoint that maximizes task-specific generalization while preventing late-stage overfitting. The resulting \(M_{\text{uft}}\) serves as a specialized value-discovery probe, exposing behavioral discrepancies relative to the source model to serve as grounded evidence for sample utility estimation.
2. Sample Utility Assessment: Penalizing Divergence under Source Uncertainty Budgets A large divergence between UFT and SRefT does not automatically indicate an undesirable sample: it could signify either a critical task-specific breakthrough or a severe disruption to preserved knowledge. To resolve this ambiguity, UFT benchmarks divergence against the source model's own uncertainty. Defining sequence-level uncertainty as \(s_i^c = -\frac{1}{T_i} \sum_{t=1}^{T_i} \log P_c(y_{i,t} \mid x_i, y_{i,<t})\) for \(c \in \{\text{ref}, \text{uft}\}\) and token-averaged forward KL divergence as \(d_i = \frac{1}{T_i} \sum_{t=1}^{T_i} D_{\text{KL}}(P_{\text{ref}}(\cdot \mid x_i, y_{i,<t}) \parallel P_{\text{uft}}(\cdot \mid x_i, y_{i,<t}))\), sample utility \(u_i\) is formulated as:
This metric segments samples into six characteristic regimes (aligned reinforcement, low-conflict exploration, high conflict risk, novel mismatch, and reliability-attenuated zones). The Sigmoid term acts as a soft reliability gate, while the exponential penalty activates strictly when teacher divergence \(d_i\) exceeds the source uncertainty threshold \(s_i^{\text{ref}}\), granting ample exploration latitude in regions where the source model is genuinely uncertain.
3. Entropy-Regularized Weight Optimization: Smooth Closed-Form Sample Weight Allocation To avoid extreme weight collapse onto a few outliers and prevent zero-weight truncation of informative examples, the authors formulate an entropy-regularized expected utility maximization problem over the probability simplex \(\Delta_{N-1}\): \(\max_{w \in \Delta_{N-1}} \sum_{i=1}^N w_i u_i - \lambda \sum_{i=1}^N w_i \log w_i\), with \(\lambda > 0\) acting as the temperature regularizer. Setting the first-order stationarity conditions of the Lagrangian yields a globally unique closed-form solution:
This temperature-scaled Softmax transformation smoothly concentrates optimization focus onto high-utility instances while guaranteeing that all samples retain continuous participation throughout training.
4. Confidence-Adaptive Dual-Teacher Distillation: Harmonizing Retention and Adaptation Equipped with sample weights \(w_i^*\), the student model requires complementary supervision rather than isolated label fitting. SRefT provides stability regularization \(\mathcal{L}_{\text{ref}}^{(i)}\), while UFT provides task-specific expert distillation \(\mathcal{L}_{\text{uft}}^{(i)}\). To dynamically arbitrate teacher dominance per sample, reliability scores \(r_i^c = \exp(-s_i^c)\) are coupled into normalized trust coefficients:
When the source model is confident (\(s_i^{\text{ref}}\) is low), \(m_i \approx 1\), compelling the student to adhere to SRefT. Conversely, when UFT exhibits superior confidence, \(n_i \approx 1\), steering the student toward target task expertise. The cumulative training objective combines weighted task loss and dual distillation:
The uniformly normalized SRefT loss anchors global pre-existing knowledge across all tokens, while the \(w_i^* n_i\)-scaled UFT distillation injects curvature contraction precisely into preservation-compatible high-value samples.
Loss & Training¶
Student optimization is conducted using the AdamW optimizer with a cosine learning rate decay schedule. Across all backbone models, hyperparameter configurations remain consistent: \(\tau = Q_{60}(s^{\text{uft}})\) (the 60th percentile of UFT uncertainty scores), reliability softness \(\gamma = 0.2\), relative-divergence coefficient \(\alpha = 0.8\), entropy temperature \(\lambda = 0.2\), and distillation factor \(\beta = 2\). All experiments are executed on 4 NVIDIA A100 GPUs. For Magicoder (111K samples), batch size is 64 with a sequence length of 4096 over 3 epochs; for MetaMathQA (395K samples), batch size is 128 with a sequence length of 1024 over 2 epochs.
Key Experimental Results¶
Main Results¶
UFT was benchmarked against leading adaptation methods across LLaMA3.2-3B, Gemma3-4B, and Qwen3.5-9B on Magicoder (code generation) and MetaMathQA (mathematical reasoning). For Magicoder, target performance is assessed via HumanEval (HE) PASS@1, while general capability is measured by the average across HellaSwag (HASW), OpenBookQA (OBQA), MMLU, and GSM8K. For MetaMathQA, GSM8K serves as the target task, and HE joins the general capability suite.
The table below presents the complete quantitative comparison on the Magicoder benchmark (% accuracy, mean ± std across 3 seeds):
| Model | Method | General HASW | General OBQA | General MMLU | General GSM8K | General Avg. ↑ | Target HE ↑ | Overall Avg. ↑ |
|---|---|---|---|---|---|---|---|---|
| LLaMA3.2-3B | Pre-trained | 55.83±0.00 | 31.23±0.00 | 56.92±0.00 | 27.89±0.00 | 42.97±0.00 | 28.16±0.00 | 35.57±0.00 |
| Standard FT | 52.12±0.47 | 28.39±0.37 | 48.61±0.43 | 20.74±0.34 | 37.47±0.29 | 45.13±0.48 | 41.30±0.30 | |
| LoRA (ICLR 22) | 53.38±0.39 | 29.46±0.34 | 51.98±0.35 | 21.17±0.39 | 39.00±0.26 | 41.92±0.40 | 40.46±0.25 | |
| FLOW (ICML 25) | 54.63±0.34 | 30.06±0.30 | 55.83±0.31 | 25.12±0.23 | 41.41±0.22 | 42.34±0.34 | 41.88±0.21 | |
| DFT (ICLR 26) | 53.36±0.37 | 29.71±0.21 | 51.97±0.33 | 25.34±0.24 | 40.10±0.21 | 45.01±0.36 | 42.56±0.23 | |
| TALR (ICLR 26) | 54.84±0.35 | 30.20±0.34 | 54.01±0.30 | 26.79±0.34 | 41.46±0.24 | 43.54±0.35 | 42.50±0.22 | |
| ADAPT (ICLR 26) | 53.76±0.41 | 29.62±0.29 | 52.87±0.36 | 22.24±0.29 | 39.62±0.23 | 45.27±0.39 | 42.45±0.24 | |
| UFT (Ours) | 55.29±0.31 | 30.96±0.26 | 56.22±0.28 | 28.12±0.31 | 42.65±0.22 | 46.73±0.40 | 44.69±0.23 | |
| Gemma3-4B | Standard FT | 53.06±0.45 | 31.24±0.38 | 53.21±0.40 | 23.79±0.28 | 40.33±0.27 | 58.17±0.49 | 49.25±0.29 |
| ADAPT (ICLR 26) | 55.66±0.39 | 31.43±0.35 | 56.77±0.33 | 32.02±0.39 | 43.97±0.26 | 58.39±0.52 | 51.18±0.30 | |
| UFT (Ours) | 56.93±0.30 | 32.56±0.29 | 59.28±0.28 | 36.27±0.46 | 46.26±0.24 | 59.53±0.34 | 52.90±0.22 | |
| Qwen3.5-9B | Standard FT | 54.21±0.41 | 31.06±0.33 | 66.22±0.36 | 78.57±0.31 | 57.52±0.25 | 75.82±0.41 | 66.67±0.25 |
| ADAPT (ICLR 26) | 56.48±0.36 | 31.78±0.34 | 68.52±0.30 | 81.94±0.43 | 59.68±0.26 | 76.23±0.41 | 67.96±0.25 | |
| UFT (Ours) | 58.60±0.27 | 32.69±0.31 | 70.91±0.25 | 84.96±0.39 | 61.79±0.22 | 77.42±0.34 | 69.61±0.21 |
Ablation Study¶
To dissect the functional contribution of each constituent within UFT, the authors evaluated component removals and alternative weighting signals on LLaMA3.2-3B / Magicoder:
| Configuration / Variant | General Avg. ↑ | Target HE ↑ | Overall Avg. ↑ | Mechanism & Remarks |
|---|---|---|---|---|
| Full UFT | 42.65 | 46.73 | 44.69 | Complete dual-teacher value discovery pipeline |
| - w/o Utility weighting | 40.53 | 44.28 | 42.41 | Reverts to Uniform-UFT; target and overall metrics drop heavily |
| - w/o SRefT distillation | 40.19 | 46.84 | 43.52 | Loses source anchor; general capability suffers a steep 2.46% drop |
| - w/o UFT distillation | 42.86 | 44.96 | 43.91 | Weakens task gradient curvature; degrades target accuracy |
| - w/o Confidence adaptive | 41.55 | 45.52 | 43.54 | Fixed teacher balance induces gradient interference |
| - w/o Relative divergence | 40.37 | 45.34 | 42.86 | Discards source uncertainty constraints; conflicts damage retention |
| Signal Ablation: Source-loss ↓ | 42.90 | 41.72 | 42.31 | Overly conservative; underfits complex task reasoning |
| Signal Ablation: UFT-confidence ↑ | 39.96 | 45.18 | 42.57 | Aggressive task fitting; severely compromises general retention |
| Signal Ablation: Shuffled-UFT | 39.78 | 43.30 | 41.54 | Permuting sample weights degrades performance, proving utility order is vital |
Key Findings¶
- Value Discovery is the Primary Driver of Adaptation-Retention Balance: Removing utility weighting triggers a 2.28% plunge in overall score, while removing relative divergence reduces general retention by 2.28%. This validates that filtering target evidence through source uncertainty bounds is crucial for discovering high-utility examples.
- Empirical Stability Validates Proposition 1: Theoretical derivation and empirical tracking (Table 10) confirm that static UFT utility rankings remain exceptionally robust across student training epochs (Spearman correlation remains 0.926 at Epoch 3, and Top 20% sample overlap exceeds 80.8%), with total variation distance staying safely below the theoretical upper bound.
- Compute-Matched Parity Demonstrates Intrinsic Algorithmic Superiority: Even when baselines are granted twice the training compute (6 epochs vs. UFT's 3 epochs to offset teacher construction cost), Standard FT (41.58%) and ADAPT (42.80%) remain substantially behind UFT (44.69%), confirming that UFT's performance gains derive from superior learning dynamics rather than extra FLOPs.
Highlights & Insights¶
- From Blind Knowledge Transfer to Active Value Discovery: Traditional distillation treats teachers as static oracle knowledge broadcasters. UFT innovatively utilizes the task-induced drift of an unconstrained fine-tuned teacher as an active radar to discover which training samples offer maximum learning value at minimal retention cost.
- Uncertainty-Budgeted Divergence Regulation: Instead of penalizing all divergence between teachers, UFT benchmarks divergence against the source model's own predictive uncertainty, establishing a principled mathematical framework where confident source regions are strictly preserved while uncertain regions are permitted wider adaptation leeway.
- Highly Practical LoRA-Based Teacher Approximation: When teacher pre-training budget is limited, constructing a LoRA-based UFT requires only 0.16× memory and 0.75× time while retaining a 0.86 Pearson utility correlation and delivering 44.18% overall performance (capturing over 98.8% of the full-parameter teacher's gains).
Limitations & Future Work¶
- Reliance on Closed-Ended Teacher-Forced Likelihoods: UFT computes uncertainties and KL divergences under teacher-forced prompt-response pairs, which may face degraded fidelity in open-ended creative dialogue or multi-answer reasoning where token likelihood does not strictly mirror semantic correctness.
- Static Pre-Computed Utilities Across Multi-Stage Tuning: While Proposition 1 proves utility stability during local single-task training, prolonged continual learning or multi-task shifts may eventually cause static utilities to drift, suggesting a need for dynamic utility refresh algorithms.
- Future Work: Extending UFT to multimodal perception, long-context document reasoning, and multilingual continual adaptation, alongside developing lightweight navigator architectures to further reduce teacher construction overhead.
Related Work & Insights¶
- vs FLOW (Sanyal et al., 2025): FLOW relies exclusively on pre-trained source model losses to upweight easy examples, representing a conservative defense that underfits complex target tasks; UFT introduces a task-specialized teacher to discover proactive adaptation value.
- vs ADAPT (Zhao et al., 2026): ADAPT applies dynamic online reweighting derived from sample embedding similarity, which can be noisy and computationally unstable; UFT delivers closed-form optimal weights and anchors optimization through dual-teacher confidence distillation.
- vs DavIR (Zhou et al., 2025): DavIR relies on loss-reduction learnability between base and fine-tuned models without evaluating compatibility with source capabilities; UFT incorporates source uncertainty bounds to outperform DavIR reweighting by 1.15~1.72 overall points across code and math benchmarks.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Reconceptualizes knowledge distillation by establishing the unconstrained teacher as a value discovery mechanism.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 3 foundation architectures, 2 complex domains, comprehensive ablations, Holm-adjusted significance tests (\(p < 0.05\)), and compute-matched controls.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical derivations (including convergence and local contraction proofs) and coherent prose structure.
- Value: ⭐⭐⭐⭐⭐ Offers a principled and practically viable methodology for mitigating catastrophic forgetting during domain-specific LLM fine-tuning.