Supervised Fine-Tuning for Reasoning Large Language Models: A Survey¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yuyongcan/Awesome-SFT-for-LRMs
Area: LLM Reasoning
Keywords: Large reasoning models, Supervised fine-tuning, Chain-of-thought, Optimization dynamics, RL cold-start
TL;DR¶
This paper provides the first systematic survey dedicated to Supervised Fine-Tuning (SFT) for reasoning large language models, establishing a dual-axis taxonomy of data-centric curation and multi-granularity algorithmic gradient modulation, unifying diverse losses under a generalized optimization objective, and organizing empirical findings into a data-dynamics-mechanisms analytical framework with explicit causal links to methodological design.
Background & Motivation¶
In recent years, the paradigm of Large Reasoning Models (LRMs) has achieved dramatic breakthroughs across demanding frontiers, including competitive mathematics Olympiads, complex program synthesis, and multi-step symbolic deduction. The primary catalyst behind this leap lies in supervised fine-tuning (SFT) on extensive long Chain-of-Thought (CoT) trajectories, which enables neural models to move beyond fragile inference-time prompting heuristics and directly internalize step-by-step deductive mechanisms into their model parameters. However, early reasoning SFT practices predominantly treated fine-tuning as vanilla behavior cloning, largely dependent on offline knowledge distillation from proprietary frontier systems. As model scale and problem complexity increased, standard SFT quickly exposed fundamental vulnerabilities—including exposure bias caused by distribution mismatch, severe overfitting to superficial lexical templates, catastrophic forgetting of general domain capabilities, and severe probability mass collapse that extinguishes the policy entropy required for subsequent reinforcement learning (RL) exploration.
The core tension in reasoning SFT stems from the fundamental divergence between static behavioral imitation and autonomous logical exploration. Unlike conventional instruction-following tasks where outputs directly present concise target responses, reasoning SFT decomposes the sequence into an extensive rationale prefix followed by the final answer (\(y = [r; a]\)). Within these protracted reasoning trajectories, individual tokens exhibit extreme heterogeneity in terms of structural importance, predictive uncertainty, and fault tolerance. Minimizing vanilla negative log-likelihood with equal token weighting forces the optimizer to expend disproportionate gradient budget on memorizing superficial transition tokens and rhetorical markers, while failing to sharpen decision-critical branching points or teach verification and backtracking. Compounding this challenge, the broader research landscape remains heavily fragmented: contemporary post-training surveys overwhelmingly prioritize RL algorithms (e.g., DeepSeek-R1), relegating SFT to a secondary initialization step, while earlier SFT literature remains restricted to empirical data construction recipes without a formalized optimization taxonomy or a rigorous resolution of the "memorization versus genuine generalization" debate.
To consolidate this fast-evolving landscape, this survey examines reasoning SFT through the dual lenses of methodological design and analytical investigation. On the methodological front, the paper introduces a clean taxonomy categorizing recent innovations into data-centric curation (data augmentation and native distribution refinement) and algorithm-centric optimization (progressing systematically along the granularity axis from token-level and sample-level modulation to distribution-level regularization and structural integration with RL). On the analytical front, it reorganizes highly dispersed empirical evidence into three core dimensions: data characteristics, training dynamics, and mechanistic interpretations, synthesizing clear causal bridges that trace how observed empirical failures motivate specific algorithmic solutions. Core idea: Establish a dual-axis taxonomy of data-centric curation and multi-granularity algorithmic modulation unified under a generalized optimization formulation, paired with a three-dimensional analytical framework and explicit causal feedback loops, to serve as a comprehensive foundational roadmap for post-training advanced reasoning models.
Method¶
Overall Architecture¶
The architectural paradigm synthesized by this survey encompasses an end-to-end evolutionary pipeline that connects raw data sourcing, fine-grained gradient modulation, synergistic RL integration, and empirical mechanistic diagnosis. In standard instruction tuning, models are optimized to predict target sequence \(y\) given instruction \(x\). In reasoning SFT, the target sequence is structurally partitioned into a multi-step rationale \(r\) and a final answer \(a\), i.e., \(y = [r; a]\). To mitigate the empirical pitfalls of naive behavior cloning, the overall pipeline first applies data-centric curation (leveraging negative sample contrast and self-alignment rewriting) to alleviate distributional mismatch; it then deploys multi-granularity algorithmic modulation to dynamically govern gradient propagation; subsequently, it implements policy-preserving transitions into reinforcement learning; and finally, it utilizes a three-dimensional analytical framework to uncover underlying training dynamics and guide future methodological refinements.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Reasoning SFT Inputs<br/>Long CoT Trajectories & Prompts"] --> B["Data-Centric Curation<br/>Negative Contrast & Native Rewriting"]
B --> C["Multi-Granularity Algorithmic Modulation<br/>Token/Sample Reweighting & KL Anchoring"]
C --> D["SFT & RL Dynamic Integration<br/>Entropy Preservation & Decoupled Transition"]
D --> E["Causal Analytical Framework<br/>Dynamics Diagnosis & Mechanistic Feedback"]
E --> F["Generalizable & RL-Compatible Reasoning LLMs"]
Key Designs¶
1. Data-Centric Curation: Negative Contrast and In-Distribution Rewriting to Eliminate Mismatch
Targeting the severe distribution shift and brittle single-path dependency caused by directly cloning off-policy teacher demonstrations, data-centric curation restructures training signals during corpus generation and filtering. Traditional pipelines rely exclusively on positive demonstration imitation, leaving models helpless when deviating from the training path during inference. In response, modern data augmentation frameworks extract dense contrastive signals from negative trajectories. NFT jointly models correct and failed reasoning traces through an implicit negative policy, actively repelling the model from common hallucinations and deductive flaws, while RIFT calibrates constant positive and negative rewards to convert erroneous steps into explicit repulsive forces. In parallel, data refinement tackles the incompatibility between external expert distributions and student prior capabilities. Self-alignment paradigms (such as Gupta et al. and Zhao et al.) prompt student models to reconstruct valid reasoning paths conditioned on known reference answers, ensuring training trajectories align natively with the student's intrinsic probability distribution. Similarly, SPOT uses minimal-edit teacher interventions to surgically rectify erroneous reasoning steps while leaving valid context untouched, fundamentally resolving the performance degradation caused by off-policy distribution mismatch.
2. Multi-Granularity Algorithmic Modulation: Unified Objective for Adaptive Token and Sample Weighting
To overcome the gradient distortion and critical-token dilution inherent in uniform cross-entropy loss, the survey unifies modern algorithmic advances under a generalized optimization formulation: $\(\mathcal{L}_{\text{gSFT}}(\theta) = - \frac{1}{|\mathcal{D}|} \sum_{i=1}^{|\mathcal{D}|} \alpha_i \sum_{t=1}^{|y^{(i)}|} w_t^{(i)} \log \pi_\theta(y_t^{(i)} \mid y_{<t}^{(i)}, x^{(i)})\)$ where dynamic weights \(w_t^{(i)}\) and \(\alpha_i\) operate across the granularity hierarchy. At the token level, distinguishing genuinely difficult reasoning steps from noisy tokens is a critical challenge. DEFT introduces an entropy-guided trust gate that transitions the optimization focus from exploring uncertain regions to sharpening confident predictions. TALR dynamically normalizes token updates using batch median losses \(w_i \propto \exp(-\ell_i/\tau)\), while discrete token selection methods like SFTKey mask intermediate rationale tokens entirely to constrain optimization strictly to final answer tokens, preventing verbose superficial reasoning patterns from overwhelming the gradient. At the sample level, scalar weight \(\alpha_i\) calibrates instance influence: FLOW upweights easy samples with low pre-training losses to combat catastrophic forgetting, whereas BFT prioritizes hard sequences containing continuous low-probability spans to force new skill acquisition. Crucially, iw-SFT and BOLT derive importance weights from policy trajectory ratios or exponential rewards, establishing a formal mathematical equivalence between offline SFT and online RL policy mirror descent. At the distribution level, ASFT and PSFT enforce KL divergence bounds or gradient clipping to preserve pre-trained knowledge bases while expanding reasoning boundaries.
3. SFT & RL Dynamic Integration: Entropy Preservation and Decoupled Structural Transitions
Addressing the fundamental trade-off between the bounded capability ceiling of pure SFT and the extreme cold-start exploration instability of isolated RL, the survey systematizes paradigms that merge imitation and reinforcement. Standard SFT heavily concentrates probability mass on reference trajectories, inducing distributional collapse that strips the policy of the generative entropy necessary for online RL exploration. To solve this, lightweight SFT (Li et al.) and Gibbs initialization (Zhao et al.) scale gradients to preserve the relative probability mass of alternative tokens, shifting the training goal from maximizing standalone SFT benchmark accuracy to optimizing initial policy search potential. In joint optimization architectures, AMFT deploys meta-gradient controllers to dynamically balance imitation and exploration weights; SRFT integrates entropy-regularized SFT with off-policy RL; CHORD enforces a deterministic temporal transition schedule from supervised imitation to pure RL; and BRIDGE isolates SFT updates within lightweight parameter adapters (LoRA) to prevent interference with base policy weights. For intractable problems, adaptive routing frameworks (e.g., ReLIFT, DYPO) route failed exploratory rollouts to multi-teacher SFT distillation while channeling successful rollouts into online RL, forming an efficient curriculum closed loop.
4. Causal Analytical Framework: Resolving the "Memorization vs. Generalization" Dichotomy
To resolve longstanding academic controversies regarding whether SFT merely induces superficial format imitation or cultivates genuine generalizable reasoning, the survey synthesizes empirical findings across data properties, training dynamics, and mechanistic interpretability into a cohesive causal diagnostic framework. Regarding CoT length, it debunks the simplistic assumption that "longer is always better," proving that reasoning accuracy follows an inverted-U curve where excessive length introduces redundant noise and compounding errors. Regarding dataset scale, multiple empirical studies confirm that fewer than 1K—or even dozens of—carefully curated, high-diversity reasoning instances are sufficient to elicit robust multi-step capabilities, proving that structural quality far outweighs sample volume. In optimization dynamics, it highlights the non-monotonic recovery phenomenon where out-of-distribution (OOD) accuracy initially drops before recovering during multi-epoch training, explaining why standalone SFT requires thorough optimization while RL initialization demands early stopping. Furthermore, the survey formalizes four explicit causal bridges in Appendix F: distributional alignment drives native data refinement; diversity requirements motivate hybrid trajectory augmentation; overfitting risks necessitate selective token masking; and SFT-RL complementarity (SFT expands the solution space via Pass@k, while RL refines trajectory selection) justifies the modern two-stage post-training pipeline.
Loss & Training¶
In Appendix G, the survey establishes a standardized mathematical formulation across five evolutionary classes of reasoning SFT objectives: 1. Variance and Loss-Governed Reweighting: Scaling gradients by token loss and gradient consistency (e.g., VCORE, TALR); 2. Entropy and Confidence-Guided Modulation: Dynamically filtering high-entropy noise and emphasizing critical low-probability, low-entropy decision points (e.g., DEFT, RANKTUNER, ProFit); 3. Distribution-Constrained Regularization: Bounding optimization within a trust region via KL penalties against reference models or Gibbs smoothing (e.g., ASFT, PSFT, GIFT); 4. Offline Policy Gradient & Importance Sampling: Re-weighting offline demonstration traces to simulate online reinforcement learning dynamics (e.g., iw-SFT, BOLT, PEAR); 5. Unified SFT-RL Joint Objectives: Harmonizing supervised imitation and policy gradient objectives via dynamic, uncertainty-aware scheduling: \(\mathcal{L}_{\text{Joint}}(\theta) = \lambda(t) \mathcal{L}_{\text{SFT}}(\theta) + (1-\lambda(t)) \mathcal{L}_{\text{RL}}(\theta)\) (e.g., SRFT, AMFT, CHORD).
Key Experimental Results¶
Main Results: Rigorous Meta-Analysis on a Shared Baseline (Qwen2.5-Math-7B + NuminaMath-CoT)¶
To overcome the pervasive issue of incomparable cross-paper evaluations caused by divergent base models, datasets, and evaluation scripts, the authors synthesize raw empirical results from literature that shares the exact same experimental configuration: a Qwen2.5-Math-7B base model fine-tuned on NuminaMath-CoT, evaluated across four standardized mathematical reasoning benchmarks (Accuracy %):
| Method | Category / Core Optimization Mechanism | GSM8K | MATH | Minerva Math | OlympiadBench | AIME 2024 | AMC 2023 | Macro Avg. (4 Tasks) |
|---|---|---|---|---|---|---|---|---|
| Base Model | Zero-shot inference without fine-tuning | 54.8 | 50.3 | 14.4 | 16.4 | 12.1 | 36.9 | 20.0 |
| EAFT | Entropy-adaptive token reweighting | – | – | 20.2 | 19.0 | 2.5 | 24.2 | 16.5 |
| ASFT | Anchored SFT with KL divergence penalty | – | 65.8 | 24.1 | 29.3 | 9.0 | 43.3 | 26.4 |
| Vanilla SFT | Standard equal-weighted cross-entropy loss | 71.1 | 60.9 | 21.8 | 32.9 | 9.2 | 43.4 | 26.8 |
| ProFiT | High-probability token selection | – | 68.5 | 28.7 | 31.4 | 7.1 | 44.1 | 27.8 |
| DFT | Dynamic loss-guided token filtering | 87.0 | 70.6 | 30.2 | 33.8 | 8.6 | 45.0 | 29.4 |
| RASFT | Relative advantage sampling fine-tuning | – | 70.7 | 30.0 | 33.0 | 8.8 | 45.8 | 29.4 |
| TALR | Normalized loss-driven token reweighting | – | – | 35.7 | 32.9 | 6.7 | 43.4 | 29.7 |
| RFT* | Self-generated trajectory augmentation | 83.3 | 69.8 | 21.3 | 31.3 | 11.2 | 58.8 | 30.7 |
| iw-SFT | Importance-weighted offline RL simulation | – | 70.3 | 25.7 | 34.5 | 16.5 | 51.1 | 32.0 |
| RANKTUNER | Joint probability-entropy rank modulation | – | – | 38.3 | 36.3 | 10.2 | 46.6 | 32.9 |
| RIFT* | Reward-informed fine-tuning on self-rollouts | 86.3 | 74.7 | 28.9 | 34.1 | 17.1 | 62.2 | 35.6 |
*Note: RFT* and RIFT* are trained on model self-generated trajectories rather than static NuminaMath-CoT annotations, representing the empirical upper bound for in-distribution data and negative sample utilization.
Comparative Survey Taxonomy¶
To explicitly contextualize the scope and depth of this paper within the broader landscape of post-training and reasoning surveys, the authors provide a multi-dimensional comparison against 12 related survey papers:
| Survey | Primary Focus | SFT Depth | Algorithm-Centric Taxonomy | Analytical Synthesis | Time Coverage | Key Distinction from This Survey |
|---|---|---|---|---|---|---|
| Thinking Hats (Chen et al., 2025e) | CoT fine-tuning & cognition | Partial (mixed with RL) | No (cognitive view) | Limited (descriptive) | Through 2024 | Focuses on human cognitive psychology rather than mathematical optimization dynamics |
| 100 Days After R1 (Zhang et al., 2025b) | R1 replication studies | Partial (distillation focus) | No (engineering focus) | Minimal (field report) | Jan–Apr 2025 | Empirical field report on R1 replication; lacks a unified algorithmic taxonomy |
| Towards Reasoning Era (Chen et al., 2025d) | Long CoT reasoning LLMs | Minimal (RL-centric) | No (RL focus) | Limited (phenomena list) | 2024–early 2025 | Centered on reinforcement learning, treating SFT primarily as an auxiliary cold-start stage |
| SFT vs RL (Jiang et al., 2026a) | General post-training comparison | Partial (high-level) | Light (broad taxonomy) | No (pipeline overview) | 2022–2025 | Broad post-training overview across generic tasks; lacks reasoning-specific granularity |
| LLM Post-Training (Kumar et al., 2026) | Full post-training pipeline | Partial (pipeline stage) | Minimal (pipeline view) | No (no deep dive) | 2024–2025 | Covers the entire spectrum from pre-training to serving, lacking depth on reasoning SFT |
| Toward Large Reasoning Models (Xu et al., 2025a) | RL for reasoning models | Minimal (SFT as prerequisite) | Yes (for RL algorithms) | Limited (RL-centric) | 2024–Feb 2025 | Complementary RL survey; covers RL algorithms in depth while this work covers SFT |
| This Survey (Yu et al., EMNLP 2026) | SFT for Reasoning LLMs | Exclusive & Core Focus | Core Contribution (Dual-Axis 4-Level) | Systematic & Deep (3-Dimensional) | Jul 2024–May 2026 | The first dedicated, exhaustive survey on reasoning SFT methodologies, optimization gradients, and mechanisms |
Key Findings¶
- Algorithmic modifications do not guarantee empirical gains: On the standardized benchmark, complex objectives like EAFT (16.5) and ASFT (26.4) underperform vanilla SFT (26.8). This demonstrates that reweighting and regularization strategies succeed only when their inductive bias matches the underlying problem distribution; ill-suited regularizers actively damage learning.
- Simpler, early baselines remain remarkably robust: Elegant foundational methods such as DFT (29.4) and iw-SFT (32.0) significantly outperform several later, more convoluted objectives. Chronological novelty does not equate to performance superiority; direct, unconfounded gradient modulation often beats overly intricate meta-schedules.
- Self-generated in-distribution trajectories dominate external annotations: Methods trained on self-generated responses (RFT* at 30.7 and RIFT* at 35.6) achieve the highest macro-averages by substantial margins (+3.9% and +8.8% over vanilla SFT). This confirms that in-distribution alignment with the student's native policy space is far more critical for internalization than surface correctness from external teachers.
- The decoupling of standalone SFT scores and downstream RL readiness: Mechanistic studies confirm that checkpoints exhibiting the highest standalone SFT validation accuracy (due to aggressive probability sharpening) frequently cause rapid policy collapse when fine-tuned via online RL. Conversely, early-stage SFT checkpoints that preserve exploratory entropy serve as significantly superior starting points for subsequent RL optimization.
Highlights & Insights¶
- Unified 2D Map of Algorithmic Innovations: The survey introduces a pioneering 2D classification mapping optimization granularity (Token-level, Sample-level, Distribution-level, RL integration) against underlying heuristic drivers (Confidence/Entropy, Sample difficulty, Semantic/Structural rules, Reference policy divergence), organizing dozens of disparate 2025–2026 techniques into a cohesive matrix.
- Explicit Causal Bridges Between Analysis and Design: Departing from conventional descriptive surveys, the authors formulate four explicit causal feedback loops (detailed in Appendix F) that illustrate how empirical observations (e.g., distribution mismatch, diversity collapse, catastrophic forgetting, SFT-RL complementarity) directly motivated specific architectural solutions (e.g., native data rewriting, hybrid augmentation, selective token masking, entropy-regularized joint objectives).
- Nuanced Deconstruction of the "Memorization Myth": By analyzing Pass@k dynamics (where SFT substantially expands the coverage of solvable problems) and skill compositional transfer, the paper refutes the sweeping claim that "SFT only memorizes while RL generalizes," demonstrating that generalization failures in SFT are primarily symptomatic of poor data diversity or suboptimal hyperparameter tuning rather than intrinsic paradigm limits.
Limitations & Future Work¶
- Author-Acknowledged Scope Boundaries: Given the explosive volume of post-2024 literature, the survey explicitly concentrates on recent developments (mid-2024 through 2026), omitting an exhaustive historical review of early heuristic data construction; additionally, uncurated meta-analysis across disparate base models remains scientifically intractable.
- The Knowledge-Reasoning Trade-off ("Knowledge Tax"): Highly confident reasoning SFT objectives inadvertently sharpen the model's existing beliefs, suppressing gradient signals for unfamiliar facts and underrepresented domains. Developing capability-aware SFT frameworks that cultivate deep reasoning without degrading factual recall remains an open challenge.
- Inference-Efficient Reasoning Trajectories: Vanilla SFT blindly mimics verbose teacher demonstrations, conditioning models to generate unnecessarily elongated reasoning chains that incur severe deployment latency and computational costs. Future research must develop techniques to distill prolonged reasoning chains into condensed, dense deductive paths without sacrificing rigor.
- Developing Genuine RL-Compatible SFT Objectives: Current SFT methods remain evaluated almost exclusively on post-SFT benchmark accuracy, ignoring compatibility with downstream RL. Future efforts must establish complete SFT-then-RL evaluation protocols and formulate objectives specifically designed to preserve exploratory policy entropy, diversity, and reward-responsiveness.
Related Work & Insights¶
- vs. RL-Centric Reasoning Paradigms (e.g., DeepSeek-R1, Xu et al. 2025a): While recent trends emphasize pure reinforcement learning (e.g., R1-Zero), empirical evidence demonstrates that unguided RL is computationally prohibitive and prone to reward hacking. Reasoning SFT serves as an indispensable prerequisite that instills structural syntax, basic verification routines, and format stability.
- vs. General LLM Post-Training Surveys (e.g., Kumar et al. 2026, Jiang et al. 2026a): General post-training reviews cover broad instruction-tuning, conversational chat, and preference modeling. This survey drills specifically into the unique optimization mechanics of long-CoT reasoning, formalizing token-level gradient dilution, distribution mismatch, and exploration entropy preservation.
- vs. Prompting and In-Context Reasoning Surveys (e.g., Plaat et al. 2025): Prompting literature explores inference-time heuristics (few-shot CoT, self-consistency). This survey focuses on parameter internalization, detailing how structural supervision reshapes the neural weight manifold to achieve durable, compute-efficient reasoning capabilities.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ The first dedicated, systematic survey on reasoning SFT, establishing a standardized mathematical optimization framework and an insightful 2D classification map.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducts a rigorous, controlled meta-analysis on a shared Qwen2.5-Math-7B baseline across 12 distinct fine-tuning paradigms, eliminating confounding experimental variables.
- Writing Quality: ⭐⭐⭐⭐⭐ Exemplary organization with deep mathematical formalization, cohesive prose narratives, and clear causal links connecting empirical insights to methodological designs.
- Value: ⭐⭐⭐⭐⭐ Delivers an essential theoretical and practical roadmap that rebalances post-training research from an exclusive focus on RL toward a principled, synergistic SFT-RL co-evolution.