INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Shea-code-xxx/INSPIRE-code
Area: LLM Reasoning
Keywords: Large Language Models, Mathematical Reasoning, Example-Driven Reasoning, Preference Optimization, Counterexample Construction
TL;DR¶
Addressing the core limitation where large language models rely excessively on abstract theorem invocation and outcome correctness rather than probing theorem preconditions, INSPIRE combines Reference-Guided Student Internalization to avoid distribution shift with a stage-wise rubric preference training strategy (M-DPO followed by C-DPO), substantially boosting counterexample-driven mathematical reasoning.
Background & Motivation¶
Mathematical reasoning constitutes a foundational benchmark for measuring artificial general intelligence in large language models (LLMs). Driven by recent advances in supervised fine-tuning and post-training reinforcement learning—such as DeepSeek-R1, OpenAI o1, and GRPO—state-of-the-art models exhibit extraordinary capabilities in long-chain formal derivations and competition-level problem solving. Nonetheless, nearly all prevailing optimization paradigms benchmark exclusively against final-answer correctness. Consequently, models frequently rely on surface-level solution heuristics and abstract theorem citations rather than internalizing genuine mathematical principles, leading to severe sycophancy and logical brittleness when subtle theorem boundary conditions are perturbed.
In human mathematics education, example-based reasoning—specifically formulating concrete instances or constructing targeted counterexamples to test theorem premises—is widely recognized as a litmus test for profound conceptual understanding. However, in contemporary LLMs, this capability remains strikingly underdeveloped. When presented with propositions containing implicit boundary conditions, such as the continuity of function quotients or monotone measure convergence, models reflexively apply abstract theorems without checking edge constraints. Directly adapting Direct Preference Optimization (DPO) to foster this capability introduces two fundamental technical dilemmas. First, due to the base model's intrinsic inability to produce effective examples, purely self-sampled candidates are uniformly low-quality, yielding negligible preference margins; conversely, importing external reference solutions from stronger teacher models introduces severe distribution shifts in lexical style and reasoning morphology, prompting models to memorize superficial tokens rather than internalize analytical strategies. Second, cognitive acquisition follows a strictly progressive curriculum: a model must first learn to actively deploy the example-construction strategy before it can master formulating precise, theorem-invalidating counterexamples. Collapsing both objectives into a single training step induces severe gradient conflict and strategy degradation.
To resolve these challenges, the authors propose INSPIRE, a principled Internalize-Then-Improve framework built upon stage-wise rubric preference optimization. Core idea: deploy Reference-Guided Student Internalization (RGSI) to generate high-quality on-policy candidate pools, followed by a stage-wise rubric preference curriculum that decouples capability acquisition into method-oriented preference learning (M-DPO) and correctness-oriented refinement (C-DPO), training models to first adopt example-based reasoning and subsequently sharpen counterexample validity.
Method¶
Overall Architecture¶
The INSPIRE framework integrates multi-dimensional rubric evaluation with a two-stage preference optimization pipeline. Given a mathematical proposition and its reference proof, the base student model generates responses via both standard self-sampling and Reference-Guided Student Internalization (RGSI). An evaluator LLM scores all candidates independently along three orthogonal axes: Method (\(m\)), Reasoning (\(r\)), and Correctness (\(c\)). In Stage 1, Method-oriented DPO (M-DPO) optimizes preference pairs filtered to maximize the divergence in example usage while enforcing reasoning quality gates, establishing a persistent habit of constructing concrete mathematical objects. In Stage 2, the Stage 1 checkpoint is resampled, and Correctness-oriented DPO (C-DPO) optimizes for final-answer correctness over the newly acquired example-driven strategy, augmented with Stage 1 replay samples to prevent catastrophic forgetting.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Math Proposition & Reference Solution Input"] --> B["Reference-Guided Student Internalization<br/>On-policy comprehension and candidate regeneration"]
B --> C["Three-Dimensional Rubric Evaluation<br/>Scoring Method, Reasoning, and Correctness"]
C --> D["Method-Oriented Preference Learning M-DPO<br/>Enforcing example adoption with reasoning gates"]
D --> E["Correctness-Oriented Preference Refinement C-DPO<br/>Correctness alignment combined with Stage 1 replay"]
E --> F["Final Policy Model π2<br/>Targeted counterexample construction & boundary verification"]
Key Designs¶
1. Reference-Guided Student Internalization: Synthesizing On-Policy High-Quality Candidates To overcome the severe trade-off between the low-quality homogeneity of on-policy self-sampling and the harmful distribution shifts induced by off-policy teacher demonstrations, INSPIRE introduces Reference-Guided Student Internalization (RGSI). Instead of directly treating external reference solutions as preferred pairs in preference learning, RGSI provides the base model \(\pi_0\) with both the mathematical problem \(x\) and a reference demonstration \(r\) as context, prompting the student model to independently comprehend the underlying approach and regenerate the complete derivation \(\tilde{y} \sim \pi_0(\cdot \mid x, r)\) in its own expressive voice and token distribution. This guarantees that candidate solutions assimilate the high-level example-construction schema of the reference while remaining strictly on-policy, providing clear, learnable quality gradients and stable margins for subsequent DPO training.
2. Three-Dimensional Rubric Evaluation: Decoupling Strategy, Coherence, and Correctness Conventional mathematical preference pipelines rely strictly on binary outcome correctness, remaining oblivious to whether a model engaged in concrete example analysis. Grounded in mathematical pedagogical assessment, INSPIRE formulates a structured three-dimensional scoring rubric. The Method dimension (\(m \in \{0, 1, 2\}\)) quantifies whether the proof introduces specific mathematical objects (such as concrete real numbers, functions, subsets, or matrices) and assesses whether they play an indispensable role in proving existence, refuting universality, or revealing critical properties. The Reasoning dimension (\(r \in \{0, 1, 2, 3\}\)) verifies deductive rigor and step-by-step logical completeness. The Correctness dimension (\(c \in \{0, 1\}\)) tracks final judgment validity. An automated judge model (DeepSeek-V3.2) independently evaluates candidate pools along these criteria to guide fine-grained pair construction.
3. Method-Oriented Preference Learning (M-DPO): Inducing Active Example Construction The primary objective of the initial preference stage is to break the model's chronic bias toward vacuous, abstract theorem invocation by instilling an active habit of example construction. Preference pairs \((y_w, y_l)\) are selected under strict method differentiation and quality gate conditions: $\(m(y_w) - m(y_l) \ge 1, \quad r(y_w) \ge 1, \quad r(y_l) \ge 1\)$ This formulation guarantees a decisive gap between example-employing and purely abstract responses, while the two-sided logic threshold screens out incoherent hallucinations. Among eligible candidate pairs, the selection prioritizes responses with higher reasoning scores \(r(y_w)\), exposing the model to sound logical exemplars as it acquires the method strategy to produce intermediate checkpoint \(\pi_1\).
4. Correctness-Oriented Preference Refinement (C-DPO): Calibrating Precondition Detection and Counterexample Accuracy While Stage 1 successfully encourages the model to generate concrete examples, empirical audits reveal that models initially generate trivial "confirmatory instances" that satisfy existing theorem conditions rather than testing boundary limitations. Stage 2 shifts optimization to rigorous answer verification. Using the Stage 1 model \(\pi_1\) to resample responses, correctness pairs are formed with \(c(y_w) = 1, c(y_l) = 0\), and \(r(y_w) \ge 1\). For instances where all candidates reach correct conclusions, quality pairs are mined with \(r(y_w) - r(y_l) \ge 1\). Crucially, to prevent the policy from abandoning its example-based reasoning habit when pursuing outcome accuracy, the training distribution \(D_2\) incorporates a replay fraction of Stage 1 pairs \(\text{Replay}(D_1)\), directing the policy to actively inspect omitted preconditions and construct targeted, lethal counterexamples.
Loss & Training¶
The entire optimization framework is implemented with LoRA using LLaMA-Factory across 4 NVIDIA L20 GPUs. Candidate scoring and verification leverage DeepSeek-V3.2, while inference evaluations employ vLLM with greedy decoding. Stage 1 M-DPO adopts the SFT checkpoint \(\pi_0\) as both initial policy and reference distribution, optimizing over method pairs \(D_1\): $\(\mathcal{L}_{\text{M-DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim D_1} \left[ \log \sigma \left( \beta_m \log \frac{\pi_\theta(y_w \mid x)}{\pi_0(y_w \mid x)} - \beta_m \log \frac{\pi_\theta(y_l \mid x)}{\pi_0(y_l \mid x)} \right) \right]\)$ yielding intermediate policy \(\pi_1\). Stage 2 C-DPO initializes from and references \(\pi_1\), optimizing over correctness and replay pairs \(D_2\): $\(\mathcal{L}_{\text{C-DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim D_2} \left[ \log \sigma \left( \beta_c \log \frac{\pi_\theta(y_w \mid x)}{\pi_1(y_w \mid x)} - \beta_c \log \frac{\pi_\theta(y_l \mid x)}{\pi_1(y_l \mid x)} \right) \right]\)$ Hyperparameters are set to a learning rate of \(1 \times 10^{-5}\) for SFT warm-up (3 epochs), and \(5 \times 10^{-6}\) for both DPO stages with regularizer temperatures \(\beta_m = \beta_c = 0.2\) (3 epochs each). Training data comprises 1,275 verified example/counterexample proof problems extracted from BrokenMath, yielding roughly 14,000 preference pairs per stage.
Key Experimental Results¶
Main Results¶
Evaluations on CounterMath—a benchmark expressly assessing example-driven and counterexample-based reasoning across four metrics (Macro-F1, Examples, Strict Alignment, and Loose Alignment)—demonstrate consistent, progressive gains across three representative model scales and architectures: Llama-3.1-8B-Instruct, Qwen2.5-Math-1.5B-Instruct, and Qwen2.5-Math-7B-Instruct.
| Model | Type / Scale | F1 (Macro) (%) | Examples (%) | Strict Align (%) | Loose Align (%) |
|---|---|---|---|---|---|
| Claude-4.5-Sonnet | Commercial Frontier | 74.24 | 72.94 | 39.30 | 45.64 |
| DeepSeek-R1 | Commercial / Large RL | 73.50 | 80.59 | 42.52 | 46.95 |
| Gemini-3-Pro | Commercial Frontier | 80.43 | 81.17 | 45.56 | 50.82 |
| Eurus-2-7B-PRIME | Open-source 7B | 36.75 | 62.83 | 14.63 | 17.35 |
| Qwen2.5-Math-72B-Instruct | Open-source 72B | 41.89 | 75.99 | 21.46 | 24.67 |
| Qwen3-32B | Open-source 32B | 45.93 | 76.97 | 25.41 | 30.67 |
| Llama-3.1-8B-Instruct | Base Foundation | 41.28 | 76.73 | 6.57 | 7.56 |
| + SFT | Cold-start Warmup | 42.98 (+1.70) | 80.10 (+3.37) | 7.40 (+0.83) | 8.31 (+0.75) |
| + Stage 1 (M-DPO) | Method Adoption | 48.15 (+6.87) | 89.88 (+13.15) | 8.22 (+1.65) | 9.78 (+2.22) |
| + Stage 1 + Stage 2 (Full) | Full INSPIRE | 50.27 (+8.99) | 93.09 (+16.36) | 9.45 (+2.88) | 11.80 (+4.24) |
| Qwen2.5-Math-1.5B-Instruct | Base Foundation | 35.71 | 69.24 | 9.04 | 10.60 |
| + SFT | Cold-start Warmup | 36.36 (+0.65) | 69.82 (+0.58) | 9.45 (+0.41) | 10.93 (+0.33) |
| + Stage 1 (M-DPO) | Method Adoption | 37.26 (+1.55) | 74.84 (+5.60) | 10.85 (+1.81) | 11.84 (+1.24) |
| + Stage 1 + Stage 2 (Full) | Full INSPIRE | 39.02 (+3.31) | 76.32 (+7.08) | 11.68 (+2.64) | 13.49 (+2.89) |
| Qwen2.5-Math-7B-Instruct | Base Foundation | 38.67 | 75.25 | 14.31 | 16.61 |
| + SFT | Cold-start Warmup | 39.06 (+0.39) | 77.63 (+2.38) | 14.39 (+0.08) | 17.02 (+0.41) |
| + Stage 1 (M-DPO) | Method Adoption | 43.05 (+4.38) | 81.33 (+6.08) | 16.53 (+2.22) | 19.08 (+2.47) |
| + Stage 1 + Stage 2 (Full) | Full INSPIRE | 45.91 (+7.24) | 84.79 (+9.54) | 17.30 (+2.99) | 20.07 (+3.46) |
Ablation Study¶
Ablations on Qwen2.5-Math-7B-Instruct dissect the roles of the training schedule, comparison baselines, and Stage 1 data augmentation schemes.
| Category | Strategy / Configuration | F1 (Macro) (%) | Examples (%) | Strict Align (%) | Loose Align (%) |
|---|---|---|---|---|---|
| Baseline Anchor | Base (+ SFT) | 39.06 | 77.63 | 14.39 | 17.02 |
| Training Strategy | Standard Correctness DPO (Single-stage outcome) | 39.38 | 73.19 | 15.21 | 17.11 |
| Chosen SFT (Supervised fine-tuning on chosen responses) | 41.75 | 79.52 | 15.04 | 18.09 | |
| Mixed DPO (Single-pass shuffled combination of Stage 1 & 2) | 42.72 | 81.17 | 15.21 | 17.92 | |
| C-DPO Only (Correctness-oriented single stage) | 40.27 | 77.88 | 15.13 | 17.43 | |
| M-DPO Only (Method-oriented single stage) | 43.05 | 81.33 | 16.53 | 19.08 | |
| Full INSPIRE (Stage 1 + Stage 2) | 45.91 | 84.79 | 17.30 | 20.07 | |
| Data Augmentation | Pure Self-sampling (Unassisted model generation) | 40.40 | 78.55 | 15.54 | 17.68 |
| Teacher Rewriting (DeepSeek-R1 rewrites references) | 39.14 | 74.01 | 15.21 | 17.51 | |
| Student Internalization RGSI (Ours) | 43.05 | 81.33 | 16.53 | 19.08 |
Key Findings¶
- Example utilization directly gates reasoning performance: CounterMath accuracy strongly correlates with example utilization rates across all model families. Standard correctness-only DPO actively represses example usage (dropping Examples from 77.63% to 73.19%), illustrating that training exclusively on outcome correctness incentivizes shortcut heuristics; conversely, INSPIRE elevates Examples to 84.79% on Qwen-7B and 93.09% on Llama-8B, both outstripping DeepSeek-R1 (80.59%).
- Stage-wise decoupling prevents conflicting gradient dynamics: Shuffling both stages into Mixed DPO yields an F1 of 42.72, falling short of Stage 1 alone (43.05). This confirms that optimizing simultaneously for strategy adoption and answer correctness produces adversarial gradient interference; models must first acquire the operational habit before calibrating factual precision.
- Distribution mismatch invalidates preference optimization: Teacher rewriting from DeepSeek-R1 produces an excessively wide margin and instantaneous training accuracy saturation, signaling that the model differentiates pairs by superficial phrasing rather than reasoning quality, plateauing at 39.14 F1. In contrast, RGSI sustains moderate margins and gradual accuracy progression throughout training.
- Out-of-distribution reasoning experiences robust positive transfer: Across GSM8K, MATH500, AIME 2024, GAOKAO-mathQA, and MMLU-collegeMath, INSPIRE incurs zero degradation. On Qwen2.5-Math-7B, AIME 2024 improves from 13.33% to 20.00% (+6.67) and GAOKAO-mathQA increases by +2.00, demonstrating that example-driven critique strengthens general competitive problem-solving.
Highlights & Insights¶
- RGSI Resolves the Cold-Start Distribution Dilemma: Conditioning the student policy on reference hints to regenerate proofs natively allows weak base models to produce elevated reasoning quality without drifting outside their generative distribution, offering a plug-and-play recipe for preference pair mining in capability-scarce domains.
- Stage-Wise Curricula Mirror Pedagogical Cognitive Progression: Splitting preference learning into method habituation (M-DPO) followed by boundary accuracy refinement (C-DPO) mirrors human conceptual education, demonstrating that complex reasoning strategies cannot be brute-forced through monolithic loss functions.
- Qualitative Shift from Theorem Confirmation to Theorem Refutation: Detailed case studies reveal that Stage 1 models construct benign examples that satisfy known preconditions (verifying theorems), whereas Stage 2 models actively identify omitted boundary assumptions (e.g., \(m(E_1) < \infty\) in measure continuity) and engineer divergent infinite counterexamples (\(E_n = [n, \infty)\)) to falsify invalid claims.
Limitations & Future Work¶
- Domain Scope and Dataset Breadth: Training was conducted on 1,275 problems derived from BrokenMath, primarily encompassing real analysis and calculus; extending the rubric methodology to discrete mathematics, abstract algebra, and combinatorics remains an open challenge.
- Scale Validation Limits: Empirical evaluations focused on 1.5B, 4B, 7B, and 8B parameter models; establishing scaling trajectories on larger dense (32B/72B) and mixture-of-experts (MoE) architectures requires future compute exploration.
- Judge Model Dependency: Pipeline construction and primary evaluation rely on DeepSeek-V3.2. While cross-validation with GPT-4o verified that performance gains are entirely judge-invariant, future iterations could integrate self-rewarding verification mechanics to reduce judge inference costs.
Related Work & Insights¶
- vs DeepSeek-R1 / GRPO Reinforcement Learning: Large-scale RL on outcome rewards fosters long chain-of-thought exploration, but absent structural method constraints, models frequently hallucinate extensive algebraic rationalizations that still overlook premise violations. INSPIRE activates counterexample validation with minimal data (1,275 problems) and modest compute.
- vs Step-DPO / Process Reward Models: Process-level DPO provides step-wise correctness supervision along fixed reasoning paths, whereas INSPIRE supervises high-level problem-solving strategy (deploying concrete constructions) via multi-dimensional rubrics, operating at the meta-reasoning tier.
- vs Rubric-Anchored Preference Learning: Rubric-based rewards have primarily targeted qualitative alignment in open-domain chat; this work establishes their concrete utility in formal mathematics to disentangle strategy acquisition from outcome verification.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Addresses example-driven reasoning via on-policy RGSI and decoupled two-stage preference optimization, presenting a fresh solution to distribution mismatch and progressive learning)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across multiple scales and model families, covering OOD benchmarks, training dynamics, judge cross-validation, and qualitative trajectory audits)
- Writing Quality: ⭐⭐⭐⭐⭐ (Cohesive structure, lucid pedagogical framing, and rigorous analytical depth)
- Value: ⭐⭐⭐⭐☆ (Provides a practical, data-efficient blueprint to eliminate model sycophancy and sharpen mathematical boundary testing in LLMs)