Skip to content

TTSR: Test-Time Self-Evolving via Reflection

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Henryhe09/TTSR
Area: LLM Reasoning
Keywords: Test-Time Training, Self-Reflection, Curriculum Synthesis, Weakness Memory, Test-Time Reinforcement Learning

TL;DR

Addressing the dual bottlenecks of learnable sample scarcity and undirected exploration in test-time training on challenging reasoning tasks, TTSR introduces a self-evolving framework where a single model alternates between Student and Teacher roles—diagnosing failure traces to synthesize frontier-calibrated curriculum variants, accumulated across iterations via a weakness memory and dynamic strategy notes.

Background & Motivation

Test-time training (TTT) and test-time reinforcement learning (TTRL) enable pretrained large language models to adapt their parameters during inference using only unlabeled test queries, providing an appealing path toward autonomous test-time improvement without ground-truth supervision. However, existing test-time adaptation methods yield marginal and unstable gains on challenging reasoning benchmarks such as Olympiad-level mathematics. This failure stems from two fundamental bottlenecks: first, the "lack of learnable samples," where target test questions lie far beyond the model's reliable solving boundary, causing majority-voting consensus labels to become severely noisy and providing uninformative zero-variance reward signals; and second, "inefficient exploration via repeated rollouts," where existing methods sample extensive reasoning trajectories without diagnosing why earlier attempts failed, leaving exploration blind and prone to repeating identical logical errors.

The core tension underlying these limitations is that directly updating model parameters on intractable target problems forces gradient updates into noisy collapse, as the model cannot bridge the capability gap from scratch; conversely, educational principles and human cognition show that diagnosing where a reasoning step derailed is far easier than constructing an entire proof from nothing, and curriculum learning proves that mastering intermediate stepping stones unlocks solutions to harder target problems. Nevertheless, prior self-play and curriculum generation frameworks heavily rely on offline multi-model architectures, abundant computation budgets, and supervised ground-truth annotations, rendering them inapplicable to the resource-constrained, strictly online, and label-free test-time adaptation setting.

This paper's angle of attack is to reformulate test-time reasoning into an internal closed-loop "reflect-then-synthesize" cycle within a single pretrained backbone, enabling the model to self-diagnose reasoning bugs and construct personalized stepping stones near its capability frontier. Core idea: alternate a single pretrained model between Student and Teacher roles at test time, where the Teacher diagnoses the Student's failed trajectories to synthesize targeted variant questions calibrated to its capability frontier via normalized variance rewards, while accumulating persistent diagnostic knowledge across iterations into an automatically fading strategy note.

Method

Overall Architecture

TTSR operates as a closed-loop self-evolution pipeline where a single shared policy alternates between two roles across test-time iterations. At iteration \(t\), the Student role receives test instances along with the current strategy note \(h_t\), generating multi-path rollouts across a mixed dataset of original test queries and previously synthesized variants to perform online Group Relative Policy Optimization (GRPO) using self-supervised pseudo-correctness rewards. Subsequently, the Teacher role analyzes trajectories with zero consensus rewards, extracts structured weakness descriptors, updates a cross-iteration weakness memory, and synthesizes targeted variant problems tuned to the Student's capability frontier, which are pre-evaluated and cached for the next adaptation cycle.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Test Questions X_test and Strategy Note h_t"] --> B["Student Rollout & Majority Voting<br/>Compute pseudo-correctness score s_t(x)"]
    B --> C["Student Online GRPO Policy Update<br/>Relative advantage updates under pseudo-rewards"]
    C --> D["Reflection-Guided Question Synthesis<br/>Teacher extracts weakness descriptor W_t from failures"]
    D --> E["Cross-Iteration Weakness Memory & Note Update<br/>Merge, resolve, evict, and compile active note h_t"]
    E --> F["Difficulty Reward & Sequence Similarity Penalty<br/>Filter frontier variants and update Teacher policy"]
    F -->|Output variants and cache pre-evaluations| B

Key Designs

1. Reflection-Guided Question Synthesis: Diagnosing Error Traces to Construct Stepping Stones

Rather than blindly drawing unguided rollouts on intractable questions or applying superficial text paraphrasing, TTSR leverages the Teacher role to perform process-level diagnosis on failed rollouts where consensus pseudo-reward is zero. The Teacher parses the failed reasoning steps to generate a structured JSON weakness descriptor \(W^{(t)}\) comprising an abstract reasoning weakness summary, triggering conditions, failure signatures, and localization summaries (e.g., pinpointing incomplete back-substitution in quartic equations). Conditioned on this diagnosis and the accumulated memory, the Teacher alters problem conditions and algebraic formulations to synthesize a self-contained variant question \(x'\). The variant preserves the underlying mathematical structure of the original query while explicitly targeting the diagnosed blind spot, ensuring the problem sits near the model's capability frontier and provides reliable, learnable gradients.

2. Normalized Variance Difficulty Reward and Similarity Penalty: Enforcing Frontier Alignment

To prevent the Teacher from synthesizing either trivially easy problems (which yield identical positive rewards across all rollouts) or intractable questions (which repeat the noisy pseudo-label failure), TTSR formulates an intrinsic frontier difficulty reward derived from the Student's empirical consensus score \(s_t(x')\): $\(R_{\text{diff}}(x') = 4 s_t(x') (1 - s_t(x'))\)$ where \(s_t(x') = \frac{1}{G}\sum_{i=1}^G \mathbb{I}[a(y_i) = \hat{a}(x')]\) denotes the fraction of trajectories agreeing with the majority answer. Theoretical analysis establishes that \(R_{\text{diff}}\) equals exactly \(4 \times \widehat{\text{Var}}(R)\), representing the normalized within-group binary reward variance. The reward is maximized at \(s_t(x') = 0.5\), precisely where the ratio of positive to negative rollout comparisons is highest and GRPO yields the strongest relative contrast signal. To eliminate redundant or degenerative generations, TTSR incorporates a sequence similarity penalty \(P_{\text{sim}}(x', x)\) based on recursive longest contiguous matching spans, yielding the overall Teacher reward \(R_T(x') = \max(0, R_{\text{diff}}(x') - \lambda P_{\text{sim}}(x', x))\), which penalizes trivial paraphrasing while permitting shared structural patterns.

3. Cross-Iteration Weakness Memory and Strategy Note: Scaffolding Reasoning and Natural Fading

Because per-iteration reflections are ephemeral and prone to being forgotten as adaptation progresses, TTSR maintains a bounded weakness memory \(M_t\) across iterations. In each iteration, newly extracted weakness descriptors are matched against existing entries: matching descriptors update a persistence counter and timestamp (Merge), entries unobserved for \(\Delta\) consecutive rounds are pruned as mastered (Resolve), and excess entries beyond budget \(K\) are dropped by lowest persistence (Evict). At the start of each iteration, the top-\(N\) most persistent weaknesses are compiled into a concise strategy note \(h_t\) and prepended to all Student prompt inputs. This note functions as an external cognitive scaffold—reminding the model of recurring pitfalls (e.g., verifying negative roots under even substitutions). As the policy's internal weights are successfully adapted via GRPO, the weaknesses cease to manifest, causing the corresponding rules to automatically drop from the note without cluttering context.

Loss & Training

Both Student and Teacher roles update the shared policy parameters \(\theta\) via GRPO without a separate critic model. The Student optimizes on a combined training batch \(D_t = X_{\text{test}} \cup X_{\text{var}}^{(t-1)}\) using pseudo-correctness rewards \(R_S(y_i \mid x) = \mathbb{I}[a(y_i) = \hat{a}(x)]\): $\(\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}\left[ \frac{1}{G}\sum_{i=1}^G \min\left(r_i(\theta)\tilde{A}_i, \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon)\tilde{A}_i\right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\theta_{\text{old}}}) \right]\)$ where \(\tilde{A}_i = \frac{R_i - \text{mean}(R)}{\text{std}(R) + \delta}\). The Teacher is subsequently optimized using \(R_T\) on the synthesized batch. To alleviate the overhead of double rollout sampling, TTSR employs a pre-evaluation rollout-reuse caching strategy: the Student rollouts sampled to evaluate Teacher difficulty reward \(R_{\text{diff}}\) are cached and reused directly as training data in iteration \(t+1\), restricting the total runtime cost per iteration to approximately \(3.0\times\) that of vanilla TTRL.

Key Experimental Results

Main Results

Evaluation spans six mathematical reasoning benchmarks (AMC23, MATH-500, Minerva, OlympiadBench, AIME 2024, AIME 2025) and two general-domain reasoning benchmarks (GPQA-Diamond, MMLU-Pro), across Qwen3-4B-Base, Qwen3-8B-Base, and OctoThinker-8B-Hybrid-Base. AIME 2024/2025 report Mean@32 accuracy; all other benchmarks report greedy decoding Pass@1.

Base Backbone Method AMC23 MATH500 Minerva Olympiad AIME24 AIME25 GPQA-D MMLU-Pro Avg Gain \(\Delta\)
Qwen3-4B-Base Base Model 45.0 72.0 32.4 40.2 12.4 5.8 25.8 52.0 –
R-Zero 55.0 76.8 40.8 44.1 18.2 9.1 28.3 55.0 +5.2
TTRL 55.0 79.0 43.8 46.0 17.6 14.7 29.3 56.0 +7.0
TTSR (Ours) 60.0 82.4 52.9 45.3 25.6 17.6 34.3 60.8 +11.7
Qwen3-8B-Base Base Model 52.5 77.8 39.7 41.2 15.9 9.8 33.3 58.6 –
R-Zero 57.5 82.0 47.8 48.7 22.6 13.4 36.9 61.5 +5.2
TTRL 62.5 84.2 50.4 50.7 26.1 15.7 38.4 62.8 +7.8
TTSR (Ours) 67.5 87.4 54.8 55.2 30.8 19.1 42.4 66.7 +11.9
OctoThinker-8B Base Model 27.5 45.6 15.1 16.8 7.1 3.9 15.2 26.8 –
R-Zero 35.0 54.0 22.8 25.4 11.6 6.8 19.7 31.4 +6.1
TTRL 37.5 57.2 26.5 28.6 13.9 8.2 21.7 33.2 +8.6
TTSR (Ours) 47.5 64.8 34.2 36.8 19.7 12.4 27.8 39.8 +15.6

Ablation Study

Full component-level ablation on Qwen3-8B-Base isolates the contribution of each architectural module:

Configuration Variant MATH500 AIME25 Olympiad GPQA-Diamond Component Impact
TTSR (Full System) 87.4 19.1 55.2 42.4 Full co-evolutionary pipeline
w/o Reflection-Guided Synthesis 85.0 (↓2.4) 14.2 (↓4.9) 49.9 (↓5.3) 38.4 (↓4.0) Reverts to random structural variations
w/o Teacher Test-Time Update 82.8 (↓4.6) 13.3 (↓5.8) 49.4 (↓5.8) 37.9 (↓4.5) Static frozen Teacher creates mismatched variants
w/o Difficulty Reward 85.2 (↓2.2) 15.4 (↓3.7) 51.8 (↓3.4) 39.9 (↓2.5) Loses selection of frontier-difficulty queries
w/o Similarity Penalty 86.0 (↓1.4) 16.4 (↓2.7) 52.8 (↓2.4) 40.4 (↓2.0) Redundant near-duplicate questions degrade diversity
w/o Strategy Note 86.4 (↓1.0) 17.2 (↓1.9) 53.6 (↓1.6) 41.4 (↓1.0) Drops immediate in-context prompt guidance
+ Synthesis Only (8B Teacher, No Reflection) 82.2 12.8 47.6 35.9 Fails to match Student's specific weak spots
+ Strong Synthesizer (14B Teacher, No Reflection) 84.6 15.0 51.0 37.4 Scaling model capacity plateaus without adaptation

Key Findings

  • Pronounced Advantages on Frontier Competition Benchmarks: TTSR achieves its largest margins on AIME 2024 and AIME 2025. On Qwen3-8B, AIME24 improves from 15.9% to 30.8% (+4.7 over TTRL), and AIME25 increases from 9.8% to 19.1% (+3.4 over TTRL), demonstrating that synthesized frontier variants break through the zero-signal deadlock of intractable target problems.
  • Co-Evolution Dominates Raw Model Scaling: Freezing the Teacher model at iteration 0 produces the steepest performance drop (up to −5.8 points on AIME25 and Olympiad). Crucially, TTSR with an adaptive, co-evolving 8B Teacher (87.4% on MATH500) significantly outperforms a static, non-reflecting 14B Teacher (+Strong Synthesizer at 84.6%), proving that dynamically aligning curriculum difficulty with the Student's evolving boundary matters more than raw teacher model size.
  • Data Efficiency and OOD Generalization: On AIME24 with Qwen3-4B, TTSR trained on only 10% of test data achieves 12.3% accuracy, outperforming TTRL trained on 10% (9.5%). Furthermore, models adapted on Olympiad exhibit strong zero-shot transfer to AIME24 (25.4% vs TTRL's 22.1%), and models trained solely on AIME25 show notable cross-domain transfer to GPQA-Diamond (+3.1) and MMLU-Pro (+2.6), proving the acquisition of generalized reasoning strategies rather than dataset-specific memorization.

Highlights & Insights

  • Unified Reflection-Synthesis Curriculum Closed-Loop: Replaces undirected rollout sampling with systematic error diagnosis, converting failed traces into structured weakness descriptors that directly condition targeted problem synthesis.
  • Rigorous Mathematical Formulation of the Capability Frontier: Proves that the difficulty reward \(R_{\text{diff}} = 4s(1-s)\) is strictly equivalent to four times the empirical reward variance, maximizing the relative comparison pairs for GRPO policy updates.
  • Dynamic Fading Cognitive Scaffolding: Implements a weakness memory mechanism that compiles recurring deficiencies into lightweight strategy notes, guiding test-time exploration and automatically resolving as model parameters absorb the capability.

Limitations & Future Work

  • Compute Overhead Relative to Single-Pass Inference: Generating variants and pre-evaluating rollouts increases per-iteration compute to approximately \(3.0\times\) that of standard TTRL, limiting direct applicability in latency-critical real-time settings.
  • Dependence on Structured Reflection Expressiveness: The failure localization template primarily targets isolated algebraic, arithmetic, or constraint-omission errors; diagnosing multi-step proof divergences spanning complex branched reasoning chains remains challenging.
  • Scope Primarily Centered on Formal Reasoning: Empirical verification is currently focused on mathematical and STEM scientific reasoning; expanding the paradigm to interactive decision-making, code refactoring, and embodied tasks is an open frontier.
  • vs TTRL (Zuo et al., 2025): TTRL relies solely on majority voting over original test queries, which produces flat zero-variance gradients on problems beyond current model reach. TTSR introduces reflection-guided variant synthesis to create learnable intermediate steps, maintaining steady optimization progress.
  • vs TTCS (Yang et al., 2026): TTCS synthesizes test-time curricula via generic question rewriting without failure diagnosis. In contrast, TTSR conditions synthesis on explicit diagnostic descriptors extracted from failed traces, boosting weakness targeting from 22% to 83%.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers a failure-conditioned, self-evolving test-time curriculum framework with cross-iteration weakness memory and frontier variance optimization]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across eight major benchmarks, three distinct model backbones, rigorous iso-compute baselines, and GPT-5.2 external judge audits]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical derivations, comprehensive casework analysis, and self-contained architectural explanations]
  • Value: ⭐⭐⭐⭐⭐ [Establishes an effective, data-free paradigm for scaling inference-time compute and parameter adaptation on complex reasoning frontiers]