ADORA: Training Reasoning Models with Dynamic Advantage Estimation on Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ShadeCloak/ADORA
Area: LLM Reasoning / Reinforcement Learning (RL)
Keywords: Reinforcement Learning, Advantage Estimation, Dynamic Reweighting, Long Chain-of-Thought, Sample Efficiency
TL;DR¶
ADORA introduces a plug-and-play RL framework that dynamically calibrates advantage estimation weights based on live rollout statistics, categorizing training examples into Temporarily Advantageous Samples (TAS) and Temporarily Disadvantageous Samples (TDS) to suppress shallow shortcuts in weak models and amplify challenging deliberation in strong models, achieving state-of-the-art reasoning gains with exceptional sample efficiency.
Background & Motivation¶
Reinforcement learning (RL) combined with rule-based outcome rewards and long chain-of-thought (CoT) reasoning has emerged as the cornerstone paradigm for eliciting advanced reasoning capabilities in large language models (LLMs) and vision-language models (VLMs). Through iterative policy gradient updates under algorithms such as PPO and GRPO, models internalize sophisticated deliberation patterns, including self-verification, reflective backtracking, and multi-step deduction. The optimization effectiveness of these policy gradient methods fundamentally hinges on the precision of credit assignment provided by the advantage function. However, existing frameworks universally adopt a static advantage estimation formulation, wherein advantages are computed via fixed group normalizations or uniform importance weighting across the entire training lifecycle.
This static advantage paradigm introduces a fundamental disconnect with the non-stationary nature of reinforcement learning. As policy optimization proceeds, the model's intrinsic competency boundary continuously shifts, meaning that the informational utility and marginal learning signal offered by any given training instance evolve dynamically across iterations. Early in training, certain problems reside within the model's proximal zone of development and provide critical gradient signals for acquiring nascent reasoning skills, while others represent either trivial patterns already mastered or noisy outliers currently beyond the policy's reachable exploration space. Treating all successful rollouts uniformly blinds the optimizer to this shifting utility: weaker models frequently overfit to superficial visual or textual shortcuts that yield quick rewards without genuine reasoning, while stronger models hit optimization plateaus dominated by uninformative, redundant successes.
To resolve this bottleneck, this paper argues that credit assignment must adapt concurrently with the policy's evolving capabilities. Rather than relying on rigid, pre-computed difficulty heuristics or expensive multi-stage curriculum teacher pipelines that fail to track rapid policy drift, the framework leverages the natural empirical statistics of online rollouts to perform real-time sample differentiation. Core idea: propose the ADORA framework to dynamically calibrate advantage weights via live rollout statistics, using complementary length and difficulty advantage criteria to categorize instances into temporarily advantageous and disadvantageous samples, applying attenuation to denoise weak models and amplification to break learning plateaus in strong models without introducing learnable parameters or breaking policy gradient unbiasedness.
Method¶
Overall Architecture¶
ADORA operates as a modular, plug-and-play enhancement seamlessly embedded into standard on-policy RL algorithms (such as GRPO, PPO, DAPO, or GSPO). During each training step, the current policy \(\pi_{\theta_{\text{old}}}\) samples a group of \(G\) candidate responses \(\{o_i\}_{i=1}^G\) for an input query \(q\). Rule-based verifiers evaluate the final answers to assign rewards \(\{R_i\}_{i=1}^G\), from which normalized group advantages \(\hat{A}_s\) are derived. ADORA then inspects the structural rollout properties of the sample—specifically the lengths of successful versus unsuccessful trajectories and the group success rate—to classify the instance into Temporarily Advantageous Samples (TAS) or Temporarily Disadvantageous Samples (TDS). An instance-level scalar weight \(w_s\) is assigned dynamically to produce calibrated advantages \(\tilde{A}_s = w_s \cdot \hat{A}_s\), which are then ingested by the standard clipped surrogate objective to update the policy parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query and Policy Rollout<br/>Sample G trajectories via π_θold"] --> B["Rule-based Reward Normalization<br/>Compute base advantages Â_s"]
B --> C["Sample Differentiation Criteria<br/>Assess Length and Difficulty Advantages"]
C --> D["Model-Adaptive Advantage Reweighting<br/>Weaken TDS for VLMs / Amplify TAS for LLMs"]
D --> E["Calibrated Advantage Ã_s = w_s · Â_s<br/>Feed into clipped surrogate objective"]
E --> F["Policy Parameter Update θ_{t+1}"]
Key Designs¶
1. Sample Differentiation Criteria: Decoupling Genuine Deliberation and Solvability Boundaries
To distinguish high-utility learning opportunities from low-value trajectories without external annotators, ADORA introduces two lightweight, complementary criteria evaluated directly on the current batch's rollouts. When advantage weights are static, short or superficial trajectories that stumble upon correct answers via heuristic shortcuts receive identical credit to thorough deductions, encouraging the model to collapse into degenerate shortcut behaviors. ADORA observes that genuine deliberation typically manifests as extended search and verification, defining a sample \(s\) as possessing a Length Advantage (\(\text{Len}_{\text{adv}}\)) if the longest successful rollout exceeds the average length of failed rollouts:
Because length alone cannot rule out over-learned, trivial problems where the model is already competent, ADORA introduces the Difficulty Advantage (\(\text{Diff}_{\text{adv}}\)), which flags samples that remain actively challenging for the current policy:
where \(R_{s}^{\text{succ}}\) represents the empirical success rate across the \(G\) rollouts for sample \(s\), and \(\tau \in (0, 1]\) is a predefined difficulty boundary (empirically optimal at \(\tau = 0.5\)). Samples with zero successes (\(R_{s}^{\text{succ}} = 0\)) offer no positive gradient, while samples with \(R_{s}^{\text{succ}} > \tau\) indicate concepts already internalized. Only instances lying in the intermediate challenge bracket yield productive exploratory gradients.
2. Attenuation Strategy for Weaker Reasoning Regimes (VLMs): Denoising and Suppressing Superficial Shortcuts
Multimodal vision-language models typically exhibit weaker intrinsic zero-shot reasoning capabilities at the onset of RL fine-tuning. Early in training, rollouts that bypass rigorous geometric and mathematical deduction but exploit spurious visual correlations to hit the correct answer can disproportionately dominate the gradient signal, preventing the policy from developing robust multi-step thinking. To counter this, ADORA frames optimization for weak models as a denoising process. Any sample failing the Length Advantage criterion is designated as a Temporarily Disadvantageous Sample (TDS), and its advantage signal is systematically attenuated using a scaling factor \(\lambda_{\text{att}} \in (0, 1)\) (set to 0.1):
By down-weighting superficial successes, this attenuation mechanism suppresses gradient updates driven by shortcut-oriented trajectories, forcing the model to depend on comprehensive, verifiable chains of thought to secure meaningful policy improvement.
3. Amplification Strategy for Stronger Reasoning Regimes (LLMs): Breaking Optimization Plateaus via High-Utility Focus
In contrast to VLMs, modern text-based LLMs initialize with substantial reasoning proficiency and naturally produce extended reasoning trajectories. Consequently, the primary training challenge is not filtering out rudimentary noise, but overcoming learning plateaus caused by diminishing marginal returns from easy, familiar instances. To push the policy beyond its comfort zone, ADORA applies an amplification strategy to strong reasoning models. Instances that simultaneously satisfy both the Length Advantage and Difficulty Advantage criteria represent the gold standard of training utility—they are genuinely difficult problems that the model can nevertheless solve through thorough, extended deliberation. ADORA designates these as high-value TAS and amplifies their advantage signal via a hyperparameter \(\lambda_{\text{amp}} > 1\) (set to 2.0):
This selective amplification strengthens the optimization signal along frontier reasoning trajectories, accelerating the internalization of reflective and self-correcting mechanisms without destabilizing the policy.
Loss & Training¶
ADORA integrates directly into standard on-policy surrogate objectives. Taking Group Relative Policy Optimization (GRPO) as the primary instantiation, the policy samples \(G\) rollouts per prompt, with raw token-level advantages normalized across the group: \(\hat{A}_i = \frac{R_i - \text{mean}(\{R\})}{\text{std}(\{R\})}\). ADORA scales these values by the instance-level scalar \(w_s\), yielding the reweighted objective:
where \(\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}\) is the importance weight, and \(\tilde{A}_{i,t} = w_s \cdot \hat{A}_{i,t}\). Because \(w_s\) is a sample-level constant derived from the forward rollouts without computational graph dependency on \(\theta\), it acts mathematically as a dynamic resampling distribution, preserving the unbiasedness of the policy gradient estimator. Implementation is based on the verl framework executed on 8 NVIDIA A100 (80GB) GPUs. VLM training utilizes only 2,000 samples from Geometry3K, while LLM training uses 12,000 samples from the MATH500 training set. Hyperparameters are fixed at rollout count \(G=8\), learning rate \(1.0 \times 10^{-6}\), temperature \(1.0\), \(\tau=0.5\), \(\lambda_{\text{att}}=0.1\), and \(\lambda_{\text{amp}}=2.0\).
Key Experimental Results¶
Main Results¶
ADORA was evaluated across vision-language geometry tasks and large language model mathematical reasoning tasks. All reported metrics reflect Avg@3 evaluations under greedy decoding (temperature 0).
VLM Geometry Reasoning Benchmarks (Avg@3 Accuracy %)
| Model | Method | MathVista | MathVerse | MathVerse (Vision-Only) | DynaMath | Overall Avg. |
|---|---|---|---|---|---|---|
| Gemma3-4b-it | Zero-shot Base | 46.3 | 25.2 | 13.5 | 10.5 | 23.88 |
| Vanilla GRPO | 47.2 | 24.9 | 13.6 | 11.0 | 24.18 | |
| + ADORA | 48.3 (+1.1) | 26.1 (+1.2) | 14.5 (+0.9) | 12.2 (+1.2) | 25.28 (+1.10) | |
| Internvl3-2b | Zero-shot Base | 57.0 | 32.5 | 25.3 | 14.6 | 32.35 |
| Vanilla GRPO | 60.7 | 34.7 | 30.7 | 15.1 | 35.30 | |
| + ADORA | 64.8 (+4.1) | 39.2 (+4.5) | 34.9 (+4.2) | 18.1 (+3.0) | 39.25 (+3.95) | |
| Qwen2.5-VL-7B | Zero-shot Base | 67.3 | 46.3 | 40.2 | 50.3 | 51.00 |
| Vanilla GRPO | 70.2 | 48.2 | 44.1 | 53.3 | 54.00 | |
| + ADORA | 73.5 (+3.3) | 52.9 (+4.7) | 48.6 (+4.5) | 58.7 (+5.4) | 58.40 (+4.40) |
LLM Mathematical Reasoning Benchmarks (Avg@3 Accuracy %)
| Model | Method | GSM8K | MATH500 | AMC23 | CollegeMath | OlympiadBench | AIME24 | Overall Avg. |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-Math-7B | Zero-shot Base | 28.4 | 19.6 | 10.0 | 12.0 | 3.0 | 0.0 | 19.83 |
| Vanilla GRPO | 68.2 | 39.5 | 20.0 | 29.8 | 12.0 | 3.3 | 28.80 | |
| + ADORA | 68.5 (+0.3) | 41.8 (+2.3) | 25.0 (+5.0) | 31.6 (+1.8) | 12.9 (+0.9) | 3.3 (+0.0) | 30.52 (+1.72) | |
| Mistral-v0.1-7B | Zero-shot Base | 21.2 | 5.4 | 0.0 | 3.8 | 2.4 | 0.0 | 5.47 |
| Vanilla GRPO | 54.0 | 26.8 | 10.0 | 11.4 | 4.1 | 0.0 | 17.72 | |
| + ADORA | 53.8 (-0.2) | 30.4 (+3.6) | 10.0 (+0.0) | 12.4 (+1.0) | 4.7 (+0.6) | 0.0 (+0.0) | 18.55 (+0.83) | |
| Llama-3.1-8B | Zero-shot Base | 40.2 | 12.7 | 2.5 | 6.4 | 3.1 | 0.0 | 10.82 |
| Vanilla GRPO | 66.1 | 33.8 | 15.0 | 22.0 | 5.3 | 0.0 | 23.72 | |
| + ADORA | 66.7 (+0.6) | 39.4 (+5.6) | 15.0 (+0.0) | 23.1 (+1.1) | 10.5 (+5.2) | 0.0 (+0.0) | 25.78 (+2.06) | |
| Qwen2.5-7B | Zero-shot Base | 56.3 | 57.2 | 37.5 | 24.3 | 26.3 | 10.0 | 35.27 |
| Vanilla GRPO | 89.1 | 73.2 | 50.0 | 28.6 | 35.1 | 13.3 | 48.22 | |
| + ADORA | 89.6 (+0.5) | 76.2 (+3.0) | 62.5 (+12.5) | 29.3 (+0.7) | 36.0 (+0.9) | 16.7 (+3.4) | 51.72 (+3.50) |
Ablation Study¶
The authors conducted extensive ablations on advantage criteria, algorithm integration, and data scaling.
1. Advantage Criteria Formulation (LLM vs. VLM Regimes)
| Task Regime / Model | Criteria Configuration | Overall Accuracy (%) | Primary Mechanism & Impact |
|---|---|---|---|
| LLM (Qwen2.5-7B) | Vanilla GRPO | 48.22 | Static baseline estimation |
| + Length Adv. only | ~49.65 | Benefits moderately challenging problems | |
| + Difficulty Adv. only | ~50.25 | Substantial gains on competitive tasks (AMC23/AIME24) | |
| + Joint (Length & Difficulty) | 51.72 | Optimal synergy: deep deliberation on frontier problems | |
| VLM (Qwen2.5-VL-7B) | Vanilla GRPO | 54.00 | Susceptible to superficial visual shortcuts |
| + Length Adv. only | 58.40 | Critical denoising: filters out shallow, non-deductive successes | |
| + Joint (Length & Difficulty) | 58.43 | Negligible marginal gain over Length Adv. alone |
2. Integration Across Modern RL Algorithms (Figure 4 Ablation)
| Base Policy Algorithm | Vanilla Score (%) | + ADORA Score (%) | Net Absolute Gain |
|---|---|---|---|
| GRPO | 48.22 | 51.72 | +3.50 |
| DAPO | 57.30 | 59.70 | +2.40 |
| GSPO | 50.50 | 53.10 | +2.60 |
3. Training Data Scalability & Sample Efficiency (Qwen2.5-VL-7B, Table 6)
| Data Volume | GRPO Accuracy (%) | ADORA Accuracy (%) | Key Insight |
|---|---|---|---|
| 2k samples | 54.0 | 58.4 | ADORA (2k) outperforms GRPO (10k at 55.3%) by +3.1 points |
| 10k samples | 55.3 | 59.4 | Marginal utility continues to scale with expanded data |
Key Findings¶
- Unprecedented Data Efficiency: Competing multimodal reasoning frameworks consume enormous data resources: MM-EUREKA requires 54k cold-start examples + 9.3k RL samples, and Vision-R1 uses 200k synthetic cold-start instances + 10k RL samples. In stark contrast, ADORA achieves 73.5% on MathVista and 52.9% on MathVerse using zero cold-start data and only 2,000 open-source Geometry3K samples, setting a new benchmark for sample efficiency.
- Cognitive Evolution without Overthinking: Linguistic analysis reveals a distinct evolutionary shift in thinking patterns: ADORA-trained models significantly increase the usage of self-reflective operators ("verify", "evaluate", "consider", "check") and contrastive discourse markers ("however", "but"), while reducing formulaic markers like "step". Under the GPT-4o Overthinking Score evaluation, ADORA achieves a comparable score to GRPO on elementary arithmetic (GSM8K: 32.2 vs. 31.5) but drops significantly on competition mathematics (AIME24: 40.1 vs. 44.8), proving that the policy learns targeted deliberation rather than vacuous verbosity.
- Strong Out-of-Domain Generalization: When trained exclusively on MATH500 with fixed hyperparameters (\(\lambda_{\text{amp}}=2.0, \tau=0.5\)), Qwen2.5-7B with ADORA achieves consistent improvements across out-of-domain benchmarks: GPQA-Diamond (+2.01%), HumanEval (+3.44%), MBPP (+2.00%), and MMLU-Pro (+2.17%), demonstrating an average OOD gain of +2.62% without task-specific retuning. Concurrently, standard multimodal perception scores (MMBench, ScienceQA, MMMU, MME) confirm that perceptual grounding is fully preserved.
Highlights & Insights¶
- In-flight Dynamic Calibration Without Auxiliary Networks: Unlike classic curriculum learning or priority sampling that demand auxiliary value heads, off-line difficulty models, or complex multi-stage schedules, ADORA extracts training utility directly from current rollout metadata (\(L^{\text{max\_succ}}, \bar{L}^{\text{fail}}, R^{\text{succ}}\)). This zero-parameter formulation introduces virtually zero computational overhead while automatically tracking non-stationary policy shifts.
- Asymmetric Reweighting Principles for Asymmetric Regimes: The paper highlights a fundamental theoretical insight regarding model capacity regimes: weak models suffer from noise contamination, necessitating attenuation of shallow shortcuts (\(\text{Len}_{\text{adv}}\)), whereas strong models suffer from exploration saturation, necessitating amplification of high-utility borderline challenges (\(\text{Len}_{\text{adv}} \land \text{Diff}_{\text{adv}}\)).
- Expansion of Solvable Problem Space: Pass@K analysis demonstrates that ADORA not only accelerates convergence at small \(K\) but achieves 100% Pass@K on the AMC benchmark within \(K < 64\), demonstrating that dynamic credit assignment fundamentally expands the reachable solution space rather than merely concentrating probability mass on already-mastered trajectories.
Limitations & Future Work¶
- Reliance on Verifiable Rule-Based Rewards: The current formulation depends on discrete, binary outcome rewards derived from symbolic ground-truth answers. Extending dynamic advantage estimation to continuous, softer rewards (e.g., neural reward models or LLM-as-a-judge scoring) remains an open challenge requiring adapted TAS/TDS thresholds.
- Single-Turn Scope: ADORA is presently validated on single-turn long-CoT reasoning tasks. Its behavior in multi-turn agentic environments (such as interactive tool use, operating system navigation, or multi-agent debate) with extended temporal credit horizons warrants future investigation.
- Persistent Outlier Failures: Longitudinal rollout analysis indicates that over 600 difficult samples in Geometry3K remained unsolved across all training epochs. Formulating dynamic hibernation or annealing mechanisms to avoid wasted rollouts on unlearnable outliers presents a promising path forward.
Related Work & Insights¶
- vs Vanilla GRPO & PPO: Conventional algorithms assume static credit assignment, weighting all successful samples equally within the normalized batch. ADORA integrates seamlessly on top of them as an instance-level dynamic prior, restoring efficiency to credit allocation without altering token-level policy dynamics.
- vs Curriculum Learning (CL) & Hard Example Mining: Traditional CL mandates rigid multi-stage ordering or teacher-guided subtask selection, which are expensive and prone to mismatch with student capabilities. ADORA conducts purely on-policy, endogenous sample selection directly reflective of the model's instantaneous frontier.
- vs Explicit Length Penalties & Lead Rewards: Prior methods attempt to regulate output length via heuristic reward penalties, which often trigger reward hacking or stifle necessary exploration. ADORA maintains objective integrity by adjusting advantage weights at the sample level without distorting the raw outcome reward.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Eloquent, zero-parameter dynamic advantage calibration grounded in empirical rollout statistics.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-model and cross-modality validation across 7 LLM/VLM families, rigorous out-of-domain checks, and detailed thinking pattern analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid technical narrative, precise mathematical grounding, and excellent structure.
- Value: ⭐⭐⭐⭐⭐ Highly practical, plug-and-play RL framework demonstrating massive data-efficiency gains (2k samples matching 10k baselines without cold-start).