Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Reinforcement Learning, Verifiable Rewards (RLVR), GRPO, Contrastive Learning, Curriculum Learning
TL;DR¶
By revealing that GRPO is mathematically equivalent to a weighted positive-negative discrimination objective with likelihood-misaligned scores and score-insensitive credit assignment, ConSPO introduces length-normalized sequence log-probabilities into a group-wise InfoNCE formulation with a curriculum-scheduled margin, significantly outperforming GRPO across challenging reasoning benchmarks.
Background & Motivation¶
Reinforcement learning with verifiable rewards (RLVR) has become the dominant post-training paradigm for enhancing complex reasoning in large language models. Among RLVR algorithms, Group Relative Policy Optimization (GRPO) is widely adopted due to its critic-free architecture, which estimates relative advantages from outcome rewards across sampled rollout groups without requiring a separate value network. To address optimization instability and entropy collapse in GRPO, recent studies have largely focused on exterior modifications—such as dynamically filtering trivial groups, introducing length penalty rewards, or designing smooth importance ratio clipping mechanisms. Despite these empirical advances, prior efforts operate strictly around the standard group-relative advantage formulation, leaving the foundational optimization mechanics of the GRPO objective largely unexplored.
A rigorous theoretical reformulation of the GRPO surrogate objective under binary verifiable rewards reveals that it is mathematically equivalent to maximizing the expected score difference between verified positive and negative rollouts, scaled by the group pass rate factor. This equivalence exposes two fundamental objective-level limitations: first, likelihood-misaligned surrogate scores, in which positive and negative rollouts are separated using averages of clipped importance sampling ratios rather than sequence likelihoods that govern autoregressive generation, creating an acute training-inference mismatch; second, score-insensitive credit assignment, where all positive rollouts within a sampled group are assigned identical gradient update coefficients and all negative rollouts share another, regardless of whether a positive rollout is dangerously close to distractors or whether a negative rollout is an adversarial high-scoring mistake. While dynamic difficulty filtering can retain queries with intermediate pass rates, it fails to fix the uniform credit allocation inherent to the loss function itself.
To resolve these structural bottlenecks, this work reframes online RLVR through the lens of contrastive sequence optimization. Rather than separating rollouts via surrogate ratio statistics, policy optimization should directly evaluate rollouts on their actual generative likelihoods and allocate updates based on instant within-group separation. Core idea: reframe GRPO as an online sequence-level InfoNCE contrastive objective driven by length-normalized sequence log-probabilities, adaptively scaling gradient updates based on within-group score gaps and preserving long-term separation pressure via a curriculum-scheduled margin.
Method¶
Overall Architecture¶
ConSPO (Contrastive Sequence-level Policy Optimization) re-architects online RLVR into an adaptive, within-group contrastive learning loop. Given an input query, the policy samples a group of candidate reasoning rollouts, which are verified by ground-truth reward checkers into positive and negative subsets. Instead of calculating advantage-weighted clipped token ratios, ConSPO evaluates the length-normalized sequence log-probability of every candidate rollout, projecting them onto a unified likelihood scale that governs autoregressive inference. These likelihood scores are fed into a group-wise InfoNCE contrastive objective modulated by a curriculum-scheduled margin, dynamically penalizing poorly separated positives and heavily suppressing high-scoring negative distractors.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Query q<br/>Sample Rollout Group"] --> B["Length-Normalized Sequence Likelihood Scoring<br/>s_θ(o, q) Aligns with Autoregressive Inference"]
B --> C["Group-Wise InfoNCE Contrastive Credit Assignment<br/>Adaptive Updates via Current Separation Gaps"]
C --> D["Curriculum-Scheduled Margin Mechanism<br/>Cosine Schedule Injects Separation Pressure mt"]
D --> E["Policy Parameter Gradient Update<br/>Adaptive Suppression of Hard Negatives"]
Key Designs¶
1. Length-Normalized Sequence Likelihood Scoring: Eliminating the Training-Inference Score Mismatch
To overcome the surrogate score misalignment of GRPO—where optimizing clipped token importance ratios does not guarantee an increase in actual generation likelihood under the updated policy—ConSPO replaces ratio-based scoring with length-normalized sequence log-probabilities. For any rollout \(o = (o_1, \dots, o_{|o|})\) generated autoregressively in response to query \(q\), the sequence score is defined as:
Because both positive and negative rollouts are scored with the exact same likelihood function, optimization operates directly on the probabilities that determine token sampling at inference time. Length normalization ensures that reasoning chains of varying lengths are evaluated fairly on a per-token average log-likelihood basis, mitigating the natural length bias that penalizes detailed step-by-step reasoning trajectories.
2. Group-Wise InfoNCE Contrastive Credit Assignment: Contrast-Sensitive Adaptive Gradient Allocation
To resolve the score-insensitive credit assignment exposed in Lemma 1—where GRPO assigns a uniform coefficient \(\frac{1}{N_+} \sqrt{\hat{p}_q(1-\hat{p}_q)}\) to all positive rollouts and \(-\frac{1}{N_-} \sqrt{\hat{p}_q(1-\hat{p}_q)}\) to all negative rollouts—ConSPO adopts a group-wise InfoNCE formulation. Each positive rollout \(o_i^+ \in \mathcal{P}_q\) is contrasted against all negative distractors \(o_j^- \in \mathcal{N}_q\) generated for the same query:
As proven in Lemma 2, the derivative of the objective with respect to the positive score \(s_\theta(o_i^+, q)\) is \(\frac{1}{N_+} (1 - P_i^+)\), while the derivative with respect to negative score \(s_\theta(o_j^-, q)\) is \(-\frac{1}{N_+} \sum_{i=1}^{N_+} P_{ij}^-\). Consequently, if a positive rollout is poorly separated from distractors (\(P_i^+\) is small), the model receives an intensified pull-up signal. Crucially, the ratio of gradient magnitudes between any two negative distractors \(o_j^-\) and \(o_k^-\) satisfies:
This mathematical property guarantees that suppressive gradients are automatically concentrated on high-scoring, deceptive hard negatives, preventing the optimization stalling caused by treating all negative trajectories alike.
3. Curriculum-Scheduled Margin Mechanism: Sustaining Separation Pressure Across Training Stages
To prevent the contrastive gradients from vanishing prematurely when positive rollouts only marginally outperform negative rollouts in later training phases, ConSPO incorporates an explicit scheduled margin \(m_t\). The full objective at training step \(t\) is formulated as:
Subtracting \(m_t \ge 0\) requires positive rollouts to exceed negative distractors by a wider margin before the contrastive probability saturates. To maintain optimization stability, \(m_t\) follows a smooth cosine warmup curriculum. Given normalized training progress \(\lambda_t \in [0, 1]\) and warmup fraction \(\alpha \in (0, 1]\) (default 30%), the schedule progress is \(\rho_t = \min(\lambda_t / \alpha, 1)\), and the margin evolves as \(m_t = \frac{M}{2} (1 - \cos(\pi \rho_t))\). Training begins with \(m_t = 0\) to encourage coarse positive-negative ordering, smoothly ramped to the target margin \(M = 0.01\), preserving continuous separation pressure throughout the remainder of post-training.
Loss & Training¶
ConSPO operates on policy rollouts evaluated with binary rewards. All-positive and all-negative rollout groups are filtered out as they provide no discriminative contrast. Key hyperparameters include contrastive temperature \(\tau = 10.0\), maximum margin \(M = 0.01\), margin warmup ratio \(\alpha = 30\%\), group size \(G = 8\), rollout generation temperature 1.0, and policy learning rate \(2 \times 10^{-6}\). The method introduces zero extra trainable parameters, bypasses auxiliary value/reward models, and preserves the computational efficiency of standard GRPO.
Key Experimental Results¶
Main Results¶
Evaluated on DeepSeek-R1-Distill-Qwen-1.5B/7B and DeepSeek-R1-Distill-Llama-8B trained on DeepScaleR-Preview-Dataset across seven competitive mathematical benchmarks: AIME 2024, AIME 2025, AIME 2026, HMMT 2025, MATH500, AMC, and OlympiadBench. Scores report avg@32 for competition benchmarks (AIME, HMMT, AMC) and pass@1 on the rest.
| Model Backbone | Method | AIME24 | AIME25 | AIME26 | HMMT25 | MATH500 | AMC | O-Bench | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| R1-Distill-Qwen-1.5B | Base Model | 20.9 | 20.7 | 13.9 | 9.7 | 79.2 | 52.8 | 37.6 | 33.5 |
| R1-Distill-Qwen-1.5B | GRPO | 28.6 | 22.9 | 20.4 | 11.7 | 85.4 | 64.4 | 46.5 | 40.0 |
| R1-Distill-Qwen-1.5B | DAPO | 28.6 | 22.9 | 20.8 | 13.5 | 86.0 | 66.2 | 50.2 | 41.2 |
| R1-Distill-Qwen-1.5B | Dr. GRPO | 27.4 | 22.0 | 19.5 | 10.7 | 83.4 | 63.5 | 46.7 | 39.0 |
| R1-Distill-Qwen-1.5B | DisCO | 28.3 | 24.8 | 21.7 | 14.4 | 86.6 | 66.5 | 49.3 | 41.7 |
| R1-Distill-Qwen-1.5B | GMPO | 31.5 | 24.6 | 23.8 | 12.8 | 87.0 | 70.0 | 49.0 | 42.7 |
| R1-Distill-Qwen-1.5B | CISPO | 31.4 | 23.0 | 23.0 | 13.3 | 87.2 | 65.2 | 47.6 | 41.5 |
| R1-Distill-Qwen-1.5B | SAPO | 30.6 | 25.3 | 23.3 | 12.8 | 87.2 | 70.1 | 50.4 | 42.8 |
| R1-Distill-Qwen-1.5B | ConSPO (Ours) | 34.7 | 26.7 | 23.9 | 14.9 | 88.8 | 70.4 | 51.1 | 44.4 |
| R1-Distill-Qwen-7B | GRPO | 50.0 | 35.9 | 43.9 | 20.7 | 92.8 | 82.8 | 59.4 | 55.1 |
| R1-Distill-Qwen-7B | DisCO | 51.8 | 36.9 | 45.1 | 19.6 | 93.2 | 82.0 | 59.3 | 55.4 |
| R1-Distill-Qwen-7B | ConSPO (Ours) | 54.9 | 39.1 | 46.8 | 21.5 | 93.8 | 83.8 | 62.4 | 57.5 |
| R1-Distill-Llama-8B | GRPO | 42.8 | 25.5 | 34.2 | 18.5 | 88.6 | 78.6 | 55.7 | 49.1 |
| R1-Distill-Llama-8B | Dr. GRPO | 44.8 | 26.8 | 32.4 | 19.6 | 89.2 | 80.5 | 56.1 | 49.9 |
| R1-Distill-Llama-8B | ConSPO (Ours) | 43.0 | 29.6 | 35.5 | 22.6 | 90.4 | 81.1 | 60.7 | 51.8 |
Ablation Study¶
Ablations on DeepSeek-R1-Distill-Qwen-1.5B isolating each component's empirical contribution:
| Configuration | AIME24 | AIME25 | AIME26 | HMMT25 | MATH500 | AMC | O-Bench | Avg. | Note |
|---|---|---|---|---|---|---|---|---|---|
| ConSPO (Full) | 34.7 | 26.7 | 23.9 | 14.9 | 88.8 | 70.4 | 51.1 | 44.4 | Full contrastive sequence policy optimization |
| w/o contrastive objective | 32.0 | 24.8 | 21.1 | 13.6 | 85.8 | 69.7 | 52.1 | 42.7 | Replaced InfoNCE with linear score difference (-1.7) |
| w/o likelihood-aligned scores | 33.1 | 25.9 | 22.1 | 13.4 | 88.6 | 70.1 | 50.7 | 43.4 | Replaced log-probs with clipped ratio surrogate scores (-1.0) |
| w/o scheduled margin | 34.3 | 25.4 | 23.4 | 14.2 | 89.0 | 70.6 | 51.7 | 44.1 | Removed scheduled margin term (-0.3) |
| w/ fixed margin | 34.1 | 25.8 | 23.8 | 14.7 | 87.8 | 70.1 | 51.3 | 43.9 | Kept constant margin M throughout training (-0.5) |
Key Findings¶
- The contrastive formulation drives the largest gain: Replacing InfoNCE with linear positive-negative score difference results in the steepest performance drop (from 44.4 to 42.7), highlighting the indispensability of contrast-sensitive credit assignment.
- Likelihood alignment rectifies optimization disconnect: Using actual length-normalized sequence log-probabilities rather than clipped importance sampling surrogates prevents sub-optimal policy drift, recovering 1.0 point on average.
- Curriculum scheduling is superior to static margins: Enforcing a static margin from iteration zero harms initial convergence (43.9 vs 44.4), whereas cosine warmup lets the policy establish basic directional ordering before enforcing strict separation.
- Consistent across base models and datasets: On base model Qwen3-4B-Base, ConSPO improves average performance from 18.4 (base) and 30.3 (GRPO) to 34.0, notably raising HMMT25 from 0.9 to 8.0. On the DAPO-Math-17k dataset, ConSPO achieves 44.1 average accuracy, beating DisCO (42.2) and GRPO (39.9).
Highlights & Insights¶
- Theoretical Elegance: Exposing GRPO as an implicit positive-negative discrimination problem provides an intuitive theoretical bridge between policy gradient algorithms and contrastive learning.
- Exponential Hard-Negative Suppression: ConSPO's gradient allocates suppressive power proportional to \(\exp((s_\theta(o_j^-) - s_\theta(o_k^-))/\tau)\), establishing an innate hard-negative mining dynamic without auxiliary filtering heuristics.
- Zero-Parameter Overhead: ConSPO requires no learned value networks or memory queues, slotting directly into standard distributed RLVR post-training pipelines.
Limitations & Future Work¶
- Empirical Scale Horizon: Primary evaluations focus on 1.5B to 8B parameter models; while 32B experiments in Appendix D demonstrate solid gains (63.0 vs 60.2 over GRPO), validation on 70B+ or MoE architectures remains for future work.
- Binary Verifiable Rewards Scope: ConSPO is currently formulated for discrete binary outcome rewards (0/1); generalizing the contrastive framework to continuous outcome metrics or process-supervised partial credit rewards is an open direction.
Related Work & Insights¶
- vs GRPO / DAPO: GRPO applies uniform advantage weighting across positive rollouts in a group; ConSPO contrasts individual rollouts using InfoNCE, directing gradients toward borderline positives and penalizing high-likelihood negative trajectories.
- vs DisCO: DisCO analyzes GRPO from a discriminative lens to tackle difficulty bias but retains linear difference updates; ConSPO addresses score-insensitive credit assignment via softmax normalization and dynamic curriculum margins.
- vs DPO / SimPO: DPO and SimPO operate as offline preference optimization algorithms on static pair datasets; ConSPO transfers contrastive sequence discrimination to the online, on-policy RLVR setting with verifiable execution checks.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant theoretical reformulation revealing fundamental flaws in GRPO, with a well-motivated contrastive solution.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive experiments across 5 backbones, multiple training corpora, and 7 difficult competition benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid mathematical narrative, tight conceptual exposition, and transparent ablation analyses.
- Value: ⭐⭐⭐⭐⭐ Provides the open-source reasoning community with an immediately actionable, high-performing drop-in replacement for GRPO.