Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning

Conference: NeurIPS 2026
arXiv: 2605.11636
OpenReview: https://openreview.net/forum?id=MSi0qsfsNZ
Code: https://github.com/MiliLab/Seirenes
Area: LLM Reasoning
Keywords: large language model reasoning, adversarial self-play, contextual interference, reinforcement learning, GRPO

TL;DR

Seirênes introduces a parameter-shared adversarial self-play reinforcement learning framework where a single model co-evolves as an Adversary synthesizing plausible deceptive distractions and a Reasoner recovering the core problem logic under interference, converting contextual vulnerability into an adaptive training signal that substantially improves standalone mathematical reasoning and contextual robustness.

Background & Motivation

Reinforcement learning with verifiable rewards (RLVR) and long-horizon chain-of-thought scaling have served as the principal driving forces behind recent leaps in LLM reasoning, propelling models to achieve competitive performance on competitive Olympiad-level mathematics and programming benchmarks. Nevertheless, state-of-the-art reasoners frequently exhibit severe fragility when confronted with non-idealized real-world contexts: introducing seemingly plausible but extraneous information, tangential background narratives, structural rephrasings, or spurious correlations causes brittle reasoning trajectories to derail catastrophically. Prior empirical evaluations demonstrate that mere structural reformulations of math problems trigger accuracy drops of nearly 30%, while irrelevant passages in retrieval-augmented environments similarly degrade reasoning performance.

The fundamental root of this vulnerability lies in the fact that standard competition benchmarks are typically clean and pristine, allowing models to achieve reward spikes by memorizing superficial templates or exploiting shortcut heuristics rather than mastering robust deductive logic. Recent self-play paradigms such as R-Zero and Absolute Zero have explored autonomous curriculum synthesis, but they predominantly operate at the level of task generation or external solution verification without reinforcing the model's internal resistance against contextual perturbations. Conversely, existing hint-augmented reinforcement learning methods almost exclusively deploy cooperative scaffolding, inadvertently conditioning reasoners to blindly trust externally provided guidance and exacerbating their susceptibility to adversarial noise.

This paper tackles the challenge by inverting contextual vulnerability into an intrinsic training asset: contextual interference should serve not merely as an evaluation-time stress test, but as an adaptive, high-quality internal training signal. Drawing inspiration from Homer's Odyssey, where the alluring songs of the sirens (Seirênes) lured sailors off course, enabling the model to self-generate deceptive contextual lures dynamically uncovers and targets its own reasoning blind spots. Core idea: establish a parameter-shared adversarial self-play loop where a single model concurrently acts as an Adversary generating locally plausible but misleading hints and as a Reasoner recovering ground-truth logic under interference, utilizing an asymmetric performance-drop reward to co-evolve standalone reasoning capabilities.

Method

Overall Architecture

Seirênes formulates the self-play dynamic within a single unified model parameterization \(\theta\), instantiating two opposing behavioral distributions via role conditioning: the Reasoner \(\pi_\theta^\mathrm{R}\) and the Adversary \(\pi_\theta^\mathrm{Adv}\). For any training problem \(q\), the task formulation and verifiable ground-truth checker remain strictly untouched. Instead, Seirênes samples a paired rollout bundle across three stages—clean baseline rollouts, adversarial hint synthesis, and hint-conditioned rollouts—to isolate the exact causal degradation induced by the distraction. The generated trajectories are then dispatched to three decoupled update streams and scheduled through bounded FIFO queues to prevent mutual gradient contamination.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Math Problem q"] --> B["Paired Rollout Bundle & Causal Isolation<br/>R1 Clean / R2 Adversary / R3 Hinted"]
    B --> C["Asymmetric Degradation Reward<br/>Compute empirical gap Δ = p(q) - p(q, h)"]
    C --> D["Decoupled Streams & Bounded FIFO<br/>Filter zero-advantage groups & buffer to flush threshold"]
    D --> E["Shared-Parameter Joint Optimization<br/>Clean GRPO + Adversary REINFORCE + Robust GRPO"]
    E --> F["Output: Robust Standalone Reasoner"]

Key Designs

1. Paired Rollout Bundle and Causal Isolation: Establishing Clean Baselines and Targeted Distractions To precisely quantify the causal disruption inflicted by contextual hints, Seirênes structures sample generation for each training instance \(q\) into three coordinated stages: First, in the clean rollout stage (R1), the policy assumes the Reasoner role to draw \(G_1\) unperturbed trajectories \(\{o_i^{(1)}\}_{i=1}^{G_1}\) from \(\pi_{\theta_\mathrm{old}}^\mathrm{R}(\cdot \mid q)\) with binary verifiable rewards \(R_i^{(1)} \in \{0, 1\}\), computing the empirical clean success rate \(\hat{p}(q) = \frac{1}{G_1} \sum_{i=1}^{G_1} R(q, o_i^{(1)})\). This provides an uncorrupted estimate of task difficulty and baseline competence. Second, in the adversarial hint generation stage (R2), the same shared policy shifts to the Adversary role, sampling \(G_2\) candidate misleading hints \(\{h_k\}_{k=1}^{G_2}\) from \(\pi_{\theta_\mathrm{old}}^\mathrm{Adv}(\cdot \mid q)\). The Adversary is prompted to craft natural, mathematically fluent continuations that embed subtle logical fallacies or false shortcut assumptions. Third, in the hint-conditioned rollout stage (R3), the prompt is augmented as \(q \oplus h_k\), and the Reasoner samples \(G_3\) trajectories \(\{o_{k,j}^{(3)}\}_{j=1}^{G_3}\) per hint. These rollouts are evaluated against the identical ground-truth verifier, yielding the hinted success rate \(\hat{p}(q, h_k) = \frac{1}{G_3} \sum_{j=1}^{G_3} R(q, o_{k,j}^{(3)})\). Because the underlying task and verification environment are preserved, all observed deviations directly reflect the causal impact of the adversarial context.

2. Asymmetric Degradation Reward: Self-Calibrated Credit Allocation for Adversarial Generation A primary failure mode in self-play is training collapse when one agent disproportionately dominates: on excessively difficult problems, a baseline Reasoner fails regardless of the hint, tempting an adversary to emit trivial noise; on trivial problems, both agents yield near-zero gradient signal. Seirênes resolves this through an asymmetric empirical gap reward:

\[\hat{R}_\mathrm{Adv}(q, h_k) = \hat{\Delta}(q, h_k) = \hat{p}(q) - \hat{p}(q, h_k)\]

Because empirical success rates are bounded within \([0, 1]\), the reward strictly resides in \([-(1 - \hat{p}(q)), \hat{p}(q)]\). This bounded formulation provides natural self-throttling across regimes: when the clean problem is too hard (\(\hat{p}(q) \to 0\)), the maximum obtainable reward is suppressed to zero, penalizing spurious attacks; when the clean problem is mastered and resists distraction (\(\hat{p}(q) \to 1\) and \(\hat{p}(q, h_k) \to 1\)), the gradient update attenuates. As a result, positive reinforcement is concentrated precisely on the sweet spot where the model solves the clean problem yet stumbles under deception. Averaging over \(G_1\) and \(G_3\) trajectories provides built-in variance reduction, enabling stable REINFORCE-style optimization of the Adversary tokens:

\[\mathcal{L}_\mathrm{Adv} = -\frac{1}{G_2} \sum_{k=1}^{G_2} \operatorname{sg}\left[\hat{R}_\mathrm{Adv}(q, h_k)\right] \frac{1}{|h_k|} \sum_{t=1}^{|h_k|} \log \pi_\theta^\mathrm{Adv}(h_{k,t} \mid q, h_{k,<t})\]

3. Decoupled Update Streams and Bounded FIFO Scheduling: Mitigating Gradient Interference and Rollout Overhead Naive batch pooling across divergent roles with disparate credit semantics induces gradient conflict. Seirênes isolates clean reasoning, adversary generation, and robustness recovery into three autonomous update streams. Each stream applies zero-advantage filtering (\(A_s(g) \neq 0\)) to discard uninformative rollouts, routing valid samples into role-specific Bounded FIFO buffers. Once a buffer accumulates its threshold flush size \(M_s\) (\(M_c\) clean prompt groups, \(M_\mathrm{adv}\) adversarial hints, \(M_r\) robust groups), an optimization step is executed and the queue is advanced, retaining surplus valid trajectories to eliminate online sample waste.

To counteract the inference overhead of multi-round generation, Seirênes implements latency-aware scheduling. Since both R1 and R2 condition solely on the initial query \(q\), R2 hint generation is submitted concurrently with R1. Given that variable-length reasoning traces create hardware compute bubbles in continuous batching engines, the compact R2 hint sequences naturally slot into idle execution windows, collapsing the effective latency \(T_{12} \approx T_{R_1}\). Concurrently, R1 outcome verification runs asynchronously during R3 generation. Furthermore, Seirênes tracks a mastered set \(\mathcal{M}\) to deprioritize instances consistently solved under distraction, preserving GPU budgets for active frontier reasoning challenges.

Loss & Training

The overall training architecture is jointly governed by three objective functions: 1. Clean Reasoning Loss (\(\mathcal{L}_\mathrm{clean}\)): Standard GRPO applied to R1 trajectories, maintaining the core unperturbed mathematical reasoning capability; 2. Adversary Loss (\(\mathcal{L}_\mathrm{Adv}\)): Token-level policy gradient guided by the empirical degradation gap reward; 3. Robustness Recovery Loss (\(\mathcal{L}_\mathrm{robust}\)): GRPO applied over the perturbed \(q \oplus h_k\) trajectories:

\[\mathcal{L}_\mathrm{robust} = \frac{1}{|\mathcal{Q}_\mathrm{rob}|} \sum_{q \in \mathcal{Q}_\mathrm{rob}} \frac{1}{|\mathcal{H}_q|} \sum_{h_k \in \mathcal{H}_q} \left[-J_\mathrm{GRPO}\left(\theta; \{(q \oplus h_k, o_{k,j}^{(3)}, R_{k,j}^{(3)})\}_{j=1}^{G_3}\right)\right]\]

Optimization utilizes AdamW with a constant learning rate of \(1 \times 10^{-6}\), zero KL penalty (\(\beta = 0\)), and asymmetric clipping thresholds (\(\varepsilon_\mathrm{low} = 0.2, \varepsilon_\mathrm{high} = 0.28\)). Trajectories lingering beyond 3 optimizer steps are evicted to constrain off-policy drift.

Key Experimental Results

Main Results

Seirênes was evaluated in clean, hint-free environments across seven challenging mathematical reasoning benchmarks (AIME 2024–2026, IMO-Bench, MINERVA Math, OlympiadBench, and HMMT 2026) across three backbone families (Qwen2.5-7B, Qwen3-4B, and Qwen3-30B-A3B MoE). All competing baselines were benchmarked under equivalent training budgets.

Model	AIME24	AIME25	AIME26	IMO-Bench	MINERVA	Olympiad	HMMT26	AVG	\(\Delta\) AVG
Qwen2.5-7B-Instruct	10.2	5.5	5.3	8.8	29.4	34.9	2.7	13.8	–
+ ExGRPO	14.2	13.2	9.0	9.9	29.9	42.0	6.0	17.7	+3.9
+ LUFFY	13.9	12.9	12.3	11.0	32.8	41.5	6.3	18.7	+4.9
+ SAGE	15.8	13.7	9.3	11.2	31.6	42.2	6.3	18.6	+4.8
+ DAPO	14.1	12.0	9.3	9.5	30.5	42.1	4.3	17.4	+3.6
+ Seirênes (Ours)	22.8	15.4	17.8	13.3	34.7	48.0	8.5	22.9	+9.1
Qwen3-4B-Instruct	61.5	48.8	56.8	34.1	35.4	66.3	31.9	47.8	–
+ Scaf-GRPO	61.6	49.1	54.3	33.9	38.6	67.1	32.2	48.1	+0.3
+ SAGE	60.1	49.1	57.8	32.5	41.1	67.2	29.9	48.2	+0.4
+ InT	66.6	58.9	64.7	38.8	38.3	67.8	37.9	53.3	+5.5
+ Dr.GRPO	65.0	63.3	64.9	38.0	39.4	68.3	37.2	53.7	+5.9
+ DAPO	63.0	60.2	62.2	37.7	40.1	68.3	33.0	52.1	+4.3
+ Seirênes (Ours)	74.3	64.9	70.8	41.9	42.8	71.9	39.4	58.0	+10.2
Qwen3-30B-A3B	75.0	60.6	75.1	40.5	35.0	70.0	41.0	56.7	–
+ DAPO	78.6	66.8	75.2	44.3	40.6	72.8	42.3	60.1	+3.4
+ Seirênes (Ours)	81.8	74.1	79.7	46.7	41.7	74.3	49.0	63.9	+7.2

Ablation Study

To verify whether Seirênes generalizes beyond the training hint distribution, models were evaluated under out-of-distribution structural math perturbations (MATH-Perturb) and generated irrelevant distractors (GSMIR, MMLU-Perturb, OpenBookQA-Perturb). In the table below, Perturb Accuracy (↑) reflects absolute performance under perturbation, while Robustness Degradation (↓) measures performance drop relative to clean accuracy (lower is better).

Model	Math-P-Hard (↑)	Math-P-Simple (↑)	Perturb Avg (↑)	GSMIR Deg (↓)	MMLU-P Deg (↓)	OBook-P Deg (↓)	Avg Deg (↓)
Qwen3-4B-Instruct	73.9	93.9	83.9	2.96	9.39	19.03	10.46
+ Scaf-GRPO	71.7	93.9	82.8	3.57	9.12	19.03	10.57
+ SAGE	76.3	95.7	86.0	4.34	7.54	19.62	10.50
+ InT	76.1	94.8	85.4	7.39	8.25	17.92	11.19
+ Seirênes (Ours)	79.1	97.4	88.2	3.05	4.91	16.21	8.06

In a wall-clock controlled comparison under a 40-hour ceiling, Seirênes converged faster and achieved higher terminal accuracy than compute-matched DAPO (\(G_1=24\)). Furthermore, offline static hints generated by Gemini 3 Flash suffered rapid effectiveness saturation (attack slope decaying to negative), whereas Seirênes's evolving Adversary maintained sustained attack pressure (positive slope \(+0.057\%\)/step), sustaining up to 11 consecutive strong-attack steps.

Key Findings

• Adversarial co-evolution drives universal reasoning gains: Seirênes yields strong positive improvements across all models and benchmarks, securing up to a +10.2 average gain on Qwen3-4B. Substantial improvements on newly released benchmarks (AIME 2026, HMMT 2026) confirm that the benefits stem from genuine reasoning enhancement rather than dataset contamination or memorization.
• Cooperative scaffolding induces dangerous credulity: Baselines trained on helpful hints (SAGE, InT, Scaf-GRPO) exhibited higher average performance degradation (10.50%–11.19%) than unaugmented base models (10.46%), indicating an acquired over-reliance on external cues. In contrast, Seirênes reduced degradation to 8.06%, fostering a rigorous tendency to scrutinize and reject misleading premises.
• Learned adversarial traps transfer across model families: The 4B Adversary learned a "Trojan horse" strategy of pairing mathematically sound preliminary derivations with subtle false assertions. When tested zero-shot against frontier proprietary models, these hints degraded GPT-5.1 accuracy by 3.9 points and Gemini 3 Flash by 5.2 points (inducing sharp 10.0 and 12.9 point drops on AIME26 and IMO-Bench), proving that the Adversary discovers transferable reasoning vulnerabilities.

Highlights & Insights

• Inverting contextual brittleness into training fuel: Rather than treating distraction vulnerability as an evaluation failure, Seirênes operationalizes it as an internal, self-adaptive training curriculum, actively converting reasoning weaknesses into developmental signals.
• Asymmetric degradation reward prevents self-play collapse: By bounding credit within \([-(1 - \hat{p}(q)), \hat{p}(q)]\), the reward naturally suppresses rewards on overly trivial or impossibly hard questions, concentrating learning gradients purely on the actionable capability frontier.
• Exploiting compute bubbles for zero-cost adversarial rollout: Through latency-aware scheduling, short adversarial hint rollouts are executed within the compute bubbles created by variable-length reasoning trajectories, enabling complex three-stage self-play with negligible wall-clock overhead.

Limitations & Future Work

• Higher inference rollout overhead than vanilla RL: Although latency scheduling and mastery pruning mitigate execution costs, maintaining three rollout stages per query incurs higher per-step FLOPs than single-stream RLVR, warranting exploration into memory-based reuse of high-value adversarial contexts.
• Dual-use considerations of transferable traps: The learned Adversary synthesizes deceptive mathematical continuations capable of degrading top-tier proprietary models, underscoring the necessity of establishing safeguards against malicious distraction deployment.
• Constrained capacity under parameter sharing: Enforcing strict parameter sharing limits the Adversary to the same model capacity as the Reasoner; decoupled asymmetric architectures warrant future investigation.

Related Work & Insights

• vs R-Zero / Absolute Zero: Prior self-play reasoning works construct games around task generation or automated program verifiers; Seirênes keeps the underlying task and verification environment strictly constant, shifting self-play into the contextual perturbation space to target reasoning execution robustness directly.
• vs SAGE / Scaf-GRPO / InT: Cooperative hint methods rely on helpful scaffolding that risks inducing cue dependency; Seirênes demonstrates that adversarial deceptive hints offer an equally potent yet orthogonal training signal that inoculates models against context distraction.
• vs Classical Adversarial RL (RARL / PAIRED): Unlike classical minimax RL approaches operating on norm-bounded continuous state disturbances across dual networks, Seirênes operates over discrete natural language reasoning within a single parameter-shared model.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of natural language adversarial hint synthesis into a parameter-shared self-play RLVR framework.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning 7 Olympiad-level benchmarks, 3 architectures, compute-controlled ablations, perturbation stress tests, and black-box transfer to closed-source LLMs.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous conceptual formulation, clear mathematical exposition, and thoughtful engineering pipeline design.
• Value: ⭐⭐⭐⭐⭐ Provides an effective, self-contained paradigm for training robust, shortcut-resistant reasoning models without reliance on external teacher data.

Related Papers

• [NeurIPS 2026] Learning Robust Reasoning through Guided Adversarial Self-Play
• [NeurIPS 2026] TTCS: Test-Time Curriculum Synthesis for Self-Evolving
• [NeurIPS 2026] From Noise to Diversity: Random Embedding Injection in LLM Reasoning
• [NeurIPS 2026] What are Key Factors for Updates in RL for LLM Reasoning?
• [NeurIPS 2026] CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning