Skip to content

Multi-Agent Evolve: LLM Self-Improve through Co-evolution

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/chenyixing/Multi-Agent-Evolve
Area: LLM Reasoning
Keywords: self-play, reinforcement learning, multi-agent co-evolution, verifiable reward alternative, curriculum learning

TL;DR

Multi-Agent Evolve (MAE) breaks the fundamental dependence of LLM reinforcement learning on human-labeled data and external deterministic verifiers (such as code interpreters or math engines) by instantiating an asymmetric Proposer-Solver-Judge triad from a single parameter-shared backbone, delivering a 4.86% average accuracy gain across 22 benchmarks on Qwen2.5-3B-Instruct.

Background & Motivation

Reinforcement Learning (RL) has emerged as an indispensable cornerstone for advancing complex reasoning and extended multi-step deduction in Large Language Models (LLMs), with Reinforcement Learning with Verifiable Rewards (RLVR) achieving remarkable milestones in mathematics and program synthesis. However, existing triumphs are heavily anchored to either vast volumes of human-curated demonstrations or specialized, hermetically sealed sandboxes offering programmatically verifiable rewards (e.g., Python compilers, formal proof assistants, or game simulation environments). In open-ended domains that constitute the bedrock of human intelligence—such as multi-hop commonsense reasoning, creative writing, nuanced factual question answering, and high-level strategic planning—deterministic reward functions and ground-truth verifiers are practically nonexistent. Furthermore, relying on human annotations imposes severe scalability and cost bottlenecks, inevitably capping model performance at human levels and impeding trajectories toward superhuman artificial general intelligence.

While the self-play paradigm has historically achieved superhuman mastery in classical symmetric zero-sum games such as Go, Chess, and StarCraft by continuously discovering dominant strategies, naively adapting it to natural language reasoning triggers severe pathologies. In standard two-player zero-sum formulation between a task proposer and a problem solver, optimizing solely to minimize the solver's success rate causes the proposer to rapidly degenerate into adversarial gaming. The proposer discovers that emitting ungrammatical gibberish, ambiguous premises, or fundamentally unsolvable logical contradictions is the most efficient path to driving the solver's reward to zero. This parasitic collapse pollutes the model's linguistic representations, destroys core generation abilities, and stalls the evolution cycle; conversely, relying on a frozen external LLM evaluator reduces the pipeline to static semi-supervised distillation rather than true autonomous evolution.

To resolve this impasse, this work proposes to extend self-play into a cooperative-competitive general-sum formulation by explicitly introducing an intrinsic evaluative dimension into the interaction loop. The Proposer and Solver must engage in fierce adversarial competition over challenge boundaries while simultaneously maintaining cooperative alignment with an intrinsic Judge on task validity, semantic clarity, and factual solvability. By instantiating all three functional roles from a single unified backbone with shared parameters, cognitive enhancements achieved by solving problems directly transfer to grading accuracy, creating an emergent, self-illuminating optimization loop without external anchors. Core idea: instantiate an interacting triad of Proposer, Solver, and Judge roles from a single shared-parameter LLM, where the Judge's ungrounded Chain-of-Thought critique serves as the intrinsic reward baseline and the Solver's empirical solve rate measures task difficulty, driving the Proposer to dynamically explore the model's reasoning frontier under strict quality filtering via Task-Relative REINFORCE++.

Method

Overall Architecture

The Multi-Agent Evolve (MAE) framework orchestrates an iterative, multi-role closed loop parameterized by a single unified language model backbone \(\theta\). Role-specific system instructions project this backbone into three distinct functional policies: the Proposer (\(\pi_P\)) for adaptive generative curriculum design, the Solver (\(\pi_S\)) for generating reasoning trajectories for deployment, and the Judge (\(\pi_J\)) for intrinsic reward modeling. Each training iteration executes four synchronized phases: First, during the Proposer Phase, candidate questions are generated either from scratch or through controlled perturbations of unannotated reference tasks; a rigorous front-stage Quality Filter powered by the Judge immediately discards ill-formed or unsolvable queries before admission into the active dataset \(\mathcal{D}\). Second, during the Solver Phase, the Solver samples questions from \(\mathcal{D}\) and produces solution trajectories, which are evaluated by the Judge via step-by-step Chain-of-Thought (CoT) reasoning. Third, during the Judge Phase, the Judge itself is optimized with structural parsing constraints. Finally, during the Synchronized Update, advantages are computed independently per role via Task-Relative REINFORCE++ and backpropagated into the shared parameters \(\theta\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: 16 self-generated seeds or unlabeled prompts"] --> B["Triad Role Asymmetric Co-evolution Architecture<br/>Parameter-shared backbone instantiating Proposer/Solver/Judge"]
    B --> C["Dual-Objective Reward and Desirable Difficulty Formulation<br/>Quality RQ and difficulty RD balancing capability boundaries"]
    C --> D["Task-Relative Policy Optimization and Front-stage Quality Guard<br/>Role-normalized REINFORCE++ with strict admission filtering"]
    D --> E["Output: Self-improved general reasoning and knowledge policy"]

Key Designs

1. Triad Role Asymmetric Co-evolution Architecture: Decoupling generation, execution, and critique to eliminate zero-sum gaming collapse

Directly pitting a question generator against a solver in ungrounded domains inevitably provokes adversarial reward hacking. MAE overcomes this by instantiating three specialized roles from a single shared LLM backbone \(\theta\) using tailored prompts. The Proposer focuses on generating self-contained, deterministic, and intellectually demanding tasks; the Solver conducts chain-of-thought problem solving within <answer> tags; and the Judge executes rigorous logical, factual, and redundancy audits within <think> tags before emitting a discrete score in \([1, 10]\) within <score> tags. Because all three agents share the exact same parameter weights, any reasoning breakthroughs achieved by the Solver directly enrich the Judge's deductive capacity, enabling it to detect subtler errors; conversely, an increasingly discerning Judge provides more refined reward gradients for both Solver answers and Proposer tasks. Empirical gradient cosine similarity analysis verifies that role gradients maintain positive alignment (mean cosine similarity ranging from 0.81 to 0.88 throughout training), demonstrating that shared parameterization prevents destructive interference while cutting memory and checkpoint storage costs by two-thirds.

2. Dual-Objective Reward and Desirable Difficulty Formulation: Aligning task feasibility and challenge to automate curriculum learning

To prevent the Proposer from generating trivial repetitions or unsolvable noise, MAE constructs a composite reward function balancing quality, difficulty, and format: \(R_P(q) = \frac{1}{3}R_Q(q) + \frac{1}{3}R_D(q) + \frac{1}{3}R_F(q)\). The quality reward \(R_Q(q) = V_J(q) \in [0, 1]\) is directly supplied by the Judge's evaluation of the question's intrinsic solvability and clarity. The difficulty reward \(R_D(q)\) is computed by sampling \(N_{\text{sample}} = 5\) response rollouts from the current Solver policy \(\pi_S\) and taking the complement of their average solve rate: $\(R_D(q) = 1 - \bar{R}_S(q) = 1 - \frac{1}{N_{\text{sample}}} \sum_{i=1}^{N_{\text{sample}}} V_J(a_i, q)\)$ When a question is easily solved by the Solver, \(\bar{R}_S(q) \approx 1\) and the Proposer receives negligible difficulty reward; the Proposer earns maximal reward only when proposing tasks that lie directly on the outer frontier of the Solver's current zone of proximal development. This mathematical incentive dynamically operationalizes the psychological principle of "Desirable Difficulty": as the Solver improves, the Proposer is forced to generate deeper multi-hop dependencies and tighter constraints, establishing a fully automated, perpetually ascending learning curriculum.

3. Task-Relative Policy Optimization and Front-stage Quality Guard: Normalizing multi-role variance and intercepting data corruption

The heterogeneous action spaces, response lengths, and reward distributions of the three distinct roles introduce substantial variance during joint reinforcement learning. MAE adapts Task-Relative REINFORCE++ by calculating baseline means \(\mu_{\text{role}}\) and standard deviations \(\sigma_{\text{role}}\) independently for each role, deriving normalized advantages: $\(A_{\text{norm}}^{\text{role}} = \frac{r - \mu_{\text{role}}}{\sigma_{\text{role}}}\)$ These advantages are incorporated into a synchronized PPO clipped surrogate objective across all roles: \(\hat{g}_t = \sum_{r \in \{P, S, J\}} \alpha_r \nabla_\theta \hat{\mathcal{L}}_t^{(r)}(\theta)\). Furthermore, to safeguard the co-evolutionary loop against data poisoning caused by Proposer entropy decay, MAE imposes a strict front-stage Quality Filter: any candidate question whose Judge quality score falls below a threshold is permanently discarded rather than admitted to the dataset \(\mathcal{D}\). Ablation experiments confirm that removing this filter causes invalid, hacked questions to accumulate in the replay buffer, triggering a disastrous 3.72% performance plunge (dropping overall score from 59.87% to 56.15%), proving that the front-stage quality guard is essential for self-evolutionary stability.

Loss & Training

MAE optimizes a joint clipped surrogate loss over parameter weights \(\theta\). At each training step \(t\), for each role \(r \in \{P, S, J\}\) with sampled batch \(\mathcal{B}_t^{(r)}\), the empirical objective is formulated as: $\(\hat{\mathcal{L}}_t^{(r)}(\theta) = \frac{1}{|\mathcal{B}_t^{(r)}|} \sum_{i \in \mathcal{B}_t^{(r)}} \min \left( \rho_{t, i}(\theta) A_{\text{norm}, t, i}^{(r)}, \; \text{clip}(\rho_{t, i}(\theta), 1-\epsilon, 1+\epsilon) A_{\text{norm}, t, i}^{(r)} \right)\)$ where \(\rho_{t, i}(\theta) = \frac{\pi_\theta(a_{t, i}|s_{t, i})}{\pi_{\text{old}}(a_{t, i}|s_{t, i})}\) represents the importance sampling ratio and clipping threshold \(\epsilon = 0.2\). The shared backbone is updated via the weighted composite surrogate gradient \(\hat{g}_t = \nabla_\theta \sum_{r \in \{P, S, J\}} \alpha_r \hat{\mathcal{L}}_t^{(r)}(\theta)\). Training is configured with AdamW optimizer, learning rate \(1 \times 10^{-6}\), gradient clipping norm 1.0, total training steps 300, batch size 64, maximum sequence length 8192, and actor rollout temperature/top-p of 1.0. Solver weights are set to \(\lambda_J = 0.5, \lambda_F = 0.5\); the Judge is explicitly trained with format parsing reward \(R_J(g) = R_F(g)\) (\(\lambda_F = 1.0\)), with its evaluative accuracy co-evolving implicitly via parameter sharing.

Key Experimental Results

Main Results

Experiments are evaluated on Qwen2.5-3B-Instruct across 22 benchmarks spanning mathematics, coding, factual knowledge, and complex reasoning. Under the Zero-supervision setting starting from only 16 self-generated seed prompts, MAE (zero) substantially outperforms the grounded baseline Absolute Zero Reasoner (AZR). Under the unannotated seed setting (967 unlabeled reference questions), MAE (no reference) achieves the highest overall accuracy of 60.19%, outperforming base model by 4.86% and surpassing full Supervised Fine-Tuning (SFT) trained with human ground-truth labels.

Method GSM8K MATH MMLU GPQA TriviaQA BoolQ HellaSwag MBPP+ HumanEval+ BBH MMLU-Pro Overall Avg.
Base (Qwen2.5-3B) 85.20 60.40 63.40 34.67 51.60 73.60 67.80 60.60 68.30 53.79 44.68 55.33
AZR (Zhao et al., 2026) 81.20 62.40 63.40 38.93 56.67 78.50 67.56 61.40 68.30 52.57 44.09 57.72
MAE (zero) 86.00 68.20 61.40 29.42 60.13 78.46 70.34 62.20 69.50 57.51 42.66 58.67
SFT (ground truth labels) 82.40 63.20 63.80 28.97 57.31 79.70 67.23 67.70 74.40 55.58 43.24 57.92
MAE (with reference) 76.00 49.00 62.40 32.60 57.13 83.45 69.62 61.00 60.30 57.17 44.35 57.11
MAE (no reference) 84.40 68.20 67.40 34.23 63.68 77.50 71.54 61.40 68.30 57.53 46.32 60.19
MAE (half reference) 82.20 65.80 69.00 30.80 61.00 78.00 79.00 61.10 68.30 53.14 47.43 59.87

Categorical breakdown highlights that MAE (no reference) achieves massive improvements in Knowledge (+7.00 points, 52.46 \(\to\) 59.46) and Reasoning (+6.48 points, 60.16 \(\to\) 66.64). Even in verifiable domains like mathematics, MAE (zero) achieves a notable gain (+3.51 points, 49.52 \(\to\) 53.03) without using any external verifier.

Ablation Study

Ablations on Qwen2.5-3B-Instruct under the MAE (half reference) setup dissect individual agent roles, parameter sharing schemes, and safeguard mechanisms (full model overall average is 59.87%):

Configuration Variant Math Avg. Code Avg. Knowledge Avg. Reasoning Avg. Overall Avg. Performance Drop
MAE Full (Shared Backbone) 50.85 64.70 58.32 66.69 59.87 -
w/o Solver Training 48.97 62.65 56.48 63.42 57.88 -1.99
w/o Proposer Training 51.19 64.75 55.45 60.21 57.90 -1.97
w/o Judge Training 49.35 64.85 54.38 60.33 57.24 -2.63
w/o Question Quality Filtering 45.20 64.85 53.48 61.07 56.15 -3.72
w/o Format Reward 51.98 64.85 57.88 63.05 59.44 -0.43
Separate Role Parameters 55.37 65.45 57.62 62.17 59.27 -0.60

Scaling evaluation on Qwen2.5-7B-Instruct demonstrates robust generalization: the base 7B model improves from 62.63% to 63.59% overall under MAE (half reference), with General QA rising from 58.37% to 59.87% and Math advancing from 55.43% to 58.13%.

Key Findings

  • Quality filtering is the fundamental stabilizer of ungrounded self-play: Eliminating front-stage question quality filtering causes the most severe drop across all ablations (-3.72%). Without the Judge's gatekeeping on question feasibility and logical soundness, the Proposer hacks the system with unsolvable or self-contradictory prompts to artificially harvest high difficulty rewards, fatally corrupting the training corpus.
  • Judge co-training is essential for continual guidance: Freezing the Judge role leads to a 2.63% drop, representing the largest drop among all single-role ablations. A static Judge fails to keep pace with the increasingly sophisticated deductions of the evolving Solver, deteriorating reward signal quality.
  • Shared parameterization outperforms separate role models: A shared backbone achieves higher overall accuracy (59.87%) than maintaining three disjoint networks (59.27%), while drastically reducing GPU memory and communication overhead. Improvements acquired in judging or solving readily cross-pollinate across roles via shared internal representations.
  • Unconstrained generation fosters superior exploration: Constantly conditioning the Proposer on reference questions (MAE with reference: 57.11%) traps exploration within narrow neighborhoods, whereas allowing the model to generate prompts from scratch (MAE no reference: 60.19%) unlocks more diverse curricula and superior generalization on complex reasoning benchmarks.

Highlights & Insights

  • Verifiable sandbox independence for open-domain self-play: This paper demonstrates that LLMs can achieve continuous, verifiable-free self-improvement across open-domain commonsense, factual understanding, and abstract reasoning purely through internally coordinated multi-agent game dynamics.
  • Cognitively inspired desirable difficulty dynamics: The empirical correlation between escalating difficulty rewards and subsequent benchmark accuracy leaps validates the Desirable Difficulty hypothesis, showing that self-play RL thrives when challenges continually press against the boundary of failure without tipping into unsolvability.
  • Economical unified triad architecture: Rather than deploying cumbersome multi-model agent ensembles, MAE activates heterogeneous specialized roles from a single shared network via prompt modulation, providing an elegant and compute-efficient recipe for scalable self-evolution.

Limitations & Future Work

  • Vulnerability to Proposer entropy decay: Training trajectories beyond 150 steps exhibit progressive entropy decay in the Proposer policy, wherein question generation begins converging to narrower semantic clusters. Incorporating explicit entropy regularization or semantic dispersion bonuses represents an urgent next step.
  • Restriction to single-turn question-answering interactions: The current triad interaction is strictly constrained to single-turn query-solve-judge exchanges; extending the co-evolutionary paradigm to multi-turn debates, dynamic tool manipulation, or long-horizon agentic planning environments remains unaddressed.
  • Scale constraints under resource limits: Experiments were validated up to 7B parameters. Exploring whether emergent self-correction and metacognitive capabilities in frontier-scale models (e.g., 70B+) amplify or alter these co-evolutionary dynamics remains a promising open frontier.
  • vs Absolute Zero Reasoner (AZR, Zhao et al., 2026): AZR pioneers data-free self-play for LLM reasoning but strictly relies on verifiable environments (Python interpreters and math sandboxes); MAE incorporates an intrinsic CoT Judge to successfully generalize self-play into unverifiable open domains.
  • vs Self-Rewarding LMs (Yuan et al., 2025) & LLM-as-a-Judge (Gu et al., 2026): Conventional self-rewarding frameworks typically freeze the evaluator or execute uncoupled multi-stage iterative SFT, leading to reward hacking and scoring drift; MAE tightly integrates the Judge into a unified, synchronized Task-Relative RL optimization loop alongside the Proposer and Solver.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First framework to achieve closed-loop co-evolutionary self-play in unverifiable open domains without external rewards.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 22 benchmarks, 4 data regimes, comprehensive role ablations, and gradient alignment dynamics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Seamless conceptual progression, crystal-clear formulation of asymmetric game theory, and thorough empirical transparency.
  • Value: ⭐⭐⭐⭐⭐ Offers a highly practical and scalable blueprint for overcoming the human annotation bottleneck on the path to self-improving AGI.