Hybrid Open-Ended Tri-Evolution Makes Better Deep Researcher¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/IQuestLab/ote
Area: LLM Reasoning
Keywords: Deep Research Agents, Open-Ended Agent Evolution, Tri-Evolution Self-Play, Hybrid-Mode Reinforcement Learning, Dynamic Rubric Evaluation
TL;DR¶
Addressing the limitations of static data and lack of verifiable ground truths in long-form deep research, this paper proposes Hybrid Open-Ended Tri-Evolution (HOTE), a tri-evolutionary framework co-evolving a proposer, solver, and judge via dual-mode reinforcement learning, enabling an 8B model to surpass strong 8-32B baselines with significantly less training overhead.
Background & Motivation¶
Deep research, characterized by an agent's capability to autonomously plan, execute multi-turn web-scale retrieval, cross-verify conflicting evidence, and synthesize comprehensive, cited reports over extended horizons, has emerged as a landmark milestone on the pathway toward artificial general intelligence. While pioneering proprietary systems like OpenAI Deep Research and Claude Research alongside recent open-source systems such as DR Tulu and WebThinker demonstrate remarkable research proficiency, their parametric research capabilities remain fundamentally upper-bounded by fixed training datasets and pre-curated trajectories. Consequently, these static systems cannot autonomously broaden their investigative horizons or overcome latent weaknesses when deployed in dynamic, open-ended environments.
To transcend the bounds of static supervised datasets, self-play reinforcement learning paradigms—such as R-Zero, Absolute Zero, Dr. Zero, and SPICE—have emerged as promising solutions by enabling models to act simultaneously as task proposers and solvers that acquire learning experiences through mutual competition. Nevertheless, existing self-evolving frameworks are overwhelmingly restricted to verifiable domains with deterministic scalar or exact-match ground truths, such as formal mathematics, code generation, and short-form factoid question answering. In contrast, real-world deep research tasks necessitate producing nuanced, multi-page reports with extensive literature citations where verifiable answers are virtually non-existent, rendering standard reward verifiers ineffective and leaving static judge models highly susceptible to reward hacking by verbose or ungrounded solvers.
To bridge this fundamental gap between open-ended deep research and autonomous agent self-evolution, this work conceives an interactive tri-evolutionary ecosystem comprising a proposer, a solver, and a dynamic judge. The judge not only evaluates report quality but also adaptively discovers evaluation rubrics and summarizes systemic weaknesses into meta-rubrics; the proposer leverages these meta-rubrics to formulate challenging yet learnable research tasks that pinpoint the solver's current vulnerabilities; and the solver continuously evolves by overcoming these targeted queries under a hybrid RL regime. Core idea: establish an interactive tri-evolution framework (HOTE) where a proposer, solver, and dynamic judge co-evolve via meta-rubric guidance and dual-mode (no-tool and tool-use) reinforcement learning, achieving sustainable, open-ended advancement in non-verifiable deep research tasks.
Method¶
Overall Architecture¶
HOTE is built upon the general ReAct agent paradigm with a unified action space defined as \(\mathcal{A} = \{\text{think}, \text{tool}, \text{answer}, \text{cite}\}\). Within this formulation, the framework coordinates three co-evolving modules alongside a dual-mode hybrid training strategy: the solver (\(\pi_{\theta_s}\)) receives research tasks and produces comprehensive, cited reports; the judge (\(\pi_{\theta_j}\)) inspects groups of sampled solver outputs, dynamically updates active rubrics, scores responses, and synthesizes meta-rubrics capturing shared failure modes; the proposer (\(\pi_{\theta_p}\)) ingests these meta-rubrics alongside task combinations to formulate novel, learnable synthetic queries; and a hybrid execution curriculum balances pure internal reasoning (no-tool mode) with external web exploration (tool-use mode) to decouple reasoning discipline from tool noise while optimizing sample throughput.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
subgraph S1["Hybrid Mode Solving and Generation"]
direction TB
A["Input Hybrid Task Batch<br/>50% Real Tasks + 50% Synthetic Tasks"] --> B["Dual-Mode Execution<br/>50% No-Tool Pure Reasoning + 50% Tool-Use Search & Citation"]
B --> C["Sample Solver Candidate Responses<br/>{o_i} (G=8 Sampled Reports)"]
end
C --> D["Dynamic Rubric Comparison and Meta-Rubric Extraction<br/>Update Active Rubrics + Score Rewards + Extract Meta-Rubrics"]
subgraph S2["Dual-Track Adaptive Evolution"]
direction TB
D --> E["Solver Policy Optimization<br/>Token-Level GRPO with Normalized Advantage Updates π_θs"]
D --> F["Diverse Targeted Proposing<br/>Synthesize Challenging Queries from Meta-Rubrics & Task Subsets"]
end
F --> G["Proposer Policy Optimization<br/>GRPO Update Maximizing Challenge Subject to Solvability π_θp"]
G --> H["Deliver High-Quality Reports & Evolved System<br/>Persist Meta-Rubrics & Queries to Next Training Iteration"]
E --> H
Key Designs¶
1. Solver Policy Evolution: Rubric-Weighted Rewards via Group Relative Policy Optimization
To eliminate reliance on deterministic truth labels in long-form generation, HOTE guides the solver's policy \(\pi_{\theta_s}\) through a dynamic rubric set \(\mathcal{R}_{s_0}\) maintained by the judge. The judge assigns each rubric \(R \in \mathcal{R}_{s_0}\) an importance weight \(w\) and a discrete evaluation score of \(0\) or \(\pm 1\), producing a normalized composite scalar reward \(r_i\) for each generated report \(o_i\). To stably optimize policies over lengthy trajectories extending up to 16,384 tokens, the solver employs Group Relative Policy Optimization (GRPO) with token-level loss aggregation across \(G=8\) sampled trajectories per task, computing normalized relative advantages:
This design completely decouples policy improvement from static gold references, using intra-group competitive dynamics to incentivize more rigorous literature synthesis, comprehensive structural coverage, and precise factual citations.
2. Dynamic Judge Evolution: Bidirectional Rubric Updating and Meta-Rubric Synthesis
Static assessment rubrics inevitably invite reward hacking, where the solver learns superficial stylistic shortcuts, circular references, or fabricated citations to inflate scores. HOTE models the rubric set as the union of persistent rubrics \(\mathcal{R}_{\text{persi.}}^{s_0}\) and active rubrics \(\mathcal{R}_{\text{active}}^{s_0}\). At each step, the judge inspects the variation across solver responses \(\{o_i\}_{i=1}^G\) and dynamically updates \(\mathcal{R}_{\text{active}}^{s_0}\): positive rubrics are generated to reward newly surfaced, legitimate knowledge domains, while negative rubrics penalize newly observed undesirable patterns such as mathematical hallucinations or calculation-result contradictions. To circumvent the context explosion that would occur if all raw evaluation trajectories were passed downstream, the judge synthesizes evaluations into compact meta-rubrics via \(\text{Meta}_{\pi_{\theta_j}}(A, \mathcal{R}_{s_0}, \{o_i\}_{i=1}^G)\), simultaneously distilling the solver's systemic weaknesses and defining persistent criteria for newly generated synthetic queries.
3. Proposer Policy Evolution: Solvability-Constrained Adversarial Proposing
If the proposer creates trivial queries, the solver experiences diminishing returns; conversely, if it proposes impossible or ungrounded questions, policy gradients vanish. To maintain an optimal learning curriculum, the proposer \(\pi_{\theta_p}\) ingests the meta-rubrics and generates novel queries using diverse task combinations from preceding steps. The reward \(r_p^i\) for a proposed task \(o_p^i\) balances difficulty against solvability:
where the indicator \(\mathbb{I}\) requires that at least one sampled solver response successfully satisfies rubric \(R\) (verifying solvability), while the parenthesized term rewards lower mean solver scores (demanding maximal challenge). Optimized via GRPO, the proposer autonomously drives the solver toward edge cases and unmastered subject intersections.
4. Dual-Mode Hybrid Training Strategy: Balancing Parametric Reasoning with Search Tool Utility
Empirical investigation revealed that conducting end-to-end RL exclusively in tool-use mode exposes the policy to heavy search engine noise and network latency, obscuring report drafting and logical reasoning signals. Conversely, training solely in no-tool mode causes the model to over-rely on internal parametric recall, triggering an initial improvement followed by sharp performance degradation when subsequently evaluated with external search tools. HOTE resolves this tension through a two-phase curriculum: the first 600 steps operate entirely in no-tool mode to solidify internal synthesis, structure, and causal reasoning; the subsequent 700 steps implement a 50/50 randomized hybrid training split (half the batch executes no-tool pure reasoning while the other half executes multi-turn tool retrieval). The authors theoretically prove in Appendix B that the expected maximum generation latency \(\mathbb{E}[M_B]\) of this hybrid allocation strictly stochastically dominates that of full tool execution \(\mathbb{E}[M_A]\), significantly slashing wall-clock training bottlenecks.
Loss & Training¶
Both the solver and proposer are optimized using the token-level aggregated GRPO objective with a clipped surrogate loss and KL divergence regularization:
Implementation specifics and hyperparameters include: - Initialization: The solver is initialized from DR Tulu-8B-SFT; the proposer is initialized from Qwen3-8B; and the judge is parameterized by a frozen Qwen3-235B-A22B-Instruct-FP8 model (balancing high evaluative fidelity with a practical 753.3 s/step wall-clock throughput). - Batch and Exploration Sizes: Total batch size \(B=48\) (24 real queries and 24 synthetic queries), solver rollout group size \(G=8\), proposer rollout group size \(G'=6\); learning rate is \(5 \times 10^{-7}\), sampling temperature \(T=1.0\), maximum tool invocations per response \(T=10\), and maximum sequence length is 16,384 tokens. - Curriculum Schedule: 600 initial steps in no-tool mode (382.0 s/step) followed by 700 steps in hybrid mode (753.3 s/step), totaling 1,300 optimization steps.
Key Experimental Results¶
Main Results¶
Evaluations were performed across three rigorous long-form deep research benchmarks: HealthBench (1,000 samples evaluating medical safety, precision, and patient communication), ResearchQA (776 samples assessing scholarly synthesis across seven scientific disciplines), and DeepResearchBench (DRB, covering 22 diverse domains with fine-grained rubric criteria).
| Model Category / Method | HealthBench | ResearchQA | DRB Overall | DRB Comp | DRB Insight | DRB Follow | DRB Read | Benchmark Average |
|---|---|---|---|---|---|---|---|---|
| Closed Deep Research (Reference) | ||||||||
| Gemini 3 Pro + Search | 38.0 | 74.3 | 46.3 | 43.4 | 44.9 | 49.8 | 49.0 | 52.9 |
| GPT-5 + Search | 59.5 | 78.2 | 50.7 | 26.7 | 21.3 | 41.0 | 29.4 | 62.8 |
| OpenAI Deep Research | 53.8 | 79.2 | 46.9 | 46.8 | 45.2 | 49.2 | 47.1 | 60.0 |
| Open Static / Retrieval Models | ||||||||
| Qwen3-8B | 5.9 | 46.3 | 18.2 | 14.3 | 8.7 | 29.5 | 24.4 | 23.5 |
| Qwen3-235B-A22B | 21.3 | 50.7 | 22.5 | 19.1 | 17.3 | 30.6 | 25.1 | 31.5 |
| WebExplorer-8B | 33.7 | 64.8 | 36.7 | 33.7 | 28.5 | 45.7 | 42.2 | 45.1 |
| Tongyi DeepResearch-30B-A3B | 46.2 | 66.7 | 40.6 | 39.1 | 34.3 | 46.8 | 45.4 | 51.2 |
| DR Tulu-8B-RL (1900 steps) | 50.2 | 74.3 | 43.4 | 41.7 | 41.8 | 48.2 | 41.3 | 56.0 |
| RL / Self-Evolution Baselines | ||||||||
| Standard GRPO | 49.6 | 73.5 | 43.1 | 40.8 | 42.1 | 46.9 | 42.6 | 55.4 |
| GSPO | 51.0 | 75.1 | 43.6 | 42.5 | 40.9 | 47.3 | 43.7 | 56.6 |
| REINFORCE++ | 50.8 | 74.8 | 43.1 | 41.2 | 42.7 | 46.1 | 42.4 | 56.2 |
| SPICE-8B | 50.2 | 73.9 | 42.1 | 40.6 | 40.9 | 46.1 | 40.8 | 55.4 |
| Dr. Zero-8B | 52.1 | 73.2 | 43.7 | 41.5 | 42.1 | 46.5 | 44.7 | 56.3 |
| Proposed Method | ||||||||
| HOTE-8B | 54.4 | 76.9 | 45.9 | 44.9 | 45.4 | 47.8 | 45.8 | 59.1 |
Ablation Study & Evolutionary Dynamics¶
The empirical analysis investigated the indispensability of the three co-evolving modules, long-horizon sustainability, and training resource efficiency.
1. Indispensability of Tri-Evolution Modules (600 Steps No-Tool Training, 3 Random Seeds)
| Configuration | HealthBench | ResearchQA | DRB Overall | Impact Note |
|---|---|---|---|---|
| HOTE Full Tri-Evolution | 65.1 ± 0.0 | 66.3 ± 0.2 | 51.5 ± 0.0 | Optimal performance across all dimensions |
| w/o Proposer Evolution (Fixed Proposer) | 62.2 ± 0.2 | 64.2 ± 0.1 | 50.6 ± 0.1 | Task difficulty stagnates, ceiling solver progress (-2.9 / -2.1 / -0.9) |
| w/o Judge Evolution (Static Rubrics) | 61.9 ± 0.0 | 65.3 ± 0.2 | 50.7 ± 0.1 | Susceptible to reward hacking and coarse feedback (-3.2 / -1.0 / -0.8) |
2. Sustainability of Extended Evolution (1200 to 1500 Training Steps, Benchmark Average)
| Method | 1200 Steps | 1300 Steps | 1400 Steps | 1500 Steps | Trajectory Trend |
|---|---|---|---|---|---|
| GRPO Baseline | 55.4 | 55.4 | 55.2 | 55.2 | Completely plateaued at 1200 steps; slight overfitting |
| GSPO Baseline | 56.4 | 56.6 | 56.6 | 56.4 | Stagnant after step 1300 |
| Dr. Zero | 56.3 | 56.3 | 56.5 | 56.1 | Limited by absence of open-ended rubric steering |
| HOTE-8B | 58.0 | 59.1 | 59.6 | 59.9 | Sustained monotonic improvement through 252 wall-clock hours |
3. Training Latency and Computational Efficiency
| Method / Regime | Wall-clock Time (s/step) | GPU Hours (h/step) | Max Tool Calls over 1300 Steps |
|---|---|---|---|
| DR Tulu (All Tool-Use) | 1136.8 | 3.8 | \(1900 B T G\) (~729,600 calls) |
| SPICE-8B | 1313.7 | 4.0 | \(1300 B T G\) (~499,200 calls) |
| HOTE (First 600 Steps, No-Tool) | 382.0 | 1.5 | \(0\) calls |
| HOTE (Latter 700 Steps, Hybrid) | 753.3 | 2.6 | \(350 B T G + 175 B\) (~142,800 calls) |
Key Findings¶
- Tri-Evolution Prevents Query Stagnation: Tracking synthetic query scores revealed that when proposer evolution was disabled, solver scores on generated queries steadily rose from 2.8 to over 5.4, signifying that static proposers fail to challenge an improving solver. Conversely, with active proposer evolution, synthetic query scores remained tightly anchored between 2.1 and 4.1, continually presenting maximally informative training signals.
- Hybrid Curriculum Eliminates Tool Brittleness: Incorporating no-tool training in the initial 600 steps reduced per-step training latency by 66.4% while instilling strong discourse and analytical discipline. In the subsequent hybrid phase, HOTE achieved higher scores under both tool-use and no-tool evaluation settings compared to purely tool-trained baselines, disproving the assumption that tool augmentation inherently undermines internal parametric reasoning.
- Robust Generalization Across Base LLMs: When trained on Llama-3.1-8B-Instruct (supervised fine-tuned on DR Tulu data), HOTE maintained superior benchmark performance (33.1 on HealthBench, 64.2 on ResearchQA, 35.1 on DRB) over DR Tulu (28.8 / 60.0 / 32.3) and Dr. Zero (28.4 / 61.0 / 31.5), verifying that the tri-evolution dynamics are architecture-agnostic.
Highlights & Insights¶
- Meta-Rubrics as Dynamic Diagnostic Bridges: Replacing brittle point-wise reward models with an evolving rubric set solves the long-standing open-ended evaluation bottleneck. Compacting evaluations into meta-rubrics provides a principled, low-bandwidth bridge that directs the proposer's exploration straight into the solver's cognitive blind spots.
- Architectural Symmetry Between Solver and Proposer: By formulating query proposal as an analogous ReAct information-seeking task (differing merely by omitting the final
citeaction), HOTE unifies the model architecture across competing agents, drastically simplifying system deployment and parameter sharing. - Mathematical Amortization of Web Interaction Latency: Leveraging first-order stochastic dominance to prove that hybridizing no-tool and tool-use steps strictly lowers the expected maximum rollout latency offers a rigorous theoretical template for accelerating reinforcement learning in tool-augmented agents.
Limitations & Future Work¶
- Dependence on Seed Initialization: While HOTE autonomously discovers training trajectories, it still relies on an initial pool of 9K seed research prompts and warm-started SFT checkpoints, leaving true cold-start zero-data evolution in long-form generation as an open challenge.
- Real-World Web Retrieval Noise: Open search engines frequently inject irrelevant, biased, or noisy documents into the context window, which can occasionally mislead solver reasoning; developing self-correcting retrieval filters during evolution remains an active priority.
- Scaling to Giant Mixture-of-Experts (MoE) Backbones: Primary evolutionary runs were benchmarked on dense 8B parameter models due to compute constraints. Scaling the tri-evolutionary loop to frontier architectures like Qwen-235B or DeepSeek-V3 represents an essential frontier for future validation.
Related Work & Insights¶
- vs DR Tulu (Shao et al., 2025): DR Tulu pioneered Reinforcement Learning with Evolving Rubrics (RLER) for deep research but remained bound to a static 9K dataset without proactive task expansion; HOTE introduces co-evolving proposing and hybrid-mode training, achieving higher report quality in substantially fewer training hours and tool calls.
- vs Dr. Zero (Yue et al., 2026): Dr. Zero realized zero-data agent evolution through search-based self-play but was restricted to short-form, easily verifiable factoid QA; HOTE successfully unlocks self-evolution for complex, non-verifiable, multi-page deep research synthesis.
- vs SPICE (Liu et al., 2025a): SPICE utilized self-play on static corpus environments for formal reasoning; HOTE generalizes self-play to tool-augmented agents operating directly over dynamic, web-scale search interfaces.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first closed-loop self-evolution framework for long-form, non-verifiable deep research tasks via co-evolving tri-agent dynamics.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across three multifaceted benchmarks, multiple model backbones, seed-variance bounds, comprehensive ablations, and formal latency proofs.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous mathematical and agentic problem formulation, transparent algorithmic structure, and clear comparative analyses.
- Value: ⭐⭐⭐⭐⭐ Delivers an efficient, scalable, and open blueprint for training frontier autonomous research agents within practical academic compute budgets.