Skip to content

DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://aclanthology.org/2026.emnlp-main.1117/
Area: LLM Reasoning
Keywords: Deep Search Agents, Self-Distillation, Verifiable Environment, Process Supervision, ReAct Distillation

TL;DR

DeepSearch-World constructs a deterministic offline Wikipedia environment with reproducible tools and entity-level process verification, enabling the DeepSearch-Evolve framework to self-distill long-horizon web search and reflection behaviors into an open-source 9B agent without proprietary teacher models.

Background & Motivation

Autonomous tool-augmented agents powered by large language models have evolved from passive text generation into goal-directed systems capable of decomposing multi-step queries, generating dynamic search queries, visiting web pages, and synthesizing complex evidence. To enable these agents to improve iteratively from their own interaction experience, the community has largely followed two standard training paradigms: supervised fine-tuning (SFT) on self-generated successful trajectories, and reinforcement learning (such as GRPO) guided by verifiable outcome rewards. However, SFT on positive self-rollouts rapidly plateaus due to the inherent capacity constraints of the backbone model and the narrow diversity of self-collected traces. Conversely, outcome-based RL provides only sparse, trajectory-level reward signals at the final answer step, offering no structural guidance on whether an interactive failure stemmed from early query reformulation flaws, inappropriate tool routing, faulty evidence reading, or final answer hallucination.

Recent exploration of on-policy self-distillation (OPSD) attempts to bridge this reward sparsity by matching student rollout prefixes to dense token-level probabilities provided by fine-grained teacher models. Nevertheless, in agentic tool use, meaningful supervision fundamentally resides at the action and decision boundaries—namely, tool selection, evidence verification, query reformulation, and state tracking. When operating over open web environments or non-deterministic simulated environments, tool-use teacher distributions inevitably suffer from severe observation noise, API latency, web drift, and uncontrolled hallucinations. Consequently, existing self-evolving pipelines remain caught in a trilemma: static imitation lacks exploratory growth, sparse outcome RL lacks credit assignment, and dense distributional distillation lacks reproducible grounding.

The crucial angle of attack in this work is that long-horizon search agents require an offline, deterministic, and verifiable sandbox where every intermediate tool interaction can be objectively evaluated against ground-truth reasoning chains. Core idea: build DeepSearch-World, a deterministic offline environment over a 10-million-page Wikipedia corpus with reproducible search/visit tools and entity-level progress verification, combined with a scaffolded teacher agent and a scaffold-to-ReAct self-distillation loop (DeepSearch-Evolve) that internalizes planning, working memory, and error recovery into deployable open-source search agents without external proprietary distillation.

Method

Overall Architecture

The DeepSearch-Evolve self-distillation framework operates through three tightly integrated components: (1) DeepSearch-World Verifiable Environment: an offline deterministic interaction sandbox constructed over a 10M-entry Wikipedia corpus with Lucene BM25 search (web_search_wiki) and SQLite offset-indexed reading (visit_wiki), providing 420K multi-hop QA instances paired with hidden ground-truth entity chains for process-level verification; (2) Three-Phase Scaffold Teacher: a teacher policy executing structured PLAN (initializing working memory), ACT (multi-turn tool invocation with staged environment-grounded reflection), and END (grounded answer synthesis) phases to gather rich exploration trajectories; (3) Self-Evolving Training Loop: an asynchronous pipeline that performs answer-correctness rejection sampling, trajectory-level quality filtering, privileged-state-stripping scaffold-to-ReAct distillation, and importance-sampling-weighted SFT to iteratively elevate student policy capabilities across evolutionary rounds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Hop QA Tasks with Target Entity Chains"] --> B["DeepSearch-World Offline Environment<br/>BM25 Search + SQLite Full Page Reading + Entity Verification"]
    B --> C["Three-Phase Scaffold Teacher Rollout<br/>PLAN State Setup + ACT Memory Interaction + END Grounded Synthesis"]
    C --> D["Rejection Sampling and Quality Filtering<br/>Answer Correctness Filter + Redundancy and Alignment Judge"]
    D --> E["Scaffold-to-ReAct Distillation Conversion<br/>Entity Masking + Reflection Rewriting + Internalized Working Memory"]
    E --> F["Importance-Sampling Weighted SFT Updates<br/>Exponential Decay Mixing + Lightweight Real-Tool GRPO Adaptation"]
    F -->|Deploy as Next Teacher Policy| C
    F --> G["Standard Deployable ReAct Search Agent"]

Key Designs

1. DeepSearch-World: Deterministic Offline Environment with Entity-Level Verification

To eliminate live web noise, latency, and irreproducibility while providing dense step-level supervision, DeepSearch-World builds an offline environment directly on the Wikipedia hyperlink graph \(G=(V, E)\). By sampling random walks \(\tau = (v_1, \dots, v_H)\) and generating 420K multi-hop QA pairs with feature fuzzification (e.g., replacing exact years or locations with approximate temporal or geographic descriptions), the environment forces genuine multi-hop tool-based exploration. The platform provides two aligned offline tools matching real-world web browser schemas: web_search_wiki utilizes a Pyserini Lucene BM25 index over the 10M-entry corpus returning ranked snippets and URLs, while visit_wiki uses a two-level SQLite byte-offset index to fetch complete article contents in milliseconds. For every question \(q_i\), the environment maintains the target entity set \(\mathcal{T}_i = \{e_{i,1}, \dots, e_{i,H}\}\) and tracks the completed entity set \(\mathcal{S}_t\). When tool outputs match an unresolved target entity in \(\mathcal{T}_i \setminus \mathcal{S}_t\), progress is verified objectively without requiring costly or non-deterministic LLM evaluators.

2. Scaffold Teacher with Staged Environment-Grounded Reflection

Directly sampling unstructured rollouts frequently causes agents to spin their wheels in repetitive queries. The teacher agent adopts a structured cognitive scaffold spanning PLAN, ACT, and END phases. In the PLAN phase, the teacher initializes a working memory state \(s_0\) featuring four writable fields: completed_list (validated subgoals), todo_list (active research agenda), experience (lessons distilled from failures), and information (extracted factual snippets). During the ACT phase (up to \(T_{\max}=30\) turns), the environment verifies progress and injects staged rule-based reflection \(r_t\) upon tool failure: initial failures receive generic reformulative cues, while repeated dead-ends provide progressively specific canonical entity hints or fuzzy descriptors. This structure reliably induces "search–fail–reflect–retry" recovery traces without leaking ground-truth shortcuts into the final deployment policy.

3. Scaffold-to-ReAct Distillation: Internalizing Planning and Self-Correction

While the external scaffold guides teacher trajectory discovery, deployable web agents must operate within a clean, single-stream ReAct framework without expecting special system-injected state prompts or privileged feedback. The scaffold-to-ReAct conversion process transforms accepted teacher traces into standard ReAct supervision. For each tool-use step \(t\), the student's target thought block is composed as: $\(\langle\text{think}\rangle_t = \mathcal{P}_t \oplus \mathcal{R}_t \oplus \mathcal{A}_t\)$ Here, \(\mathcal{P}_t\) is rewritten from the progress state \(s_t\), internalizing completed subgoals and remaining tasks into natural-language reasoning; \(\mathcal{R}_t\) systematically strips out privileged raw environment tags and canonical entity strings, converting environment reflection into an autonomous self-correction rationale; and \(\mathcal{A}_t\) preserves the local justification for action \(a_t\). This conversion instills high-level planning, memory management, and error self-correction directly into the student's autoregressive reasoning stream.

4. Asynchronous Evolutionary Training Loop with Importance Sampling

To maximize rollout throughput across GPU clusters, trajectory generation runs continuously with vLLM while SFT fine-tuning executes asynchronously on collected batches. In each evolution round \(R\), when 4,000 trajectories successfully pass answer-correctness rejection sampling (RS) and a 3-criterion trajectory quality filter (QF, screening for non-redundancy, goal alignment, and logical accuracy), fine-tuning is triggered. To prevent catastrophic forgetting across iterative rounds, the data mixer applies importance sampling with an exponential decay factor \(\gamma=0.5\), blending newly acquired verified trajectories with historical experience. Finally, to eliminate the offline-to-online domain gap, the model undergoes lightweight GRPO adaptation over 1,600 real-world queries using Google SerpAPI and Jina page retrieval, adapting search schemas to open-web noise.

Loss & Training

The core optimization relies on supervised self-distillation over environment-verified ReAct trajectories \(\tilde{\tau} = (x, y_{1:T})\): $\(\mathcal{L}_{\text{SFT}}(\theta) = \mathbb{E}_{\tilde{\tau} \sim \mathcal{D}^{(R)}} \left[ \sum_{t=1}^T \text{KL}\left(\delta_{y_t} \,\|\, \pi_\theta(\cdot \mid x, y_{<t})\right) \right]\)$ This hard-label policy distillation avoids the high computational overhead and off-policy drift associated with token-level soft distillation objectives (\(\mathcal{L}_{\text{OPSD}}\)) over noisy intermediate states. Training is conducted with Llama Factory using DeepSpeed ZeRO-2 in BFloat16 precision, an initial learning rate of \(5 \times 10^{-6}\) with cosine decay, 10% warmup, and a maximum sequence context of 32,768 tokens across 11 evolutionary rounds.

Key Experimental Results

Main Results

DeepSearch-World-9B is evaluated across seven demanding deep search and reasoning benchmarks against leading proprietary models, established open-source search agents, and its base backbone (scores reported as percentages):

Model / Agent BrowseComp BrowseComp-ZH HLE GAIA xbench HotpotQA
Proprietary Agents
OpenAI Deep Research 51.5 42.9 26.6 67.4 – –
OpenAI-o3 49.7 58.1 20.2 70.5 65.0 –
Open Source Agents
R1-Searcher (7B) 1.0 – 5.4 8.3 – 62.4
Search-R1 (7B) 0.4 – 13.0 18.7 – 63.2
ZeroSearch (7B) 1.4 – 8.6 9.9 – 32.4
ASearcher (7B) 3.2 – 13.8 22.1 – 71.0
DeepResearcher (7B) 1.8 – 6.0 24.0 – 56.6
PokeeResearch (7B) 6.2 – 17.6 49.2 – 71.6
WebSailor (7B) 6.7 14.2 12.8 37.9 34.3 92.8
OffSeeker-DPO (8B) 12.8 26.6 17.6 51.5 48.0 –
WebExplorer (8B) 15.7 32.0 17.3 50.0 53.7 –
Marco-DR (8B) 31.4 47.1 – 69.9 42.0 –
MiroThinker-v1.0 (8B) 31.1 40.2 21.5 66.4 34.0 –
DeepDive (9B) 6.3 15.1 – – 35.0 –
Backbone & Ours
Qwen3.5-9B-Instruct (Backbone) 7.4 13.5 16.7 23.9 20.0 45.3
DEEPSEARCH-WORLD-9B (Ours) 31.2 36.4 25.7 61.5 49.0 93.4
Net Gain (\(\Delta\)) +23.8 +22.9 +9.0 +37.6 +29.0 +48.1

Ablation Study

Ablation experiments isolate the impact of trajectory filtering, process-level supervision, real-tool GRPO, and conversion strategies (all figures in percentages):

Table 1: Ablation of Rejection Sampling and Quality Filtering on Search-QA

Filtering Configuration Search-QA Accuracy (%) Core Impact Description
w/o both (No filtering) 46.4 Contaminated with erroneous answers and rambling tool calls
QF only (Quality filter only) 48.1 Removes redundant reasoning but includes unverified answers
RS only (Rejection sampling only) 54.9 Guarantees answer correctness but retains inefficient trajectories
SDAR (Distributional distillation baseline) 49.0 Soft targets drift during multi-turn action selection
Skill-SD (Skill-conditioned self-distillation) 47.8 Fixed skill schemas lack multi-hop flexibility
DeepSearch-World-9B (RS + QF) 58.2 Combines ground-truth correctness with streamlined reasoning

Table 2: Ablation of Process Supervision and ReAct Conversion on DeepSearch-Val

Component Ablation DeepSearch-Val (%) Search-QA (%) Note
(A) Process Reflection Supervision
Full staged reflection 31.9 58.2 Staged feedback recovers from failed searches
Generic-only reflection 26.5 52.0 Identical generic retry hints slow recovery
No reflection 24.7 49.0 Degrades to pure outcome-reward rejection sampling
(B) Trajectory Conversion Pipeline
Full conversion pipeline 31.9 – State internalization + reflection rewriting
w/o state internalization 23.5 – Omitting progress state weakens long-term goal tracking
w/o reflection rewriting 16.7 – Raw environment tokens corrupt reasoning distribution
w/o both 14.8 – Severe format mismatch causes immediate rollout breakdown

Key Findings

  • Rejection sampling acts as the foundational anchor: Filtering out factually incorrect trajectories yields the largest single performance leap (+8.5% on Search-QA), while trajectory quality filtering adds another +3.3% by pruning unaligned or redundant steps.
  • Drastic expansion of tool-use persistence: Qwen3.5-9B-Instruct terminates prematurely after only 4.7 turns on average (averaging 0.9 visit calls), whereas DeepSearch-World-9B sustains 18.0 interaction turns with 12.6 searches and 5.4 full-page visits. Its Advanced Capability Score leaps from 19% to 70%.
  • Reflection rewriting protects deployment distributions: Directly training on privileged environment tokens without rewriting collapses validation accuracy to 16.7%, demonstrating that environment feedback must be synthesized into first-person autonomous self-correction.

Highlights & Insights

  • Autonomous self-evolution without proprietary teacher dependencies: Most competitive open-source agents rely heavily on synthetic trajectories generated by GPT-4o or Claude. DeepSearch-World proves that an open-source 9B model can bootstrap itself into frontier performance purely through verified sandbox exploration.
  • Decoupled high-throughput simulation and real-world adaptation: By utilizing an offline BM25/SQLite Wikipedia engine for the bulk of iterative training, the authors avoid API rate limits and network latency, requiring only 1,600 real-world tool rollouts at the final stage to bridge the distribution gap.
  • Scaffold-to-ReAct internalization as a general distillation pattern: Using structured multi-field JSON states during rollout exploration and subsequently transforming them into clean natural-language <think> traces offers a robust recipe for training complex reasoning agents without prompt overhead at inference time.

Limitations & Future Work

  • Domain coverage bounded by Wikipedia: The deterministic environment relies exclusively on English Wikipedia, leaving open challenges in reasoning over multimodal web interfaces, dynamic e-commerce portals, or real-time breaking news.
  • SFT-centric update paradigm: While avoiding the instability of token-level OPSD, iterative SFT remains constrained by trajectory-level supervised objectives; integrating step-level advantage estimation within online PPO/GRPO remains an open frontier.
  • vs. Marco-DR & WebSailor: While Marco-DR synthesizes trajectories from frontier proprietary models, DeepSearch-Evolve trains entirely on self-generated experience validated by an offline environment, drastically reducing distillation costs.
  • vs. Search-R1 & ZeroSearch: Rather than relying purely on sparse trajectory-level outcome rewards that suffer from weak credit assignment, DeepSearch-World introduces intermediate entity-level progress verification and staged reflections to actively guide recovery.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant synthesis of offline verifiable sandbox construction, scaffolded cognitive rollouts, and privileged-to-autonomous ReAct conversion.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 7 search benchmarks, extensive ablations on process supervision, data scale, and conversion mechanics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, transparent prompt engineering documentation, and thorough mathematical framing.
  • Value: ⭐⭐⭐⭐⭐ Delivers an accessible, reproducible blueprint for developing self-improving deep search agents on modest academic compute budgets.