ToolForge: A Data Synthesis Pipeline for Multi-Hop, Multi-Turn, and Self-Reflective Tool-Use Data¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Buycar-arb/ToolForge
Area: LLM Reasoning
Keywords: Tool Learning, Data Synthesis, Multi-Hop Reasoning, Self-Reflection, Multi-Turn Interaction
TL;DR¶
ToolForge introduces an end-to-end data synthesis pipeline that constructs multi-hop, multi-turn, and self-reflective tool-use trajectories from QA seeds without live API calls, enabling Qwen3-8B fine-tuned on just 4,250 samples to improve by 21.6% over the TIR baseline across five out-of-domain QA benchmarks while demonstrating strong generalization on BFCL and API-Bank.
Background & Motivation¶
Augmenting large language models with external tool-use capabilities is essential for breaking the bounds of static parametric memory, accessing timely external facts, and solving intricate domain tasks. In realistic information-seeking and multi-hop scenarios, an autonomous agent cannot rely purely on single-step function execution; it must continuously decompose questions across turns, track evolving dependencies, digest intermediate observations, and flexibly rectify course when retrieved evidence is ambiguous or misleading. However, standard post-training pipelines predominantly rely on final-answer supervision or single-turn API instruction tuning, failing to provide supervisory signals over structured reasoning and dynamic recovery behavior.
Constructing high-quality trajectories that exhibit such multi-hop dependencies and reflective recovery poses severe economic and scalability bottlenecks. Mainstream data creation frameworks typically execute vast numbers of live API queries or web search requests, incurring substantial API fees, execution latency, and network vulnerabilities that hinder multi-domain scaling. Conversely, template- or heuristic-based synthesis approaches sidestep monetary execution costs but largely collapse into single-step tool interactions with clean observations, leaving cross-turn planning and self-correction unaddressed. Consequently, when deployed agents encounter noisy or superficially relevant retrieval results, they frequently suffer from confirmation bias and hallucination, treating flawed evidence as definitive truth.
To bridge this supervision gap without incurring prohibitive live API expenses, this work investigates how to synthesize multi-step, error-tolerant tool trajectories directly from standard QA seed pairs. Core idea: construct an extensible virtual-tool space with 19 base tools and dual-gated variants, simulate paired clean and noisy retrieval observations across 29 perturbed interaction flows, and filter trajectories through a multi-layer validation engine audited by MCTS-mined hard negatives, enabling high-fidelity synthesis of multi-hop, multi-turn, and self-reflective tool data without live API execution.
Method¶
Overall Architecture¶
ToolForge takes original question-answer seeds \((Q, A)\) accompanied by supporting contexts and synthesizes verified multi-turn tool-calling trajectories without relying on large-scale live API execution. The overall architecture is structured into three sequential stages: Knowledge Space Preparation (KSP), Generative Interaction Modeling (GIM), and Multi-Layer Validation (MLV).
Before trajectory generation, original samples lacking explicit background contexts are augmented with candidate passages via hybrid retrieval and reranking, separating evidence into golden and noisy distractors. Next, KSP configures the virtual tool inventory, selects a tool sequence, determines the interaction paradigm, and outputs a structured task plan. GIM subsequently instantiates this plan into interactive dialogue turns under simulated paired retrieval conditions (good vs. noisy observations), injecting procedural error perturbations to teach error recovery. Finally, MLV performs rule-based syntactic checking and model-based semantic verification to purge corrupted or ungrounded trajectories before supervised fine-tuning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Original QA Seed (Q, A)<br/>& Evidence Distractor Pool"] --> KSP["Knowledge Space Preparation<br/>19 Base Virtual Tools + Dual-Gated Schema Variants + Task Planning"]
KSP --> GIM["Generative Interaction Modeling<br/>Paired Retrieval Simulation (Good/Noisy) + 29 Perturbed Interaction Flows"]
GIM --> MLV["Multi-Layer Validation<br/>Rule Verification (Syntax/Schema) + Model Verification (Semantics/Alignment)"]
MLV -->|Rejected| Discard["Discard Flawed Samples"]
MLV -->|Validated| Out["Curated SFT Tool-Use Dataset D_final<br/>(Multi-Hop / Multi-Turn / Self-Reflection)"]
Key Designs¶
1. Knowledge Space Preparation: Limited tool diversity and brittle schema coverage -> 19 base virtual tools with dual-gated variant generation -> Balances semantic intent stability with schema diversity
Pre-existing tool datasets frequently overfit to narrow function signatures and static naming conventions, causing models to fail when exposed to novel tool interfaces. To decouple data synthesis from external service APIs while preserving broad domain coverage, ToolForge establishes 19 retrieval-oriented base virtual tools spanning 18 specialized knowledge domains (e.g., people, creative works, historical events, geography, science, academic literature, law, economics, medicine, culture) plus a general-purpose fallback tool.
To expose models to rich schema variations, an LLM generates candidate variants \(\tilde{T}\) for each base tool, which must satisfy a dual-gating condition before acceptance into the tool inventory:
Here, \(V\) denotes the set of accepted variants. The semantic lower bound \(\theta_c = 0.8\) based on embedding cosine similarity ensures that candidate variants preserve the base tool's functional intent, while the lexical upper bound \(\theta_b = 0.5\) based on BM25 similarity prevents redundant, near-identical schemas; the cold-start indicator \(\delta_V = \mathbf{1}[|V| < 2]\) temporarily relaxes the semantic gate during initial variant collection. Furthermore, KSP structures tool usage into four interaction paradigms: Single-Round Single-Tool (SRST), Single-Round Multi-Tool (SRMT), Multi-Round Single-Tool (MRST), and Multi-Round Multi-Tool (MRMT). Given query \(Q\), the planner yields the control tuple \(\text{KSP}(Q) = \langle S^*, C^*, R^* \rangle\) representing the target tool sequence, paradigm, and reasoning outline.
2. Generative Interaction Modeling: Absence of realistic retrieval noise and recovery supervision -> Paired observation simulation with 29 perturbed interaction patterns -> Instills critical evidence inspection and adaptive query reformulation
Typical synthetic datasets presume ideal retrieval where every tool call instantly surfaces the exact golden context. Consequently, fine-tuned agents lack the critical awareness to detect superficial relevance and revise erroneous paths. GIM resolves this by simulating paired retrieval observations for each decomposed sub-query \(q_i\): a clean observation \(I_{\text{good}, i} = K_i \cup \{g_i\}\) containing the required golden context \(g_i\) along with top-k distractors \(K_i\), and a noisy observation \(I_{\text{bad}, i} = K_i\) consisting solely of topical distractors that fail to resolve the target relation.
Building upon the four KSP paradigms, GIM introduces three procedural perturbation types: tool misselection, argument misselection, and tool switching. Systematically combining these perturbations yields 29 representative reasoning-tool flows \(M = \{\text{flow}_i\}_{i=1}^{29}\). During dialogue generation, the model produces explicit <think> deliberation blocks, <tool_call> actions, and receives simulated <tool_response> observations. When exposed to \(I_{\text{bad}, i}\), the trace is guided to reflect upon entity discrepancies, recognize information insufficiency, and reformulate queries or switch tools, establishing active self-correction mechanisms in conversational contexts.
3. Multi-Layer Validation: Latent syntax flaws and ungrounded semantic drift in synthetic traces -> Rule-based and model-based cascaded verification audited by MCTS -> Eliminates syntax violations and reasoning hallucinations
Unchecked synthetic dialogues often exhibit syntax corruptions or subtle logical disconnects between thought rationales and tool invocations. ToolForge implements Multi-Layer Validation (MLV) comprising two complementary tiers:
- Rule Verification Layer (RVL): Applies nine deterministic checks to ensure that output formats adhere strictly to executable protocols. It validates
<tool_call>XML enclosure, tool naming fidelity against predefined schemas, argument completeness and typing, observation placeholder alignment, and answer format integrity without requiring physical code execution. - Model Verification Layer (MVL): Employs an LLM expert evaluator to inspect deeper semantic attributes across three dimensions: functional correctness (whether the selected tool fits the sub-task intent), logical soundness (whether intermediate reasoning steps are strictly grounded in retrieved evidence), and rationale-action alignment (whether the thoughts in
<think>logically justify the ensuing<tool_call>).
To rigorously benchmark MLV's filtering precision, the authors curate MLV-Bench, consisting of 580 human-verified positive trajectories, 1,305 rule-level negatives, and 87 semantic negatives. Crucially, rule-level hard negatives are synthesized via Monte Carlo Tree Search (MCTS) to explore complex multi-violation permutations across diverse interaction paradigms. On MLV-Bench, MLV achieves 98.47% accuracy and a 97.39% F1 score, ensuring that only structurally sound and logically grounded data is passed to training.
Loss & Training¶
Synthesized samples are formatted into standard chat interactions. SFT optimizes the base model using standard autoregressive cross-entropy loss over assistant-generated tokens:
Loss computation is restricted exclusively to <think>...</think>, <tool_call>...</tool_call>, and <answer>...</answer> spans, masking out system instructions, user queries, and simulated <tool_response> observations. Fine-tuning uses Qwen3-8B across four A100 80GB GPUs with DeepSpeed ZeRO-2, bfloat16 mixed precision, and FlashAttention-2. Training runs for 2 epochs with an effective batch size of 8, a learning rate of \(1 \times 10^{-5}\), and a maximum sequence length of 12k tokens, taking approximately two hours for 4,250 samples.
Key Experimental Results¶
Main Results¶
Primary evaluations are conducted across three general QA datasets (NQ, TriviaQA, PopQA) and two complex multi-hop QA benchmarks (MusiQue, Bamboogle) under an open-domain retrieval setting using the 2018 Wikipedia dump and a BM25s retriever (top-5). Evaluation follows the Exact Match (EM) metric.
| Model / Method | Category | NQ | TriviaQA | PopQA | MusiQue | Bamboogle | 5-Benchmark Avg. | Relative vs. TIR |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B + Direct Inference | Direct | 0.200 | 0.519 | 0.191 | 0.058 | 0.334 | 0.260 | -24.0% |
| Qwen3-8B + CoT | Chain-of-Thought | 0.209 | 0.512 | 0.182 | 0.055 | 0.332 | 0.258 | -24.6% |
| Qwen3-8B + TIR (Baseline) | Tool-Integrated | 0.301 | 0.593 | 0.358 | 0.094 | 0.365 | 0.342 | — |
| Qwen3-8B + RAG | Retrieval-Augmented | 0.299 | 0.572 | 0.346 | 0.096 | 0.274 | 0.317 | -7.3% |
| Qwen3-8B + IRCoT | Interleaved CoT | 0.264 | 0.542 | 0.331 | 0.113 | 0.342 | 0.318 | -7.0% |
| Qwen3-8B + Search-R1 | RL-Trained Search | 0.305 | 0.600 | 0.394 | 0.143 | 0.380 | 0.364 | +6.4% |
| Qwen3-8B + ZEROSEARCH-base | RL-Trained Search | 0.334 | 0.597 | 0.402 | 0.138 | 0.398 | 0.374 | +9.3% |
| ToolForge-8B (4.25k SFT, Ours) | Data Synthesis | 0.309 | 0.602 | 0.410 | 0.278 | 0.480 | 0.416 | +21.6% |
In addition, ToolForge is benchmarked against existing synthetic tool-data pipelines on the Berkeley Function Calling Leaderboard (BFCL) using Qwen3-8B as the common backbone:
| Training Dataset | Sample Count | Non-live AST | Live AST | Multi-turn | Relevance | Irrelevance | BFCL Overall Avg. |
|---|---|---|---|---|---|---|---|
| ToolACE | 25k | 0.941 | 0.765 | 0.386 | 0.931 | 0.707 | 0.746 |
| ToolLLM | 25k | 0.421 | 0.561 | 0.000 | 1.000 | 0.135 | 0.423 |
| xLAM | 25k | 0.911 | 0.576 | 0.156 | 0.805 | 0.338 | 0.557 |
| ToolForge (Ours) | 4.25k | 0.899 | 0.827 | 0.437 | 0.954 | 0.775 | 0.778 |
Ablation Study¶
Ablation investigations quantify the specific contributions of the Multi-Layer Validation (MLV) components, tool library configuration in KSP, and the paradigm composition ratio (SRST vs. complex multi-turn trajectories).
| Validation Configuration | NQ | TriviaQA | PopQA | MusiQue | Bamboogle | Avg. Score | Relative Drop |
|---|---|---|---|---|---|---|---|
| Full MLV (RVL & MVL) | 0.309 | 0.602 | 0.410 | 0.278 | 0.480 | 0.416 | — |
| RVL only | 0.244 | 0.582 | 0.323 | 0.261 | 0.327 | 0.347 | -16.6% |
| MVL only | 0.254 | 0.561 | 0.334 | 0.253 | 0.382 | 0.357 | -14.2% |
| Without RVL/MVL | 0.181 | 0.432 | 0.287 | 0.201 | 0.293 | 0.279 | -32.9% |
In paradigm mixture ablations, pure single-round data (SRST:Complex = 10:0) yields solid simple QA performance (NQ 0.334) but fails on multi-hop questions (MusiQue 0.183). Progressively introducing complex trajectories steadily boosts multi-hop reasoning, peaking at a 4:6 ratio (Avg. 0.451, MusiQue 0.301, Bamboogle 0.557), while the default 9:1 mixture provides a stable, conservative operational point (Avg. 0.416).
Key Findings¶
- Substantial Leap in Multi-Hop Reasoning: On MusiQue, ToolForge-8B achieves 0.278 EM, nearly tripling the TIR baseline (0.094) and outperforming advanced RL-based search methods such as Search-R1 (0.143) and ZEROSEARCH-base (0.138). On Bamboogle, it reaches 0.480 EM, confirming the effectiveness of explicit multi-turn dependency training.
- Strict Complementarity of Validation Tiers: Omitting semantic checking (RVL only) retains plausible but hallucinated reasoning paths, causing a 16.6% performance drop. Conversely, omitting syntactic checking (MVL only) leads to schema formatting failures (-14.2%). Completely removing validation collapses accuracy by 32.9%, demonstrating that uncurated synthetic traces severely corrupt agent policies.
- Sample Efficiency Over Data Scale: On BFCL, ToolForge trained on only 4,250 samples achieves an average score of 0.778, surpassing models trained on 25,000 samples from ToolACE (0.746) and xLAM (0.557), with decisive advantages in multi-turn interactions (0.437) and irrelevance detection (0.775).
- Extreme Construction Cost Efficiency: Compared to Search-R1, which requires ~64k live API requests totaling $657.0 in data construction, ToolForge requires zero live tool API requests. Its total construction cost is constrained to \(92.7 (\)80.6 LLM synthesis + $12.1 GPU compute).
Highlights & Insights¶
- Virtual Schema Space Decoupled from Live APIs: Formulating 19 base tools paired with dual-gated variant generation transforms tool learning into a controlled retrieval intent mapping. This avoids rate limits, network fragility, and execution expenses while allowing seamless schema expansion.
- Internalizing Retrieval Noise as a First-Class Supervisory Signal: Generating paired clean and noisy observations (\(I_{\text{good}} / I_{\text{bad}}\)) forces the model to deliberate on entity mismatches and dynamically reformulate queries, instilling active self-reflection rather than blind trust in retrieved content.
- MCTS-Audited Validation Benchmark: Establishing MLV-Bench with MCTS-mined hard negatives provides an adversarial diagnostic filter for synthetic data pipelines, attaining 98.47% accuracy in identifying corrupted trajectories.
Limitations & Future Work¶
- Manual Initialization of Base Tool Inventory: The initial specification of 19 base virtual tools relies on human intent analysis across Wikipedia corpora, which may require manual adaptation when extending to specialized enterprise domains (e.g., private APIs or robotics control).
- Corpus Bias Towards Wikipedia Knowledge: Seed instances originate primarily from HotpotQA and 2WikiMultiHopQA, biasing retrieved evidence towards encyclopedic entities rather than live interactive environments or code execution.
- Simulated Leaderboards vs. Live Deployment: Benchmarks like BFCL and API-Bank assess offline function-calling accuracy but do not fully reflect live network latency, intermittent HTTP 500 errors, or dynamic multi-agent concurrency.
Related Work & Insights¶
- vs. ToolACE / ToolLLM: ToolACE and ToolLLM rely on extensive collections of real-world APIs or single-turn executions (25k+ samples) with limited multi-turn and self-reflective coverage. ToolForge demonstrates that a compact, highly curated dataset (4.25k samples) featuring explicit recovery trajectories delivers superior multi-turn and out-of-domain reasoning.
- vs. Search-R1 / ZEROSEARCH: Reinforcement learning pipelines require protracted trial-and-error search and costly live environment rollouts. ToolForge demonstrates that lightweight SFT on multi-paradigm synthetic dialogues matches or exceeds RL performance across multiple QA benchmarks, serving as an ideal initialization phase for post-training.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Decouples tool-use trajectory synthesis from live APIs using dual-gated virtual tools and paired noisy observation modeling.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across five QA benchmarks, BFCL, API-Bank, ToolScan, extensive ablations, and detailed cost accounting.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous mathematical formulations and clear qualitative case studies.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient, practical blueprint for synthesizing robust, self-reflective tool-augmented agent data at low cost.