Skip to content

ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialogues

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/IBM/ToolWeave
Area: LLM Agent
Keywords: Multi-Turn Tool Calling, Synthetic Data Generation, Parameter Provenance, Tool Graph, LLM Agents

TL;DR

ToolWeave establishes a modular, verifiable synthetic data generation framework that constructs curriculum-driven tool graphs with built-in dataflows, extracts motif-aligned workflows, and tracks parameter provenance across fine-grained plans, driving the share of true multi-step tool turns from under 7% to over 40% and cutting parameter hallucinations from over 50% to under 20%.

Background & Motivation

The evolution of large language models into autonomous decision-making agents hinges on robust multi-turn tool calling: the capacity to autonomously invoke APIs, chain dependent operations across conversational turns, request user clarification when required parameters are missing, and maintain coherent execution states. However, curating diverse, enterprise-grade multi-turn tool-use dialogues from real human interactions is prohibitively expensive, operationally difficult to scale, and fraught with privacy and compliance constraints. Consequently, synthetic data generation (SDG) pipelines have emerged as the standard paradigm for training and fine-tuning tool-calling models. Yet, existing synthetic pipelines (such as ToolFlow, ToolDial, and Magnet) suffer from two pervasive and intertwined pathologies: semantically incoherent tool workflows and widespread parameter hallucinations.

Existing pipelines predominantly construct tool graphs by taking a static set of APIs (such as those crawled from RapidAPI) and inferring connectivity edges via heuristic semantic similarity over parameter names and descriptions. This heuristic strategy routinely yields spurious linkages between conceptually incompatible tools—such as joining create_support_ticket(issue) to cancel_movie_ticket(ticket_id) simply because both schemas expose an argument named ticket_id. Because these workflows reflect superficial syntactic overlaps rather than genuine operational dependencies, upstream tool executions rarely produce outputs that downstream tools actually require. As an inevitable consequence, synthesized dialogues almost never contain authentic multi-step tool interactions within a turn; across prior pipelines, fewer than 7% of conversational turns feature multi-step tool calls, and 0% exhibit true dataflow propagation where later calls ingest the results of earlier ones. Furthermore, prior approaches typically prompt an LLM to generate entire dialogues or high-level plans in a single unconstrained pass. Leaving the execution structure implicit forces the model to guess argument origins, resulting in severe parameter hallucinations where values are invented out of thin air; in practice, over 50% of dialogues generated by existing pipelines exhibit hallucinated arguments or outputs.

These failures share a common root cause: the complete absence of structured, verifiable intermediate representations capable of explicitly modeling tool-to-tool dependencies and tracking parameter provenance. To overcome these limitations, the generation process must be decomposed into modular, verifiable stages where tools are constructed with intrinsic information flows and dialogue parameters are formally bound to their exact historical sources. Core idea: decompose multi-turn tool dialogue synthesis into four verifiable stages—curriculum-driven tool graph construction with built-in dataflow dependencies, motif-based workflow sampling aligned with user goals, fine-grained planning with explicit parameter provenance tracking and clarification injection, and stateful multi-agent execution with deterministic failure injection—to generate rich, hallucination-free multi-step agent trajectories.

Method

ToolWeave treats dialogue synthesis not as an unconstrained open-ended text generation problem, but as an executable, step-by-step transformation over formal graph representations. The pipeline operates in four staged phases: synthesizing domain-specific tool graphs with verified data dependencies; sampling realistic user workflows via structural motifs; partitioning workflows into fine-grained dialogue plans with explicit parameter binding; and executing these plans via coordinated, memory-backed agent personas followed by robustness-enhancing error injection.

Overall Architecture

The synthesis pipeline flows systematically from domain descriptors to final robust multi-turn interaction traces:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Domain Descriptor d"] --> B["Stage 1: Curriculum-Driven Tool Graph Synthesis<br/>Knowledge Retrieval → Entity Expansion → Schema Enrichment → Connection Discovery"]
    B --> C["Stage 2: Motif-Based Workflow Sampling & Goal Alignment<br/>Structural Motif Mining + LLM Goal Synthesis + MMR Diversity Pruning"]
    C --> D["Stage 3: Fine-Grained Segmentation & Parameter Provenance Planning<br/>Path Partitioning + Probabilistic Clarification + Explicit Parameter Binding"]
    D --> E["Stage 4: Stateful Multi-Agent Dialogue Synthesis & Robustness Augmentation<br/>Role-Based Execution + Shared Memory Tracking + Deterministic Error Injection"]
    E --> F["Output: Realistic Multi-Turn Tool-Calling Dataset"]

Key Designs

1. Curriculum-Driven Tool Graph Synthesis: Surface schema matching causes spurious connectivity -> Staged knowledge grounding and explicit connection discovery -> Highly connected tool graphs with intrinsic dataflow

Prior pipelines rely on static real-world APIs and link them post-hoc via textual similarity, which frequently bridges incompatible business entities. ToolWeave addresses this by synthesizing domain-specific APIs via a five-stage curriculum (Seed Generation \(\rightarrow\) Entity Expansion \(\rightarrow\) Schema Enrichment \(\rightarrow\) Connection Discovery \(\rightarrow\) Pattern Expansion). The synthesizer first anchors domain contexts using structured entities from Wikidata and summaries from Wikipedia, incorporating domain-specific operational rules (e.g., establishing that when recurring is true in equipment maintenance, interval_days becomes mandatory). During Schema Enrichment, it injects structural depth via nested objects, enums, defaults, and optional flags. In Connection Discovery, rather than hoping existing tools match, the pipeline explicitly prompts the LLM to design new tools that take the output fields of earlier tools as inputs (e.g., designing get_product_details to accept product_id output by search_product). Candidate connections are verified through sentence-embedding cosine similarity and LLM judgment, after which parameter names are paraphrased (e.g., renaming feedback_id to feedback_record_id) so downstream models cannot rely on trivial string matching. In human blind evaluation of graph edges, 74% of ToolWeave connections formed semantically coherent workflows, compared to only 12% for ToolDial.

2. Motif-Based Workflow Sampling and Goal Alignment: Random walks yield unnatural tool loops -> Structural motif mining with hybrid heuristic scoring -> Task-oriented workflows strictly aligned with human goals

Unconstrained graph traversals such as random walks regularly generate circular dependencies or illogical operational sequences (e.g., dispatching a shipment before an order is created). ToolWeave restricts workflow generation to three structural motifs reflecting real-world task topologies: Linear Chains, extracted via bounded beam search over the tool graph to capture progressive workflows (e.g., search tickets \(\rightarrow\) get details \(\rightarrow\) update status \(\rightarrow\) notify user); Fan-in / Fan-out Structures, which identify common descendant nodes to model parallel information gathering before a consolidated action (e.g., checking stock and calculating risk concurrently before updating an order); and Conditional Branching, which scans tool schemas for boolean or enum predicate outputs to branch dynamically based on runtime return values. For each extracted path, ToolWeave prompts an LLM to synthesize an overarching natural-language goal \(G\) and scores the pair using a hybrid objective: $\(S_{\text{final}} = w_l \cdot (S_{\text{coherence}} + S_{\text{relevance}}) + w_d \cdot S_{\text{dataflow}} + w_b \cdot S_{\text{length}}\)$ where \(S_{\text{coherence}}\) and \(S_{\text{relevance}}\) denote LLM ratings in \([-2, +2]\), \(S_{\text{dataflow}}\) is the cosine similarity across adjacent tool schemas using all-MiniLM-L6-v2, and \(S_{\text{length}}\) provides a length bonus (\(w_l=0.5, w_d=0.8, w_b=0.3\)). Linear paths meeting quality thresholds are diversified via Maximal Marginal Relevance (MMR, \(\lambda=0.7\)), while all valid complex motifs are preserved.

3. Fine-Grained Segmentation and Parameter Provenance Planning: One-shot generation causes severe parameter hallucinations -> Explicit path partitioning, proactive clarification, and formal source binding -> Elimination of hallucinated arguments and dense multi-step execution

Generating full conversational turns directly from abstract task descriptions leaves parameter origin and state transitions implicit, leading to hallucination rates above 50%. ToolWeave shifts the reasoning burden into an explicit planning stage composed of interaction act primitives. The planner partitions the tool sequence into contiguous sub-groups (e.g., \([[A], [B, C], [D]]\)), mapping each group to an executable dialogue segment with its own sub-goal, which naturally enables multi-step tool calls within a single conversational turn. Within each segment, tool arguments are categorized into user-provided parameters (\(P_{\text{user}}\)) and derivable parameters (\(P_{\text{derived}}\)). For user inputs, the planner withholds required fields with probability \(p_{\text{clar}}=0.3\), explicitly scheduling assistant clarification requests (ASSISTANT_CLARIFICATION) and user clarifications (USER_RESPONSE_TO_CLARIFICATION) before any tool execution occurs. Crucially, the planner formally binds every downstream tool parameter to its exact origin (e.g., assigning feedback_record_id ← $submit_feedback.feedback_id). During generation, these bindings are deterministically resolved against preceding tool responses, completely eliminating fabricated arguments and driving verified dataflow tracing accuracy to 94.2%.

4. Stateful Multi-Agent Dialogue Synthesis and Robustness Augmentation: Idealized static traces lack real-world resilience -> Memory-agent orchestration and deterministic error injection -> High fault recovery and error-handling capability

To faithfully instantiate the fine-grained plan without hallucination, ToolWeave employs a role-decoupled multi-agent framework. Dedicated agents handle specific interaction acts, accompanied by a Tool Agent that generates schema-compliant synthetic mock returns (eliminating live API execution requirements) and a Memory Agent that maintains a persistent state blackboard tracking resolved arguments, tool outputs, and historical entities. Once the dialogue transcript is generated, it undergoes three deterministic post-processing augmentations: first, with probability \(p_{\text{inject}}=0.7\), systematic failure scenarios are injected—including out-of-order execution (calling a downstream tool before prerequisite outputs exist, receiving an error, and executing the proper recovery sequence), semantic confusion / wrong tool invocation (calling a confusable tool identified via SBERT and Levenshtein distance before self-correcting), and schema violations / missing functions (mutating JSON types or temporarily withholding a function definition to train dynamic schema ingestion); second, user utterances are paraphrased to introduce linguistic diversity; and third, API function names are masked with generic identifiers (e.g., func_01), compelling the agent to reason over parameter semantics and schemas rather than memorizing superficial function names.

A Worked Example

Consider a multi-step customer support workflow involving three APIs: get_ticket_details, submit_feedback, and trigger_feedback_followup.

  1. Dependency Binding: In the synthesized tool graph, submit_feedback requires customer_id, which is bound to get_ticket_details.user_id. The subsequent trigger_feedback_followup tool requires feedback_record_id, which is strictly bound to submit_feedback.feedback_id.
  2. Plan & Clarification Generation: The planner partitions the tools into two segments. Segment 1 executes get_ticket_details({"support_ticket_id": "ST-20251118-8392"}), returning user ID U-54892. In Segment 2, the user expresses a desire to log feedback and follow up on low ratings, but omits the exact score and contact channel. The planner inserts ASSISTANT_CLARIFICATION(["rating", "comments", "followup_method"]).
  3. Execution Trace: The user responds with rating 2, comment "The issue is still unresolved," and preferred method "phone". In a single turn, the assistant executes submit_feedback using the bound customer_id: "U-54892", receives {"feedback_id": "FB-20251121-68317"}, immediately resolves the provenance variable, and invokes trigger_feedback_followup({"feedback_record_id": "FB-20251121-68317", "followup_method": "phone"}). Every argument is grounded in preceding execution outputs, producing a 100% verified trajectory.

Key Experimental Results

Main Results

The models were evaluated on the multi-turn tool-calling benchmark BFCL-V3 across 800 dialogues (648 containing multi-step interactions). Base models across three families (Llama-3.1-8B, Llama-3.1-70B, and Phi-4) were fine-tuned via QLoRA on 3.2K dialogue trajectories generated by ToolDial, ToolFlow, and ToolWeave variants. Metrics report Overall Average Accuracy (Avg. Acc.), Base tool execution (Base), Missing Function refusal (Miss Func.), Missing Parameter clarification (Miss Param.), and Long Context handling (Long Ctx.).

Model & Training Corpus Avg. Acc. (%) Base (%) Miss Func. (%) Miss Param. (%) Long Ctx. (%)
Llama-3.1-8B (Base) 9.25 11.00 8.00 8.50 9.50
+ ToolDial (GPT-4o) 1.75 0.50 3.50 2.50 0.50
+ ToolFlow (GPT-5-mini) 7.50 9.50 5.50 5.50 9.50
+ ToolWeave-R (RapidAPI, GPT-5-mini) 18.38 24.00 18.00 14.50 17.00
+ ToolWeave (GPT-OSS-120B Open) 19.88 23.00 21.50 15.50 19.50
+ ToolWeave (GPT-5-mini Frontier) 21.12 28.00 18.00 18.50 20.00
Llama-3.1-70B (Base) 12.50 17.00 13.00 10.50 9.50
+ ToolDial (GPT-4o) 3.75 4.50 3.00 4.00 3.50
+ ToolFlow (GPT-5-mini) 23.50 29.50 20.50 20.00 24.00
+ ToolWeave-R (RapidAPI, GPT-5-mini) 37.38 40.00 43.00 34.00 32.50
+ ToolWeave (GPT-OSS-120B Open) 33.25 37.50 35.00 32.00 28.50
+ ToolWeave (GPT-5-mini Frontier) 39.75 42.50 43.00 37.00 36.50
Phi-4 (Base) 3.12 7.50 0.00 2.50 2.50
+ ToolDial (GPT-4o) 2.00 2.00 0.50 3.50 2.00
+ ToolFlow (GPT-5-mini) 10.12 11.00 10.00 11.50 8.00
+ ToolWeave-R (RapidAPI, GPT-5-mini) 26.25 33.50 33.50 20.50 17.50
+ ToolWeave (GPT-OSS-120B Open) 24.50 24.00 28.50 31.00 14.50
+ ToolWeave (GPT-5-mini Frontier) 29.50 36.00 30.50 31.50 20.00

Across additional multi-turn benchmarks, ToolWeave consistently outperforms baselines: on CONFETTI (evaluating goal switching and complex conversation dynamics), ToolWeave fine-tuned Llama-3.1-70B reaches 45.45% accuracy compared to 33.00% for the Base model and 28.46% for ToolFlow (+16.99 points); on API-Bank Level 1 and Level 2, it achieves 71.43% and 63.51%, respectively.

Ablation Study

Component ablations were performed on BFCL-V3 with Llama-3.1-8B-Instruct fine-tuned on data generated with the open-weight GPT-OSS-120B seed model:

Pipeline Configuration Average Accuracy (%) Absolute Difference Key Takeaway
Full ToolWeave Pipeline 19.88 - Complete 4-stage pipeline with post-processing
w/o Post-Processing (no error injection) 13.62 -6.26% Model lacks exposure to recovery from syntax and tool errors
w/o Structured Sampler (random walk) 12.25 -7.63% Workflows lack semantic coherence and authentic task motifs
w/o Fine-Grained Planner (ToolFlow-style) 7.50 -12.38% Catastrophic collapse; model cannot infer multi-step dataflows

Direct structural properties of the synthesized corpora show marked differences:

Synthesis Pipeline Avg. Turns Avg. Tool Calls % Multi-Step Turns % True Multi-Step Turns Clean (No Hallucination) (%)
ToolWeave 2.71 3.22 44.87% 40.67% 80.0%
ToolFlow 6.28 2.20 6.77% 0.0% 16.5%
ToolDial 4.48 1.51 4.47% 0.0% 46.5%

Key Findings

  • Fine-Grained Planning Prevents Catastrophic Collapse: Replacing turn-level parameter planning with a monolithic high-level plan causes accuracy to collapse from 19.88% to 7.50%, underscoring that open-weight models fundamentally require explicit state transitions during synthesis.
  • Synthetic APIs Outperform Real-World APIs: Models trained on ToolWeave's fully synthetic schemas outperform those trained on RapidAPI schemas (39.75% vs. 37.38% on Llama-3.1-70B), demonstrating that rich synthetic schemas with nested types and explicit dataflows provide a cleaner, more rigorous training signal than messy real-world APIs.
  • Open-Weight Synthesis Surpasses Frontier Baselines: ToolWeave driven by the open-weight GPT-OSS-120B yields a 33.25% BFCL-V3 score on Llama-3.1-70B, easily besting ToolDial generated via proprietary GPT-4o (3.75%) and ToolFlow via GPT-5-mini (23.50%), proving that pipeline structure matters far more than raw generator scale.

Highlights & Insights

  • Shifting Reasoning to Planning Time: Rather than forcing the synthesis LLM to simultaneously reason about user intent, parameter provenance, and dialogue flow in a single pass, ToolWeave externalizes dataflow into deterministic graph constraints, yielding 80.0% clean, hallucination-free dialogues.
  • Endogenous Dependency Construction: By designing new tools specifically to ingest upstream outputs during the Connection Discovery stage, the framework guarantees dense, authentic multi-step dependencies by construction (40.67% true multi-step turns vs. 0% in prior pipelines).
  • Self-Healing Trajectory Injection: Artificially injecting out-of-order execution, semantic tool confusion, and schema errors into 70% of traces trains downstream models to effectively self-correct and recover when encountering runtime faults.

Limitations & Future Work

  • Non-Executable Tool Simulation: The tool agent produces schema-valid but static JSON responses without executing real backend logic, meaning it cannot simulate runtime side effects, database constraints, or state mutations needed for interactive online reinforcement learning.
  • Higher Synthesis Computational Cost: Decomposing generation into structured stages increases computation; synthesizing 1,000 dialogues requires 30.2K LLM calls and 2.6 hours (versus 13.6K calls and 2.2 hours for ToolFlow), representing a one-time tradeoff of throughput for data quality.
  • Text-Only Modality: Current tool interactions are restricted to JSON-based text APIs and do not support multimodal environments such as GUI computer use or visual perception interfaces.
  • vs ToolFlow (Wang et al., 2025): ToolFlow constructs tool graphs via lexical similarity and uses unconstrained high-level planning, resulting in bloated dialogues (6.28 turns) with shallow execution (2.2 calls, 0% true multi-step); ToolWeave uses motif sampling and parameter binding to achieve compact, dense trajectories (2.71 turns, 40.67% true multi-step).
  • vs ToolDial (Shim et al., 2025): ToolDial relies on 21 rigid interaction templates and caps dialogues at 2 tool calls, generating stiff dialogues that leak variable names; ToolWeave employs flexible interaction act primitives and probabilistic parameter withholding to naturally trigger clarifications, outperforming ToolDial on BFCL-V3 by over 36 percentage points on 70B models.
  • vs APIGen / ToolACE (Liu et al., 2024; 2025): While APIGen focused heavily on single-turn verification, ToolWeave scales verifiable structured synthesis to complex, multi-turn, multi-tool dependency graphs.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering framework that solves semantic incoherence and parameter hallucination in multi-turn tool calling through structured graph synthesis and explicit provenance tracking.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across three distinct benchmarks, three LLM families, both open and frontier synthesis models, comprehensive ablations, and human blind reviews.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional conceptual clarity, clean formalization of stages and interaction acts, and rigorous root-cause analyses.]
  • Value: ⭐⭐⭐⭐⭐ [Fully open-sources code, synthetic schemas, and datasets, offering an essential recipe for training commercially licensable autonomous agents without frontier API dependence.]