Skip to content

HardGen: Generating Hard Samples for Tool-use Agents

Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.285.pdf
Code: https://github.com/xxx
Area: LLM Agent
Keywords: Tool Use, Autonomous Agents, Hard Sample Synthesis, API Dependency Graph, Closed-loop Verification

TL;DR

Addressing the issue where current tool-use data generation adheres to shallow random sampling along simple happy paths, HardGen proposes a three-phase pipeline—failure-driven dependency graph sampling, reverse advanced tool abstraction for implicit hard query generation, and feedback-guided closed-loop CoT refinement—synthesizing 27k challenging trajectories that elevate a 4B model to 79.14% overall accuracy on BFCLv3, surpassing frontier proprietary models.

Background & Motivation

Equipping Large Language Models (LLMs) with tool-use capabilities through Function Calling is the foundation for developing autonomous agentic systems. As agents are deployed into intricate environments—ranging from enterprise workflow automation to dynamic database analytics and interface manipulation—their ability to coordinate multiple tools over extended horizons while resolving latent dependencies has become the primary benchmark of agentic competence. However, the scarcity of high-quality, complex, and verifiable interaction corpora presents a formidable bottleneck hindering further advancements in agentic reasoning.

Existing synthetic data pipelines (such as APIGen, ToolACE, and TOUCAN) predominantly follow a paradigm of uniform random sampling over static tool sets coupled with shallow user-assistant simulations. Trajectories synthesized under this regime typically follow linear, straightforward "happy paths", where task difficulty is superficially inflated through expanded candidate tool pools or explicit structural constraints rather than genuine cognitive hurdles. In realistic operational workflows, problems frequently require bridging implicit logical gaps: the output of one tool serves as a latent prerequisite or hidden input parameter for subsequent invocations, and user requests are naturally underspecified without stating every intermediate action. Lacking exposure to such implicit causal dependencies during training, contemporary agents remain fragile when faced with complex, logically intertwined real-world instructions.

Overcoming this bottleneck requires moving beyond unguided random generation toward an automated framework that directly targets model failure modes to generate structurally challenging, verifiable trajectories with non-trivial logical jumps. Core idea: construct a dynamic API dependency graph grounded in agent failure cases to guide legality-constrained hard sequence sampling, reversely synthesize advanced tools to formulate hard queries that omit intermediate steps, and iteratively refine Chain-of-Thought reasoning through closed-loop diagnostic feedback, producing a high-fidelity corpus of structurally deep and verifiable tool-use trajectories.

Method

Overall Architecture

The HardGen pipeline comprises an initialization phase and three sequential operational phases: First, over a comprehensive sandbox environment of 2,095 tools, dual-model self-evaluation identifies 1,204 failure APIs to construct a dynamic API Graph \(G=(V, D, P)\) filtered via semantic compatibility and sandbox validation. Subsequently, the pipeline executes Phase I (Execution Sequence Sampling), Phase II (Execution-conditioned Tool Evolution), and Phase III (Feedback-guided CoT Refinement), ultimately yielding a verified dataset of 27,000 multi-turn trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["API Simulation Environment<br/>2,095 Candidate Tools"] --> B["Failure-driven Legality-constrained Sequence Sampling<br/>Dual-model self-evaluation identifies 1,204 failure APIs, greedy graph walk toward target"]
    B --> C["Execution-conditioned Tool Evolution & Hard Query Generation<br/>Tool Maker abstracts multi-step trace into advanced tool, generating underspecified Q_hard"]
    C --> D["Closed-loop Feedback-guided CoT Refinement<br/>Reasoner generates plan while Verifier provides non-leaking diagnostic feedback"]
    D -->|All trajectory steps validated| E["High-Quality 27k Hard Sample Dataset<br/>Powers Supervised Fine-Tuning and Reinforcement Learning"]

Key Designs

1. Failure-driven Legality-constrained Sequence Sampling: targeting architectural weaknesses via dynamic API graphs

Uniform sampling over API pools inevitably produces invalid, unexecutable calls while failing to target the specific reasoning deficits of language models. HardGen deploys 2,095 tools as vertices \(V\) in an API Graph \(G=(V, D, P)\) and conducts self-evaluation across two lightweight heterogeneous models: Qwen3-4B and Llama-3.2-3B-Instruct. A tool is designated as a failure target only if both models produce incorrect outputs, yielding a Failure API Set \(T\) of 1,204 tools while eliminating single-model inductive bias. The graph edges \(D\) represent parameter prerequisites, pruned from 8,462 candidate connections down to 5,783 verified dependencies via static type/value-range checking and active sandbox execution validation.

When constructing a trajectory, the sampler anchors on a failure tool \(T_a \in T\) and step-wise selects candidate tools from the legal set \(T_{legal}(H) = \{T_c \in V \setminus H : Pre(T_c) \subseteq H\}\), where all prerequisite tools \(Pre(T_c)\) must reside in the execution history \(H\). To prevent random walk divergence, a greedy heuristic prioritizes legal tools that minimize the shortest path distance \(\text{dist}(T_k, T_a)\) in \(G\):

\[T_s = \begin{cases} \text{rand}(T_{legal}(H)), & T_a \in H \\ T_a, & I(T_a, H) = 1 \wedge T_a \notin H \\ \arg\min_{T_k \in T_{legal}(H)} \text{dist}(T_k, T_a), & \text{otherwise} \end{cases}\]

Executed calls update environment states and dynamically refine parameter constraints in \(P\) (e.g., propagating empirical output ranges to input schemas). Sampling continues until budget \(M\) is met, yielding a ground-truth execution sequence \(\Gamma_i = (C_1, E_1, \dots, C_M, E_M)\).

2. Execution-conditioned Tool Evolution & Hard Query Generation: enforcing logical bridging by omitting intermediate steps

Directly converting an execution trace \(\Gamma_i\) into a user prompt typically produces explicit instructions like "first call tool A, then pass its result to tool B", reducing the task to surface-level keyword matching without genuine causal reasoning. To create true cognitive jumps, Phase II introduces a reverse evolution mechanism. First, the Tool Maker agent (\(A_T\)) abstracts the entire multi-step sequence \(\Gamma_i\) into a single high-level Advanced Tool:

\[T_{adv} = A_T(\Gamma_i)\]

This advanced tool encapsulates the composite functionality and parameter handovers of the underlying trace into a coherent macro-operation.

Next, the Hard-query Generator (\(A_Q\)) conditions on the high-level description of \(T_{adv}\) to synthesize a user query \(Q_{hard} = A_Q(T_{adv})\) that specifies the end goal while deliberately omitting intermediate steps. This abstraction introduces implicit logical jumps: the agent cannot directly solve the query with a single API call and must autonomously infer latent variables and prerequisite chains—such as deducing that a flight purchase API requires zip codes rather than city names, necessitating prior conversion queries.

3. Closed-loop Feedback-guided CoT Refinement: iterative diagnostic correction for verifiable trajectory synthesis

While hard queries introduce rich implicit dependencies, they substantially increase the failure rate during Chain-of-Thought (CoT) synthesis. HardGen deploys an interactive pair: a Reasoner (\(A_R\)) and a Verifier (\(A_V\)). Initially, \(A_R\) is prompted with the hard query \(Q_{hard}\), the primitive tool library, and a subtle hint derived from \(T_{adv}\)'s description. If the generated function call diverges from the reference call \(C_i\), \(A_V\) inspects the mismatch and generates targeted corrective feedback \(Error_i^{(k)}\) without revealing the ground-truth parameters.

The diagnostic feedback is appended to the reasoning prompt across successive attempts:

\[Prompt_i^{(k+1)} = \text{Concat}(Prompt_i^{(k)}, Error_i^{(k)})\]

The Reasoner utilizes the enriched context to refine its reasoning trace \(CoT_i^{(k+1)}\) and function call, repeating for up to \(K_{max}=3\) attempts. A trajectory is retained only if every step succeeds within the attempt budget. In multi-turn dialogue expansion, subsequent user turns condition on prior validated dialog and execution states, ensuring context consistency across long interaction horizons.

A Worked Example

The authors provide a representative case study in flight booking (Rivermist to Stonebrook): - Original Execution Trace: First execute get_zipcode(city="Rivermist") returning "83214"; execute get_zipcode(city="Stonebrook") returning "74532"; then execute buy_tickets(zipcodeA="83214", zipcodeB="74532") returning the ticket confirmation. - Advanced Tool Synthesis: Synthesizes buy_tickets_adv(cityA, cityB), described as "Purchase air tickets between two cities by city names, returning the purchased ticket information." - Easy vs. Hard Query Formulation: An easy baseline query states: "Please check the zip code of Rivermist and Stonebrook first, then purchase air tickets between the two cities using the zip codes." In contrast, HardGen's \(Q_{hard}\) cleanly demands: "Please purchase air tickets between the city 'Rivermist' and the city 'Stonebrook'." - Internal Model Reasoning: Confronted with \(Q_{hard}\), the agent analyzes tool definitions, identifies that buy_tickets requires zip codes while inputs are city names, plans parallel calls to get_zipcode, gathers environment responses, and executes the final purchase call autonomously.

Loss & Training

The synthesized 27,000 trajectories power both Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL): - SFT Setup: Utilizes LLaMA-Factory with standard cross-entropy loss applied exclusively to Assistant response tokens. Training employs a global batch size of 1024, learning rate \(4\times 10^{-5}\), sequence length 20,480 tokens, over 5 epochs. - RL Setup: Implemented via the open-source Verl framework, applying reinforcement learning directly on base models without preliminary SFT. A deterministic verifiable binary reward is defined:

\[Reward = \begin{cases} 1, & \text{format correct } \wedge \text{ answer correct} \\ 0, & \text{otherwise} \end{cases}\]

If a tool call is required, the generation must pass strict AST parsing, exactly match reference call parameters, and adhere to the reasoning template; if no tool call is required, valid non-call textual answers receive reward 1 while spurious invocations receive 0. Training runs with KL coefficient \(1\times 10^{-3}\), 16 rollout samples, temperature 0.7, learning rate \(1\times 10^{-6}\), across 8 NVIDIA A100 80GB GPUs.

Key Experimental Results

Main Results

On the Berkeley Function Calling Leaderboard v3 (BFCLv3), models trained with HardGen demonstrate remarkable improvements over leading proprietary and open-source models (reported in official Accuracy %):

Model Parameters Single-Turn Overall Multi-Turn Overall BFCLv3 Overall
GPT-4o (2024-11-20) - 77.21 42.50 65.64
GPT-5.2 (2025-12-11) - 76.12 28.13 60.12
Gemini-3-Pro-Preview - 86.89 60.75 78.17
Claude-Opus-4-5-20251101 - 84.19 68.38 78.92
DeepSeek-V3.2-Exp 671B 80.77 44.88 68.81
Qwen3-235B-A22B-Instruct 235B 84.51 44.63 71.21
Qwen3-4B (Base Model) 4B 82.14 22.13 62.13
HardGen-4B-SFT (Ours) 4B 85.73 (+3.59) 49.65 (+27.52) 73.70 (+11.57)
HardGen-4B-RL (Ours) 4B 87.14 (+5.00) 63.13 (+41.00) 79.14 (+17.01)

Under identical SFT training on Qwen2.5-7B-Instruct, HardGen substantially outperforms existing state-of-the-art data synthesis pipelines:

Synthesis Pipeline Single-Turn (%) Multi-Turn (%) Overall (%) Gain over Base (Δ)
Base Model (Qwen2.5-7B) 77.30 7.62 54.07 -
TOUCAN 76.51 22.62 58.55 +4.48
InfTool 81.18 22.75 61.70 +7.63
ToolACE-MT 80.51 27.57 62.86 +8.79
HardGen-SFT (Ours) 83.99 40.75 69.58 +15.51

Ablation Study

Ablation across pipeline components on Qwen3-4B evaluated on BFCLv3:

Pipeline Variant Phase I (Sampling) Phase II (\(T_{adv}\) Abstraction) Phase III (CoT Refinement) BFCLv3 Overall (%)
Base Model (Qwen3-4B) - - - 62.13
(a) Random Sampling Random traces - - 64.90
(b) Failure Graph Sampling Failure graph - - 67.64
(c) Failure + Advanced Tools Failure graph \(T_{adv}\) and \(Q_{hard}\) - 70.18
(d) Failure + CoT Refinement Failure graph - Verifier feedback 69.23
(e) Full HardGen Pipeline Failure graph \(T_{adv}\) and \(Q_{hard}\) Verifier feedback 73.70

On holistic agentic evaluation (BFCLv4 Agentic), HardGen-8B-RL attains an overall score of 20.42, outperforming base Llama-3.1-8B-Instruct (6.88) by +13.54 points, and surpassing ToolACE-2-8B (13.50) and xLAM-2-8b-fc-r (10.24).

Key Findings

  • Multi-turn reasoning yields massive leaps: HardGen achieves its most dramatic improvements on multi-turn conversations (+41.00 via RL and +27.52 via SFT on Qwen3-4B). Abstracting explicit hints forces the model to maintain state tracking and infer chained dependencies independently.
  • Quantitative hardness strongly correlates with downstream gain: All three hardness metrics—dependency depth \(d\), latent variable count \(\ell\), and branching factor \(b\)—correlate positively with base model error rates (\(r \ge 0.42, p < 0.01\)) and accuracy improvements after training \(\Delta Acc\) (\(r \ge 0.37\)), proving that structural difficulty directly fuels performance gains.
  • Orthogonal contributions across phases: Omitting any pipeline component incurs substantial degradation; failure-driven sampling establishes target orientation (+2.74), advanced tool abstraction enforces logical jumps (+2.54), and closed-loop refinement guarantees strict correctness (+3.52).

Highlights & Insights

  • Reverse abstraction for authentic logical bridging: Rather than attempting prompt engineering to generate "hard questions", HardGen abstracts verified multi-step execution traces into macro-tools, subsequently prompting queries that omit intermediate steps. This information-hiding paradigm effectively manufactures true cognitive leaps without prompt leakage.
  • Conjunction criterion for failure isolation: Constructing the failure set using the conjunction rule across two distinct architectures (Qwen3-4B and Llama-3.2-3B) filters out idiosyncratic tokenization or format artifacts, retaining genuine structural graph bottlenecks.
  • Superior Pareto frontier between tool use and general capability: Evaluation on IFEval demonstrates that HardGen induces minimal degradation in general instruction following while vastly boosting tool execution, with RL proving strictly more Pareto-efficient than SFT.

Limitations & Future Work

  • Executable sandbox prerequisite: The pipeline relies on active environment execution for dependency pruning and verifier feedback, which restricts deployment in proprietary ecosystems where interactive execution sandboxes are unavailable.
  • Abstraction capacity of Tool Maker: Synthesizing high-level tools depends on LLM abstraction fidelity; irregular, heterogeneous, or semantically noisy API combinations can lead to suboptimal tool descriptions.
  • Scaling to open-web ecosystems: Future work should explore extending the API Graph paradigm to open-world tool repositories with tens of thousands of dynamic web endpoints via mock-environment synthesis.
  • vs. APIGen / ToolACE: Earlier pipelines focus on scaling tool breadth and syntactic validity over static pools, predominantly generating single-turn or simple linear traces; HardGen targets structural logical complexity, using failure graphs and reverse abstraction to train implicit causal inference.
  • vs. TOUCAN: While TOUCAN scales real-world MCP environments, task difficulty remains tied to environmental properties; HardGen introduces adaptive, failure-directed closed-loop synthesis that actively matches the model's frontier weaknesses.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Reverse advanced tool abstraction coupled with failure-driven graph traversal is exceptionally creative
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage spanning BFCLv3/v4, APIBank, ACEBench, scaling curves, and human correlation audits
  • Writing Quality: ⭐⭐⭐⭐⭐ Highly articulate, rigorous mathematical framing, and clear qualitative demonstrations
  • Value: ⭐⭐⭐⭐⭐ Provides an indispensable, reproducible data synthesis blueprint for advancing compact open-weight models to frontier agentic proficiency