Adaptive Knowledge Routing for Complex Reasoning in LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Tianxingyi-fu/EFPL
Area: LLM Reasoning
Keywords: Tool-use Agents, Interaction Mode Planning, Mixed-intent Reasoning, Failure-routed RL, Self-distillation
TL;DR¶
This paper introduces EFPL, a decoupled planner-executor framework that structures the agent decision space into explicit ASK, ANSWER, and ACT interaction modes and dynamically routes all-failed rollout groups to privileged self-distillation, substantially curbing tool over-calling, argument fabrication, and unfeasible task hallucination on BFCL and When2Call benchmarks.
Background & Motivation¶
In practical agentic workflows, tool-augmented large language models (LLMs) must not only master tool selection and argument generation, but also discern whether invoking an external tool is genuinely appropriate in the first place. An autonomous agent should possess the discernment to navigate mixed-intent dialogues: recognizing when a request is fully specified and actionable via an external tool, when the query can be resolved directly via conversational memory, when critical constraints are missing and necessitate proactive user clarification, or when the request completely exceeds the operational envelope of the available APIs and demands faithful abstention. However, contemporary tool-augmented models frequently manifest an intrinsic over-calling bias, aggressively hallucinating parameters for underspecified requests or forcing unwarranted tool calls for unanswerable queries, severely undermining operational safety and reliability.
The core tension underlying these pervasive failure modes stems from the conflation of interaction-level judgment with low-level execution semantics within monolithic policy architectures. Under standard end-to-end training paradigms, policies are optimized directly over surface-level outputs—such as interleaved reasoning traces, raw JSON API schemas, or final natural language responses. When a rollout trajectory fails, the monolithic supervision signal conflates multiple orthogonal error sources, including incorrect interaction intent, incorrect tool indexing, syntactical parameter mismatch, and conversational miswording. More critically, when encountering an ambiguous or unanswerable query, if all sampled trajectories mistakenly attempt premature tool execution, standard outcome-based reinforcement learning (such as GRPO) assigns uniformly minimal scalar rewards across the entire sample group, resulting in an information-free gradient vacuum that fails to teach the model how to abstain or clarify; conversely, unconstrained offline teacher distillation destabilizes online exploration dynamics.
To bridge this supervision void and render interaction boundaries directly optimizable, the authors formulate mixed-intent tool use as an executor-grounded, instruction-level planning problem. By decoupling high-level intent planning from mechanical execution, the planner explicitly selects among active clarification (ASK), direct answering or abstention (ANSWER), and tool execution (ACT), leaving the downstream execution to a frozen executor. Core idea: decouple the agent into an explicit ASK/ANSWER/ACT interaction mode planner and a frozen executor, optimizing the policy via executor-grounded GRPO while conditionally routing completely failed rollout groups to privileged-context self-distillation for dense directional correction.
Method¶
Overall Architecture¶
EFPL decouples interactive tool-use into a lightweight interactive planner \(\pi_\theta\) and a frozen downstream executor \(E\) (e.g., Qwen3-8B). Given a conversational context \(x = (h, u, \mathcal{T})\) comprising dialogue history \(h\), current user utterance \(u\), and available tool signatures \(\mathcal{T}\), the planner generates a structured instruction \(y = (m, s)\). Here, \(m \in \{\text{ASK}, \text{ANSWER}, \text{ACT}\}\) denotes the discrete interaction mode, and \(s\) provides the corresponding semantic payload (such as clarification goals, abstention justifications, or targeted tool constraints). The frozen executor translates \(y\) into an observable output \(o = E(y)\), realizing either an environment-facing <tool_call> or a user-facing <response>. A reward manager \(\Phi\) compares \(o\) against ground truth \(g\) to compute format and execution outcome rewards. For each group of \(G\) sampled rollout trajectories, EFPL evaluates the maximum execution reward: groups with informative reward variance undergo Group Relative Policy Optimization (GRPO), whereas completely failed groups are routed to an online privileged self-distillation branch.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Dialogue Context & Toolset x = (h, u, T)"] --> B["Instruction-Level Interaction Planner<br/>Selects y=(m, s) over ASK / ANSWER / ACT"]
B --> C["Frozen Executor Grounding<br/>Instantiates observable action o = E(y)"]
C --> D["Outcome & Schema Reward Evaluation<br/>Computes scalar reward R_i and diagnostic signal f_i"]
D -->|Informative reward variance max R_exec > r_min| E["Executor-Grounded GRPO<br/>Group-relative advantage normalization & clipping"]
D -->|Group-level complete failure max R_exec = r_min| F["Privileged Failure-Routed Self-Distillation<br/>EMA teacher with ground-truth guidance injects dense correction"]
E --> G["Update Planner Policy π_θ"]
F --> G
Key Designs¶
1. Instruction-Level Interaction Mode Decoupling: Abstracting API Execution into Discrete Behavioral Modes To resolve the structural ambiguity of monolithic models where intent recognition and argument generation are entangled, this design establishes a strict schema forcing the planner to output exactly one mode \(m \in \{\text{ASK}, \text{ANSWER}, \text{ACT}\}\) alongside its semantic payload \(s\). Under the ACT mode, the payload specifies the target tool, achievable sub-goal, and necessary contextual parameters, allowing the frozen executor to construct well-formed function calls. Under the ASK mode, the payload identifies the precise missing parameter or ambiguous user constraint, directing the executor to produce a targeted clarifying inquiry. Under the ANSWER mode, the payload articulates direct resolution or explicit, safe abstention when the tool inventory cannot satisfy the goal. Decoupling frees the planner from parsing complex JSON syntaxes, allowing it to focus exclusively on resolving the boundary question of whether to act, ask, or answer.
2. Failure-Routed Privileged Self-Distillation: Rescuing Reward Vacuums via Conditional Online Guidance When a policy explores underspecified or out-of-domain queries, all sampled trajectories within a rollout group \(G\) often catastrophically collapse into incorrect tool invocations, yielding the minimal execution reward \(r_{\min}\). In this state, relative advantage normalization in GRPO degenerates, offering zero gradient guidance to discover alternative clarification or abstention paradigms. EFPL resolves this by introducing a group-level failure router: $\(z_i^{\text{Teach}} = \mathbf{1}\left(\max(R_{\text{exec}}^G) = r_{\min}\right)\)$ When a rollout group fails completely, it is routed to a privileged self-distillation branch. Here, a privileged context \(f\) (incorporating executor-side ground truth and reference instructions) conditions an online teacher \(\bar{\pi}_\theta\), parameterized via an exponential moving average (EMA) of student weights \(\theta\). The student planner is updated by minimizing distribution divergence over its past states under teacher supervision, providing dense, token-level corrective gradients that guide the policy out of over-calling local optima toward proper clarification or abstention.
3. Dual-Stream Executor-Grounded Reward Architecture: Harmonizing Schema Adherence with Action Realization To ensure planner instructions translate into functional and verifiable environment operations, EFPL deploys a dual-component reward structure \(R_i = R_{\text{fmt}, i} + R_{\text{exec}, i}\). The format reward \(R_{\text{fmt}, i} \in \{0, 1\}\) strictly validates whether the planner output satisfies action-conditioned schemas, guaranteeing semantic key presence without premature lexical penalization. The execution outcome correctness reward \(R_{\text{exec}, i}\) evaluates the downstream product: for tool executions, it computes a deterministic rule-based matching score combining tool name accuracy \(r_{\text{name}}\), parameter key Jaccard overlap \(r_{\text{param}}\), and normalized parameter value identity \(r_{\text{value}}\), scaled into \([-R_{\max}, R_{\max}]\); for user-facing clarification and direct responses, a frozen evaluator (GPT-4o-mini) grades semantic alignment and completeness into \(\{0, 1, 2\}\). This outcome-grounded formulation ensures planner learning is driven by downstream behavioral utility rather than superficial trace imitation.
Loss & Training¶
The overall training loss combines environment-driven exploration, conditional privileged correction, and a reference policy KL penalty: $\(\mathcal{L}_{\text{total}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G \left( z_i^{\text{RL}} \mathcal{L}_{\text{RL}}^{(i)}(\theta) + \alpha z_i^{\text{Teach}} \mathcal{L}_{\text{Teach}}^{(i)}(\theta) \right) + \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\right]\)$ where \(z_i^{\text{RL}} = 1 - z_i^{\text{Teach}}\). For rollout groups exhibiting informative reward differences, \(\mathcal{L}_{\text{RL}}\) optimizes the surrogate clipped objective using group-relative advantages \(A_i = (R_i - \mu_R)/(\sigma_R + \epsilon_{\text{std}})\). For all-failure groups, \(\mathcal{L}_{\text{Teach}}\) minimizes the Jensen-Shannon divergence over the Top-K student tokens relative to the EMA teacher distribution (\(\tau=0.1\)). Optimization is implemented using the verl framework across 16 Ascend 910B NPUs, configured with batch size 64, rollout group size \(G=8\), learning rate \(1\times 10^{-6}\), distillation weight \(\alpha=0.1\), and KL penalty \(\beta=5\times 10^{-4}\).
Key Experimental Results¶
Main Results¶
EFPL was evaluated across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B on BFCL (multi-turn base, missing-parameter, missing-function, and irrelevance subsets) and When2Call (macro F1 and accuracy across ToolCall, Ask Info, and Cannot Answer). Scores are reported as percentages.
| Backbone | Paradigm & Method | BFCL Base | BFCL Miss. Param. | BFCL Miss. Func. | BFCL MT Avg. | BFCL Irrelevance | When2Call F1 | When2Call Acc. |
|---|---|---|---|---|---|---|---|---|
| Qwen3-1.7B | Vanilla Base | 15.00 | 12.00 | 6.00 | 11.00 | 76.54 | 67.74 | 67.33 |
| Vanilla GRPO | 17.00 | 12.50 | 9.00 | 12.83 | 82.35 | 65.56 | 66.00 | |
| Decoupled Base-Planner | 8.50 | 8.50 | 4.00 | 7.00 | – | 55.06 | 58.67 | |
| Decoupled SFT-Planner | 5.50 | 5.00 | 6.00 | 5.50 | 87.19 | 59.91 | 61.00 | |
| Decoupled GRPO-Planner | 20.00 | 15.00 | 16.00 | 17.00 | 82.74 | 70.07 | 69.00 | |
| Decoupled EFPL (Ours) | 22.50 | 19.00 | 17.50 | 19.67 | 83.36 | 71.78 | 71.67 | |
| Qwen3-4B | Vanilla Base | 36.50 | 31.00 | 29.50 | 32.33 | 78.65 | 69.85 | 69.33 |
| Vanilla GRPO | 40.00 | 32.00 | 31.50 | 34.50 | 83.19 | 71.76 | 70.67 | |
| Decoupled Base-Planner | 28.00 | 29.00 | 23.50 | 26.83 | – | 67.12 | 66.67 | |
| Decoupled SFT-Planner | 19.00 | 7.00 | 3.50 | 9.83 | 83.45 | 67.15 | 66.67 | |
| Decoupled GRPO-Planner | 39.50 | 34.50 | 35.00 | 36.33 | 86.92 | 74.09 | 72.67 | |
| Decoupled EFPL (Ours) | 42.00 | 36.00 | 35.50 | 37.83 | 88.88 | 77.76 | 76.00 | |
| Qwen3-8B | Vanilla Base | 41.50 | 27.00 | 38.50 | 35.67 | 82.27 | 67.84 | 68.00 |
| Vanilla GRPO | 42.00 | 35.50 | 26.50 | 34.67 | 85.06 | 76.08 | 76.00 | |
| Decoupled Base-Planner | 31.00 | 24.50 | 28.00 | 27.83 | – | 67.42 | 66.33 | |
| Decoupled SFT-Planner | 20.00 | 7.00 | 6.00 | 11.00 | 76.60 | 57.17 | 58.67 | |
| Decoupled GRPO-Planner | 48.00 | 40.00 | 43.00 | 43.67 | 85.32 | 74.91 | 74.00 | |
| Decoupled EFPL (Ours) | 49.00 | 42.00 | 46.50 | 45.83 | 88.61 | 78.23 | 77.33 |
Ablation Study¶
The contribution of failure-routed privileged correction was validated on Qwen3-4B against three supervision variants under the identical planner-executor decoupled setup. Long-horizon environmental generalization was tested on \(\tau^2\)-bench Airline, and task-level efficiency was evaluated over 600 BFCL Multi-turn tasks.
| Strategy / Configuration (Qwen3-4B) | BFCL MT Avg. (%) | BFCL Irrelevance (%) | When2Call F1 (%) | When2Call Acc. (%) | Description |
|---|---|---|---|---|---|
| GRPO-Planner (RL-only ablation) | 36.33 | 86.92 | 74.09 | 72.67 | Deprived of guidance on all-failure batches |
| Pure OPD (Distillation-only ablation) | 31.67 | 87.44 | 71.71 | 71.33 | Lacks trial-and-error reward reinforcement |
| Fixed Mix (Unconditional 50/50 mix) | 35.50 | 87.60 | 75.02 | 73.67 | Dilutes policy gradient on successful trajectories |
| Flipped Router (Inverted routing logic) | 29.17 | 83.12 | 68.94 | 68.00 | Inverts optimal conditions, destabilizing training |
| EFPL (Full Method) | 37.83 | 88.88 | 77.76 | 76.00 | Selective failure routing achieves maximal synergy |
Environmental generalization and efficiency metrics: - \(\tau^2\)-bench Airline Success Rate: On Qwen3-4B, EFPL achieved 34% (vs. 28% for GRPO-Planner, a +6% absolute gain); on Qwen3-8B, EFPL reached 40% (vs. 32% for GRPO-Planner, a +8% absolute gain). - Task Efficiency on BFCL 600 Tasks (Qwen3-8B): Compared with Vanilla GRPO, EFPL improved complete task success from 34.67% to 45.83% while shrinking total required interaction steps from 10,310 to 6,853 (average steps per task dropped from 17.18 to 11.42, a 33.5% reduction in step count).
Key Findings¶
- Boundary Delineation as the Primary Bottleneck: Accuracy on unambiguous ToolCall inputs is already near saturation (92%–96%) across backbones. The primary empirical divergence lies in distinguishing Ask Info from Cannot Answer. EFPL specifically reallocates probability mass toward valid abstention, lifting true Cannot Answer accuracy by 7.3 percentage points over GRPO-Planner.
- Architectural Decoupling Requires Grounded Optimization: Decoupling alone without grounded policy optimization impairs performance; Base-Planner and SFT-Planner significantly underperform the vanilla base model on multi-turn tracking (e.g., SFT-Planner yields only 9.83% on Qwen3-4B MT Avg), proving that imitation learning of schema syntax fails to impart boundary-sensitive decision policies.
- Invariance to Downstream Executor Capacity: Substituting the frozen Qwen3-8B executor with Qwen3-4B, GPT-4o-mini, or DeepSeek-V3.2 while keeping the EFPL-trained planner fixed results in near-identical performance (BFCL MT Avg remains within a narrow band of 44.33%–45.83%), confirming that system proficiency is governed by planner mode-routing rather than executor reasoning.
Highlights & Insights¶
- Explicit High-Level Behavioral Action Space: Structuring tool interaction as a three-way categorical choice (ASK/ANSWER/ACT) isolates the strategic boundary problem from messy low-level argument token generation.
- Conditional Rather Than Unconditional Self-Distillation: Restricting privileged teacher guidance strictly to rollout groups with completely failed execution avoids degrading exploratory policy gradients on successful instances while effectively rescuing reward vacuums.
- Substantial Step Reductions in Multi-Turn Deployments: By curbing speculative API calling and hallucinated parameter trials, the system slashes cumulative interaction steps by 33.5%, directly lowering inference token budgets and preventing environmental state corruption.
Limitations & Future Work¶
- Inference Latency Overhead: The decoupled two-stage architecture introduces unavoidable serial generation overhead, requiring an intermediate planner instruction before emitting final executable tokens.
- Myopic Single-Step Horizon: The current planner formulates immediate single-step actions rather than multi-step dynamic decision graphs, potentially limiting long-range proactive planning in complex graph-structured workflows.
- Future Directions: Exploring dynamic adaptive merging where the intermediate planner step can be bypassed for high-confidence trivial tool calls, and extending the framework to multimodal agents operating under partial observability.
Related Work & Insights¶
- vs SAGE-Agent & TRUST: SAGE-Agent and TRUST incorporate parameter uncertainty directly into reward scaling to deter over-calling. In contrast, EFPL structuralizes the discrete action space into ASK/ANSWER/ACT and applies conditional failure-routed privileged supervision, providing explicit directional gradients for non-actionable prompts.
- vs ToolRL & Torl: ToolRL leverages verifiable environment execution rewards to train monolithic policies. EFPL extends this principle by using executor-grounded outcomes to optimize high-level interaction modes, preventing argument overfitting while teaching proactive dialogue clarification and safe abstention.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Decouples high-level interaction modes from execution and pioneers failure-routed privileged self-distillation for boundary learning.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously evaluated across BFCL subsets, When2Call, \(\tau^2\)-bench, and multiple external evaluator and executor models.
- Writing Quality: ⭐⭐⭐⭐⭐ Highly structured, mathematically clear, with thorough ablation and error categorization.
- Value: ⭐⭐⭐⭐⭐ Offers an actionable, production-ready recipe to combat agent over-calling and hallucinated API arguments in complex environments.