When the Bottleneck Shifts: Diagnosing and Closing Information Gaps in Interactive Decision-Making¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/authurlord/PASCAL_Release
Area: Dialogue Systems
Keywords: Interactive Decision-Making, Interactive Text-to-SQL, Information Acquisition Bottleneck, Cost-Aware Scheduling, Retry Breaker Control
TL;DR¶
This paper formalizes the Information Sufficiency Hypothesis (ISH) in interactive decision-making and Text-to-SQL, proposes a five-class taxonomy of information gaps, and develops PASCALโa training-free acquisition-control data agent framework that prioritizes cost-normalized evidence gathering, active clarification, and adaptive retry breaker gating, allowing an on-premise 35B model to outperform frontier proprietary models by up to +14.7 pp on BIRD-Interact.
Background & Motivation¶
In real-world analytical database interfaces and conversational decision-making systems, user natural language queries are inherently under-specified and ambiguous. Translating these ambiguous analytical requests into valid executable SQL queries requires far more than parsing the database relational schema; the essential semantic evidence is frequently distributed across external domain knowledge bases, unstated business definitions, conversational context, and runtime user intent. Traditional static Text-to-SQL pipelines operate under a closed-world input paradigm where questions, schema metadata, and auxiliary context are prepared prior to inference, focusing primarily on syntactic query decomposition, execution-guided self-correction, or supervised fine-tuning. In interactive environments, however, critical domain rules and thresholds cannot be assumed upfront and must be sequentially acquired by an agent interacting with the environment under a finite interaction budget.
Standard autonomous agent scaffolds, such as ReAct, expose all environment tools (schema exploration, knowledge base search, user clarification, and SQL execution) indiscriminately to the language model. When deployed in interactive Text-to-SQL environments like BIRD-Interact, these unconstrained agents exhibit severe failure modes: they frequently commit prematurely to generating and submitting SQL before acquiring necessary domain definitions, subsequently falling into futile SQL retry loops that burn budget without resolving the binding information gap. Conversely, when attempting knowledge retrieval without structural guidance, agents query broad terms that flood the context window with irrelevant definitions. This challenge is further exacerbated because environment interactions in database systems incur non-trivial real-world costs: SQL execution places heavy load on production database engines, knowledge base lookups query external enterprise services, and user clarifications consume finite human attention. Consequently, the primary scarce resource in interactive decision-making is not inference tokens, but the constrained external interaction budget.
Through controlled trajectory ablation experiments holding backbone model capability constant, the authors discover that strong LLMs fail not from deficient SQL reasoning or synthesis, but from premature commitment to SQL generation prior to closing external semantic gaps. To formalize this diagnosis, the paper presents the Information Sufficiency Hypothesis (ISH) and establishes a rigorous five-source gap taxonomy. Core idea: reframe interactive decision-making from generation-centric trial-and-error into budgeted information acquisition, using a cost-normalized priority rule to front-load low-cost schema grounding and typed knowledge retrieval, gate user clarification behind structural signals, and enforce an adaptive retry breaker to prevent ungrounded generation loops.
Method¶
Overall Architecture¶
PASCAL (Priority-Aware Source-Calibrated Acquisition Layer) is a training-free, model-agnostic interaction-control framework designed for interactive data agents. The overall execution flows across three sequential phases under a finite interaction budget \(B\): proactive information acquisition (closing schema, knowledge base, and user gaps), query construction with retry breaker control (gating SQL execution and halting unproductive revisions), and multi-phase state preservation (carrying forward intermediate results into follow-up queries). Each environment interaction deducts an action-specific material cost \(c(a_t)\) from budget \(B\), and a lightweight routing controller dynamically restricts the permitted action space to enforce the cost-calibrated acquisition hierarchy.
The end-to-end operational pipeline is illustrated in the vertical flowchart below, showing the progression from zero-cost schema pre-injection through namespace-guided knowledge retrieval, evidence-constrained user clarification, execution feedback with stagnation-triggered retry breaking, to multi-phase CTE composition:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Natural Language Query and Environment Inputs"] --> B["1. Schema Pre-Injection and Structural Grounding<br/>Mount full DDL at zero cost and validate table references"]
B --> C["2. Typed Knowledge Retrieval and Namespace Discovery<br/>Two-stage namespace overview and on-demand definition fetch"]
C --> D["3. Evidence-Constrained User Clarification<br/>Parameter-level confirmation gated behind cheaper channels"]
D --> E["4. State-Monitored Adaptive Retry Breaker<br/>Stagnation hash tracking and error-repeat channel switching"]
E -->|Phase-1 Pass / Execution Match| F["5. Multi-Phase State Carry and CTE Composition<br/>Materialize prior SQL as CTE and inherit acquired evidence"]
F --> G["Final Validated Execution Output"]
Key Designs¶
1. Schema Pre-Injection and Structural Grounding: Eliminating Low-Leverage Exploration -> Zero-Cost DDL Injection -> Conserving Acquisition Budget
In standard interactive agent setups, models consume substantial portions of their finite interaction budget issuing exploratory tool calls (explore_tables, list_fkeys) to map out relational schema geometry, directly starving higher-leverage external semantic channels. Because schema mapping is a structural prerequisite rather than the primary reasoning bottleneck, PASCAL injects the complete schema metadata \(\mathcal{M}\) (DDL, primary keys, foreign-key relationships) directly into the agent context at task initialization as a zero-cost operation. The controller then executes local structural validation against \(\mathcal{M}\), rejecting invalid column names or table aliases before issuing queries. This upfront grounding completely eliminates the schema gap \(\Delta_{\text{sch}}\) without deducting from the interaction budget \(B\), ensuring all available interaction steps are preserved for discovering complex domain logic.
2. Typed Knowledge Retrieval and Namespace Discovery: Unguided Semantic Search -> Two-Stage Namespace Routing -> Eliminating Hallucinated Business Thresholds
Enterprise domain knowledge bases contain extensive formula specifications, regulatory criteria, and categorical ordinal scales that cannot fit within a prompt context without diluting attention. When agents search blindly with arbitrary natural language keywords, terminology mismatches result in retrieval failures, prompting the model to rely on parametric guesses that produce plausible but incorrect thresholds. PASCAL organizes knowledge bases into typed, callable entries indexed by canonical titles and brief summaries. Acquisition proceeds in two stages: the agent first invokes list_kb_names to gain complete namespace visibility at minimal cost, and subsequently issues targeted get_kb_def calls only for matching domain terms. At turn \(t\), for each gap class \(c\), the controller estimates an unresolved need signal \(\text{need}_t(c)\) and prioritizes actions using cost-normalized priority \(\text{priority}_t(c) = \text{need}_t(c) / \text{cost}(c)\). This design guarantees that the low-cost, high-yield knowledge base channel \(\Delta_{\text{kb}}\) is systematically queried before any expensive downstream actions are permitted.
3. Evidence-Constrained User Clarification: Costly Open-Ended Inquiries -> Evidence-Grounded Confirmation -> Minimizing Human Interaction Overhead
Human user clarification (\(\Delta_{\text{usr}}\)) incurs the highest material and latency cost among all interaction channels, and open-ended queries ("What does this mean?") often elicit vague responses. PASCAL enforces structural gating that strictly locks the ask_user channel until cheaper structural and knowledge base channels have been thoroughly exhausted. Clarification is triggered only when one of three observable conditions is met: (i) an ambiguous query term lacks any corresponding entry in the retrieved knowledge base namespace; (ii) repeated SQL submissions produce identical query hashes across \(N_v\) consecutive turns; or (iii) a retrieved knowledge definition contains an unbound hyperparameter. Furthermore, the agent is constrained to formulate questions grounded in previously acquired context (e.g., asking "The KB defines instability as TempVar24h > 1 OR HumVar24h > 3; should this threshold be applied?" rather than a generic intent question), transforming expensive disambiguation into high-precision, single-parameter confirmations.
4. State-Monitored Adaptive Retry Breaker: Infinite SQL Tweaking -> Trajectory-Aware Intervention -> Breaking Stagnation by Channel Redirection
Standard text-to-SQL pipelines heavily rely on database execution feedback to iteratively repair syntax errors; however, when an error stems from missing domain logic, repeated SQL retries merely waste budget. PASCAL incorporates an adaptive retry breaker monitoring the submission history \(H_t = \{e_1, \dots, e_t\}\), where each entry records the error category and normalized SQL hash. The breaker triggers according to:
$$
\text{fire}(H_t) = \begin{cases}
\text{STOP}, & |H_t| \ge N_{\text{cap}} \
\text{STUCK}, & \text{last } N_v \text{ hashes match} \
\text{SWITCH}, & \text{last } N_e \text{ errors match}
\end{cases}
$$
The STOP condition imposes a hard ceiling on cumulative submissions to preserve execution budget; STUCK detects when the model resubmits identical query structures, immediately escalating the trajectory to targeted user clarification (ask_user); and SWITCH detects repeated identical database error types, forcibly breaking the generation loop and redirecting the agent back to examine missing knowledge base definitions. Additionally, failed executions receive structured value-diff feedback capturing discrepancies in output cardinality, column shapes, and sample value divergence.
5. Multi-Phase State Carry and CTE Composition: Multi-Turn Context Breakage -> State Inheritance and CTE Materialization -> Bridging Multi-Phase Analytical Workflows
In real-world business intelligence workflows, follow-up questions frequently reference the output of earlier questions (e.g., "for each of those showcases, list the artifact IDs on display"). Without systematic state tracking, agents fail to resolve co-references across dialogue turns (\(\Delta_{\text{ph}}\)). Upon successfully passing Phase 1, PASCAL preserves all accumulated information items \(I^{(1)}_T\) and the validated SQL solution \(\sigma_1\), initializing Phase 2 with state \(I^{(2)}_0 = I^{(1)}_T \cup \{\sigma_1\}\). In the Phase-2 context, \(\sigma_1\) is automatically materialized as a named Common Table Expression (CTE, WITH phase1_result AS (...)), allowing the Phase-2 agent to join directly against prior results without re-discovering Phase-1 filtering conditions, completely resolving multi-phase dependency gaps with zero additional environment queries.
Diagnostic Protocol and Closed-Loop Optimization Policy¶
To quantify information acquisition during interaction, the paper introduces a post-hoc Information Sufficiency Score \(S(q)\): $$ S(q) = \frac{1}{|\mathcal{C}|} \sum_{c \in \mathcal{C}} \frac{|I^c(q) \cap I{|I^}|_c(q)|} $$ where \(\mathcal{C} = \{\text{tab}, \text{col}, \text{kb-id}, \text{kb-term}, \text{intent}\}\) indexes five acquisition facets across schema, knowledge base, and user intent, \(I^*_c(q)\) denotes the gold required evidence subset, and \(I_{T, c}\) represents the items acquired by trajectory end.
This diagnostic formulation demonstrates that task success is strictly monotonically linked to information coverage: tasks with \(S(q) \in [0.7, 1.0]\) achieve a 57.8% pass rate, whereas those with \(S(q) < 0.3\) collapse to 16.7%. Guided by this diagnostic insight, PASCAL enforces a strict closed-loop acquisition schedule: low-cost structural grounding precedes typed semantic retrieval, which precedes user clarification, leaving SQL generation and retry breakers to operate only after sufficient evidence has been assembled.
Key Experimental Results¶
Main Results¶
On the BIRD-Interact benchmark lite_300 split (comprising 195 Business Intelligence tasks and 105 Data Management tasks across 18 PostgreSQL databases), PASCAL was evaluated against static generation pipelines, official unconstrained ReAct agents, and frontier proprietary models. Metrics report Phase-1 success rate (P1%), two-phase conversational success rate (P2%), and the BIRD-Interact Normalised Reward.
| Method | Backbone Model / Framework | BI Success(%) | DM Success(%) | P1 Success(%) | P2 Success(%) | Normalised Reward |
|---|---|---|---|---|---|---|
| DIN-SQL | Qwen3.6-35B | 10.8 | 14.3 | 12.0 | 6.7 | 10.40 |
| DAIL-SQL | Qwen3.6-35B | 15.9 | 29.5 | 20.7 | 13.0 | 18.37 |
| MAC-SQL | Qwen3.6-35B | 7.7 | 10.5 | 8.7 | 1.7 | 6.57 |
| CHESS | Qwen3.6-35B | 14.4 | 8.6 | 12.3 | 6.0 | 10.43 |
| Claude-Opus-4.6 | Official a-Interact scaffold | 24.2 | 58.8 | 36.3 | 24.0 | 32.63 |
| Claude-Sonnet-3.7 | Official a-Interact scaffold | 22.1 | 56.2 | 34.0 | 17.7 | 29.10 |
| GLM-4.7 | Official a-Interact scaffold | 18.6 | 48.4 | 29.0 | 9.0 | 23.00 |
| MiniMax-M2.1 | Official a-Interact scaffold | 18.5 | 46.5 | 28.3 | 13.0 | 23.73 |
| Kimi-2.5 | Official a-Interact scaffold | 17.8 | 44.9 | 27.3 | 11.3 | 22.53 |
| Qwen3.6-27B | Official a-Interact scaffold | 19.0 | 48.6 | 29.3 | 19.3 | 17.70 |
| Qwen3.6-35B | Official a-Interact scaffold | 10.8 | 44.8 | 22.7 | 11.3 | 12.27 |
| PASCAL 27B | Qwen3.6-27B (on-premise) | 41.8 | 60.0 | 48.3 | 35.4 | 44.42 |
| PASCAL 35B | Qwen3.6-35B (on-premise) | 43.6 | 64.8 | 51.0 | 37.0 | 46.80 |
Ablation Study¶
An additive ablation on lite_300 using the Qwen3.6-35B backbone isolates the contribution of each architectural layer. Tracked metrics include P1 pass rate, Normalised Reward, InfoFail% (failures due to missing KB or intent evidence), 1st-KB (normalized position of the first ground-truth KB call, \(0=\text{start}, 1=\text{never}\)), Rpt-SQL (peak repeat ratio of identical SQL), and Overthink (fraction of turns following information saturation).
| Configuration Variant | P1 Success(%)โ | Normalised Rewardโ | InfoFail(%)โ | First KB Call (1st-KB)โ | Repeat SQL Ratio (Rpt-SQL)โ | Overthink Ratioโ |
|---|---|---|---|---|---|---|
| No protocol (Baseline) | 32.3 | 33.5 | 86.67 | 1.000 | 0.430 | 0.354 |
| + Schema Pre-injection & Progressive Access | 36.9 | 28.5 | 67.70 | 1.000 | 0.680 | 0.307 |
| + Typed KB Retrieval | 42.1 | 37.1 | 40.69 | 0.680 | 0.300 | 0.311 |
| + Harness & Retry Breaker (Full Model) | 51.0 | 46.80 | 31.10 | 0.427 | 0.258 | 0.154 |
Key Findings¶
- Strategic budget allocation dominates model scale in driving performance: PASCAL with an on-premise 35B open-weights model achieves 51.0% P1 and 46.80 Reward, outperforming the leading frontier model (Claude-Opus-4.6 at 36.3% P1 and 32.63 Reward) by +14.7 pp P1 and +14.17 Reward. Running the exact same Qwen-35B model under the official unconstrained ReAct scaffold yields only 22.7% P1, demonstrating that unstructured exploration severely suppresses model potential.
- Acquisition routing disproportionately benefits complex Business Intelligence (BI) queries: While Data Management (DM) tasks rely primarily on schema context, BI tasks demand external business formulas and thresholds residing outside the database schema. PASCAL 35B increases BI success from 10.8% to 43.6% (+32.8 pp gain), compared to a +20.0 pp gain on DM tasks, verifying that typed knowledge acquisition resolves the primary external information bottleneck.
- Tool-calling entropy reveals distinct interaction regimes: Static generation baselines (DAIL-SQL) exhibit minimal tool entropy (\(H=0.81\)) by spending 96.9% of actions purely in SQL generation; unconstrained a-Interact displays dispersed high entropy (\(H=2.20\)) by invoking tools blindly without closing target gaps; PASCAL maintains a moderate, calibrated entropy (\(H=1.78\)), concentrating actions on targeted evidence collection before transitioning smoothly to SQL generation.
- The retry breaker successfully eliminates unproductive tail generation: Adding knowledge retrieval reduces InfoFail% from 67.7% to 40.7% but leaves Overthink elevated (0.311); adding the deployment harness and adaptive retry breaker cuts Overthink by half to 0.154, reduces repeated SQL resubmission to 0.258, and advances the first ground-truth KB call from 0.680 to 0.427, maximizing budget utility.
Highlights & Insights¶
- Diagnosing the core failure mode as premature commitment rather than generation weakness: By demonstrating that generation failure rates are zero (GenFail% = 0.0%) when tasks fail on baseline models, the paper rigorously proves that models fail because they attempt to write queries without the necessary context. The Information Sufficiency Hypothesis shifts the community focus from tuning SQL generators to engineering evidence-gathering policies.
- Cost-normalized priority provides a principled, training-free scheduling rule: Using \(\text{priority}_t(c) = \text{need}_t(c) / \text{cost}(c)\) aligns interaction scheduling directly with operational costs in enterprise systems (database execution load, API costs, human cognitive fatigue), offering an interpretable, plug-and-play mechanism that requires no reinforcement learning or policy training.
- Two-stage namespace discovery coupled with parameter-level user confirmation: Exposing lightweight namespace listings before fetching full definitions prevents prompt pollution, while gating user clarification behind prior evidence transforms vague queries into precise parameter verificationsโa paradigm directly transferable to general-purpose conversational software agents.
Limitations & Future Work¶
- Dependence on structured knowledge base organization: The current benchmark assumes external knowledge bases can be pre-processed into clean namespace listings and typed definitions. In noisy enterprise environments with sprawling unstructured documentation, building and maintaining these namespaces remains a challenging engineering hurdle.
- Evaluation under simulated user feedback: The user clarification experiments rely on automated rule-based simulators that return deterministic, gold-grounded answers. In practice, human users frequently introduce additional ambiguity, express contradictory requirements, or misinterpret technical questions, which may test the robustness of the clarification gating schedule.
- Static design-time cost assumptions: PASCAL currently relies on fixed, pre-set cost constants for each channel. In production environments where database query latency, server load, and user responsiveness fluctuate dynamically, adaptive cost-learning algorithms could provide more resilient scheduling.
Related Work & Insights¶
- vs Static Text-to-SQL Pipelines (DIN-SQL, MAC-SQL, CHESS): Static approaches optimize \(P(\text{SQL} \mid \text{context})\) assuming complete context is statically provided in the prompt, remaining below 20.7% P1 on BIRD-Interact; PASCAL addresses the interactive setting where context must be actively acquired under finite budgets, prioritizing \(P(\text{sufficient context})\).
- vs Autonomous ReAct Scaffolds (a-Interact): a-Interact leaves tool ordering entirely to autonomous model reasoning, leading to speculative tool exploration and ungrounded retry loops; PASCAL imposes structured phase boundaries and cost-calibrated channel gating that transform chaotic trial-and-error into structured evidence convergence.
- vs Conversational Clarification Systems (PRACTIQ, ECLAIR, AmbiSQL): Prior clarification work isolates user dialogue from external knowledge retrieval and budget constraints; PASCAL embeds user clarification within a broader cost hierarchy, demonstrating that exhausting cheaper knowledge base channels first significantly reduces the frequency and complexity of user interruptions.
Rating¶
- Novelty: โญโญโญโญโญ Formulates the Information Sufficiency Hypothesis, creates an exhaustive five-source gap taxonomy, and designs a cost-calibrated acquisition framework for interactive agents.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive 300-task evaluation, fine-grained trajectory case studies across six real tasks, ablation of each component, and cross-benchmark validation on PRACTIQ.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, elegant formalization of budgeted interaction, and rigorous diagnostic attribution linking empirical observations directly to system architecture.
- Value: โญโญโญโญโญ Demonstrates that structured interaction control enables accessible 35B open-weights models to outperform leading frontier models, offering immense practical value for on-premise enterprise deployments.