Don’t Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/zhangrongyu0101/VagueBench
Area: LLM Reasoning
Keywords: Proactive Elicitation, Dialogue State DAG, Group Relative Policy Optimization, VagueBench, Multi-Turn Interaction
TL;DR¶
Addressing the pervasive habit of LLMs silently guessing unstated user needs, this paper introduces VagueBench with known hidden information sets and proposes DialogGRPO, an offline RL framework that formulates multi-turn elicitation as a dialogue-state DAG with turn-level dense rewards, achieving >88% proactive asking rate without requiring runtime user simulators.
Background & Motivation¶
Contemporary large language models default to answering rather than asking, silently hallucinating unstated intents and constraints when presented with underspecified user prompts. In everyday human communication, users routinely leave vital parameters implicit due to the cognitive phenomenon known as the curse of knowledge, presuming that the conversational assistant already possesses the unexpressed context. An empirical audit of 5,000 real-world queries from a commercial AI assistant reveals that more than half of user requests lack essential information; however, current state-of-the-art models spanning from 8B parameters to frontier systems ask clarifying questions in fewer than 11% of such ambiguous scenarios, with model scaling yielding no discernible mitigation.
The fundamental tension driving this passive response pathology lies in the structural mismatch between user-side vagueness and model-side post-training incentives. User ambiguity naturally manifests across two distinct tiers: intent ambiguity (where high-level objectives are unclear, such as "work on this article") and detail absence (where the goal is definite but operational constraints like budget or scope are missing). Conversely, prevailing alignment pipelines dominated by single-turn supervised fine-tuning and immediate terminal answer rewards penalize models for pausing to query the user. Existing clarification benchmarks remain largely restricted to rigid domain-specific slot schemas or multiple-choice word disambiguation, lacking naturalistic open-domain queries grounded in known, quantifiable hidden information sets; furthermore, training methods heavily rely on unstable online user simulators or sparse trajectory-level returns, failing to provide fine-grained step-level credit assignment.
To break the model habit of premature guessing and foster proactive clarification, this field requires both an audit benchmark with strictly defined missing information and a credit-assignment mechanism capable of offline optimization. The core idea of this paper is to deconstruct real-world queries into intent and detail clarity anchors to construct the controlled-degradation benchmark VagueBench, while organizing multi-turn elicitation into a dialogue-state directed acyclic graph (DAG) trained via DialogGRPO using group-relative advantage estimation and dense turn rewards without runtime user simulation.
Method¶
Overall Architecture¶
DialogGRPO resolves the twin bottlenecks of multi-turn clarification training: sparse trajectory rewards failing to pinpoint effective intermediate questions, and on-policy rollouts incurring prohibitive latency and cost from runtime user simulation. The pipeline consists of three sequential offline stages: first, an external strong LLM and a verified simulator expand candidate queries over real seed queries to construct a compact Dialogue State DAG where nodes represent identical sets of acquired information; second, each transition edge receives a dense turn-level reward blending immediate information gain with Monte Carlo lookahead returns estimated directly within the precomputed graph; third, candidate questions departing from each node form a natural GRPO comparison group, allowing the policy to be optimized via advantage-weighted regression over offline tuples without invoking an interactive simulator during parameter updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Underspecified Query q0<br/>Intent Ambiguity / Detail Absence"] --> B["Dialogue State DAG Construction<br/>Acquired Set Representation & Node Merging"]
B --> C["Dense Turn-Level Reward Modeling<br/>Immediate Information Gain + Lookahead Value"]
C --> D["Group-Relative Advantage Training<br/>Local Candidate Comparison & Offline Update"]
D --> E["Proactive Elicitation Policy πθ<br/>Timely Inquiring·Targeted Recovery·Tailored Answering"]
Key Designs¶
1. Dialogue State DAG Construction: State Representation and Equivalent Node Merging Traditional tree-based expansion in multi-turn dialogues suffers from combinatorial explosion and fragments reward signals across identical information states achieved in different conversational orders. DialogGRPO formalizes the dialogue progression as a directed acyclic graph \(G=(V, E)\) over the target query's total information units \(U^\star = \mathcal{T} \cup \mathcal{D}\). Each node \(v\) represents a distinct subset of acquired units \(K_v \subseteq U^\star\), initialized at the root with units present in the raw query \(q_0\). An external teacher LLM (GLM-5.1, serving strictly as an exploration engine rather than a distillation target) proposes \(N\) diverse candidate questions conditioned on the dialogue history and the remaining hidden units \(U^\star \setminus K_v\). When a candidate question hits hidden units via a simulated user \(S\), the transition leads to a child node \(v'\) with updated information \(K_{v'} = K_v \cup \Delta_e\). Crucially, state merging collapses all nodes that share the exact same acquired set \(K_v\) into a single canonical DAG node, retaining the shortest-path trajectory as its canonical prompt \(p_v\). This consolidation drastically reduces redundant state expansions and enables subsequent advantage computations to be shared across converging trajectories.
2. Dense Turn-Level Reward Modeling: Decoupling Immediate Gain from Lookahead Value Sparse endpoint rewards cannot differentiate whether an intermediate question was sharp and productive or redundant and uninformative. To address this, DialogGRPO decomposes the trajectory-level objective (answer coverage F1 penalized by conversational length) into a dense, per-edge reward \(R_{\text{turn}}(e)\). This reward synthesizes two complementary terms: immediate information gain \(R_{\text{imm}}(e) = \alpha \min(|\Delta_e|, \delta_{\max})\) that credits queries uncovering more hidden units while capping outliers to preserve discriminative variance, and lookahead value \(Q_{\text{la}}(e)\) estimated by averaging returns over \(K\) Monte Carlo paths sampled from the child state \(v_e\) inside the pre-built DAG: $\(Q_{\text{la}}(e) = \frac{1}{K} \sum_{k=1}^K R_{\text{traj}}(\tau_k \mid v_e)\)$ The unified turn reward is formulated as \(R_{\text{turn}}(e) = Q_{\text{la}}(e) + R_{\text{imm}}(e)\), whereas null-progress questions that waste a turn without acquiring new units receive an explicit penalty \(R_{\text{turn}} = \bar{Q}_{\text{la}}(v) - \lambda\). This dual structure incentivizes the model to elicit multiple critical constraints per turn while steering towards terminal dialogue states that maximize downstream answer accuracy.
3. Group-Relative Advantage Training: Simulator-Free Offline Policy Optimization To bypass the computational overhead of invoking interactive user simulators at every gradient step, DialogGRPO treats the outgoing edge set \(G_v = \{e : \text{src}(e) = v\}\) of any non-terminal DAG node as a self-contained GRPO comparison group. The relative advantage for candidate response \(y_e\) is computed via intra-group normalization: $\(\hat{A}_e = \frac{R_{\text{turn}}(e) - \mu_v}{\sigma_v + \epsilon}\)$ where \(\mu_v\) and \(\sigma_v\) denote the mean and standard deviation of \(R_{\text{turn}}\) within group \(G_v\). The policy model \(\pi_\theta\) is then optimized offline using the advantage-weighted objective: $\(\mathcal{J}(\theta) = \mathbb{E}_{v} \left[ \frac{1}{|G_v|} \sum_{e \in G_v} \exp\left(\frac{\hat{A}_e}{\beta}\right) \log \pi_\theta(y_e \mid p_v) \right]\)$ Derived from KL-regularized policy optimization, the KL penalty against the reference model is analytically absorbed into the exponential advantage weight \(\exp(\hat{A}_e / \beta)\). This eliminates the need for importance-sampling clipping or online value critics, enabling scalable, stable parameter optimization directly on pre-extracted training tuples.
Loss & Training¶
The framework is instantiated on Qwen3.5-27B (primary backbone), Gemma3-27B, and Llama-3.1-8B-Instruct using LoRA adapters (\(r=16, \alpha=32\)). Optimization employs a cosine learning rate scheduler with an initial learning rate of \(10^{-4}\), warm-up ratio of 0.1, batch size of 48, and runs for 3 epochs on a single node equipped with 8 NVIDIA H200 GPUs. The dialogue termination depth is bounded at \(T_{\max}=10\). Node terminality is determined by an LLM-based answerability classifier evaluating whether \(K_v\) suffices for a complete answer. The entire parameter-update phase proceeds completely offline without runtime interaction.
Key Experimental Results¶
Main Results¶
Evaluated on the 750 instances of the VagueBench test set (250 seeds decomposed into Complete, Intent-Vague, and Fully-Vague splits), the primary metrics include decision appropriateness (Ask%: percentage of instances where the model opts to clarify on Turn 1), final answer coverage (Cov. F1: F1 score between final response and gold information units), and information acquisition rate (IAR: average hidden units recovered per asking turn).
| Model / Method | Intent-Vague Ask% (↑) | Intent-Vague Cov. F1 (↑) | Intent-Vague IAR (↑) | Fully-Vague Ask% (↑) | Fully-Vague Cov. F1 (↑) | Fully-Vague IAR (↑) |
|---|---|---|---|---|---|---|
| Closed-source Frontier | ||||||
| GPT-5.4 | 1.6 | 0.58 | 0.06 | 10.8 | 0.53 | 0.65 |
| Claude Sonnet 4.6 | 8.8 | 0.60 | 0.08 | 8.4 | 0.52 | 0.56 |
| Gemini 3.1 Pro | 2.4 | 0.57 | 0.00 | 3.6 | 0.54 | 0.80 |
| Open-source Base | ||||||
| Qwen3-8B | 1.2 | 0.47 | 0.00 | 1.6 | 0.36 | 0.21 |
| Llama-3.1-8B-Instruct | 0.8 | 0.50 | 0.00 | 1.2 | 0.35 | 0.23 |
| Qwen3.5-27B | 1.6 | 0.49 | 0.08 | 2.0 | 0.38 | 0.37 |
| GPT-oss-120B | 2.0 | 0.46 | 0.02 | 3.6 | 0.36 | 0.35 |
| Qwen3.5-27B Alignment | ||||||
| + Prompt | 38.4 | 0.52 | 0.22 | 47.2 | 0.50 | 0.38 |
| + UoT | 58.8 | 0.50 | 0.45 | 62.4 | 0.49 | 0.41 |
| + SFT (Ours) | 82.0 | 0.60 | 0.64 | 85.6 | 0.63 | 0.80 |
| + DialogGRPO (Ours) | 88.4 | 0.64 | 0.76 | 91.2 | 0.67 | 0.87 |
On information-complete seed queries, where models should answer directly without unnecessary questions (lower Ask% is better), DialogGRPO exhibits superior calibration: while Prompting and UoT excessively over-ask (Ask% surging to 42.8% and 56.4% respectively), DialogGRPO maintains a restrained 5.2% Ask% (close to base 1.6%) while lifting answer Cov. F1 from 0.58 to 0.74.
Ablation Study¶
The ablation experiment conducted on VagueBench validates the critical complementarity of immediate and lookahead reward signals:
| Configuration | Intent-Vague Cov. F1 | Intent-Vague IAR | Fully-Vague Cov. F1 | Fully-Vague IAR | Note |
|---|---|---|---|---|---|
| Full Method (DialogGRPO) | 0.64 | 0.76 | 0.67 | 0.87 | Full dual-signal reward formulation |
| w/o lookahead reward (\(Q_{\text{la}}\)) | 0.58 | 0.74 | 0.60 | 0.84 | Cov. F1 drops by 0.06–0.07; myopic greedy questioning |
| w/o immediate reward (\(R_{\text{imm}}\)) | 0.60 | 0.68 | 0.66 | 0.85 | IAR drops by up to 0.08; loses multi-unit question packing |
Key Findings¶
- Pervasive Clarification Suppression Across Scales: Without targeted training, all base and proprietary LLMs ask questions in fewer than 11% of underspecified cases, demonstrating that scaling model size alone does not overcome the alignment bias toward premature direct answers.
- Prompting Awakes Latent Ability but Hits Efficiency Ceiling: Elicitation-encouraging system prompts activate clarifying behavior (Ask% reaches 38.4%–47.2%) but cannot guide models to ask high-value questions (IAR languishes at 0.22–0.38) and causes catastrophic over-asking on complete inputs (42.8%). In contrast, DialogGRPO suppresses questions naturally when group advantages are uniformly low.
- Orthogonal Reward Dynamics: The ablation proves that lookahead value governs final answer coverage (Cov. F1), whereas immediate reward drives per-turn elicitation efficiency (IAR).
- Strong Zero-Shot Transfer: DialogGRPO trained on VagueBench successfully transfers to out-of-distribution benchmarks without retraining: on MultiWOZ 2.4, task completion increases from 11.4% to 24.7%; on AmbigQA, clarification Ask F1 jumps from 2.1% to 26.3%, while preserving general reasoning capabilities across GSM8K, MATH-500, MMLU, and LiveCodeBench.
Highlights & Insights¶
- Formulating Multi-Turn Dialogues as Graph-Based GRPO Groups: Converting multi-turn conversational trajectories into an information-state DAG enables natural group-relative advantage estimation, allowing offline training to achieve dense credit assignment without simulator rollouts.
- Bilingual Grounding of Clarity Anchors in VagueBench: Grounded in linguistic argument-structure and speech-act theories, the benchmark separates predicate/speech-act anchors (intent layer) from output-type/scope anchors (detail layer) to generate rigorously controlled, measurable information gaps.
- Analytical Absorption of KL Divergence in Advantage Weights: Deriving the optimization objective directly from KL-regularized policy maximization results in an elegant closed-form advantage-weighted regression that completely circumvents PPO clipping instabilities.
Limitations & Future Work¶
- Cooperative Simulator Assumption: The evaluation and DAG construction assume a cooperative user simulator that truthfully reveals information upon matching prompts, bypassing complex human phenomena like contradictory responses, sudden topic pivots, or refusal to disclose details.
- Coverage Constraints in Open Preference Spaces: While DAG state merging effectively aggregates discrete constraints, unbounded subjective domains (e.g., dynamic taste shifts in shopping) remain challenging to fully model via pre-indexed state graphs.
- Monolingual Commercial Origin: The benchmark derives from English-only assistant traffic, requiring future expansion into multi-lingual, multi-modal settings, as well as guardrails preventing models from overly soliciting private personal data.
Related Work & Insights¶
- vs. ClarifyGPT / STaR-GATE / CollabLLM: Previous approaches either operate on narrow synthetic domains or rely on online user simulation and coarse trajectory-level rewards; DialogGRPO pioneers offline dense credit assignment on real open-domain queries.
- vs. Conversational Search (ClariQ / Qulac): Traditional search clarification resolves ambiguities among known document topics, whereas VagueBench addresses open-ended requests where underlying goals and parameters were never stated.
- Insights: The DAG-based offline advantage estimation paradigm can be seamlessly extended to tool-use agents and proactive collaborative coding assistants to refine underspecified software specifications.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates multi-turn elicitation as a dialogue-state DAG, enabling offline group-relative advantage optimization without runtime simulation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 7 LLMs, complete query over-asking calibration, detailed dual-reward ablations, and zero-shot OOD transfers.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous linguistic deconstruction, elegant mathematical derivations, and crystal-clear presentation.
- Value: ⭐⭐⭐⭐⭐ Solves the critical real-world failure mode of LLMs blindly hallucinating answers to vague prompts, offering high practical utility.