When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/jaaack-wang/benchmarking-interative-visual-grounding
Area: Multimodal VLM
Keywords: Interactive Visual Grounding, Large Vision-Language Models, Information Seeking, Common Ground, Confidence Calibration
TL;DR¶
This paper introduces the first controlled benchmark for interactive visual grounding that disentangles static visual matching from proactive information seeking via four graded protocols (STATIC, FULL, PARTIAL, and NONE), revealing that frontier LVLMs lag significantly behind humans and fail catastrophically when forced to ground targets solely through self-directed dialogue.
Background & Motivation¶
Visual grounding serves as a cornerstone capability for multimodal artificial intelligence, requiring an embodied or conversational agent to accurately recognize and localize specific regions or objects in a visual environment based on natural language expressions. With the rapid advent of cross-modal pretraining and large vision-language models (LVLMs), models have achieved remarkable single-turn comprehension performance, nearing saturation on standard referring expression comprehension benchmarks. However, conventional grounding benchmarks universally rest upon a restrictive and idealized assumption: the model is presented upfront with an informative, unambiguous, and self-contained referring expression before grounding begins. Under this non-interactive formulation, grounding is reduced to static perceptual pattern matching, leaving unexamined whether a model can actively recognize descriptive incompleteness, seek clarification, or dynamically update its hypothesis as dialogue unfolds.
This static formulation departs fundamentally from the mechanics of real-world human communication. In natural human-to-human interaction, initial referring expressions are frequently underspecified, ambiguous, or only informative relative to a shared local context. Interlocutors collaboratively establish reference through clarification questions, conversational repair, and incremental feedback, progressively building common ground over time. Prior efforts that introduced interactionโsuch as goal-oriented guessing games or human-robot clarification dialoguesโdemonstrated the value of conversational reference but lacked systematic experimental control over the availability of target information. Consequently, when overall task accuracy drops, existing benchmarks cannot determine whether failure stems from inadequate visual matching or from broken interactive information seeking. This confounding obscures fundamentally different failure modes and provides little actionable diagnostic guidance for model development.
To bridge this critical gap, this paper introduces a controlled evaluation framework for interactive visual grounding across four human-grounded visual contexts. By formalizing an asymmetric reference game between a simulated Director and an LVLM Matcher, the framework systematically manipulates how much target information is provided upfront versus how much must be actively acquired through multi-turn dialogue. Core idea: formalize visual reference as a two-player asymmetric collaborative game with a four-stage graded information protocol (STATIC / FULL / PARTIAL / NONE), systematically disentangling and quantifying the interplay between visual matching, proactive information seeking, and meta-cognitive confidence calibration across 38,848 target selections.
Method¶
Overall Architecture¶
The proposed interactive visual grounding evaluation framework reformulates grounding as a collaborative, multi-turn referential game between two distinct participants: a Director and a Matcher. The Director is presented with an ordered sequence of \(K\) visual target items \(D = (d_1, \dots, d_K)\), while the Matcher is shown a candidate set \(M = (m_1, \dots, m_N)\) comprising \(N\) items (\(N \ge K\), containing distractors) with independently randomized spatial indices. The two participants operate under partial observability: neither can view the partner's display, and spatial coordinates, grid indices, and relative neighbor relationships are strictly excluded from natural language dialogue. The ultimate task objective is for the Matcher to output the correctly identified target sequence \(\hat{D} = (\hat{d}_1, \dots, \hat{d}_K)\) via conversational collaboration.
The pipeline comprises multimodal environment initialization, progressive protocol branching, and multi-turn decision-making with confidence calibration. For each target, the players exchange natural language utterances within a strict turn budget; the Matcher is empowered to pose clarifying questions, output candidate selections, and retrospectively repair prior choices upon uncovering new evidence.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Multimodal Input<br/>Director target items + Matcher candidate grid"] --> Des1["Asymmetric Referential Game Formulation<br/>Partial observability and multi-turn conversational exchange"]
Des1 --> Des2["Graded Information Availability Protocols<br/>STATIC / FULL / PARTIAL / NONE four-stage control ladder"]
Des2 --> Des3["Dynamic Self-Repair and Confidence Calibration<br/>Retrospective selection revision + 10-bin ECE assessment"]
Des3 --> Out["Evaluation Outputs<br/>Target selections + Accuracy, Question Rate, Calibration"]
Key Designs¶
1. Asymmetric Referential Game Formulation: Eliminating spatial shortcuts and enforcing semantic common ground Standard visual grounding setups often permit models to exploit layout priors or trivial spatial alignments. This framework enforces strict partial observability and visual asymmetry: the Director sees only the target items in their canonical sequence, whereas the Matcher observes a concatenated single-image grid where targets and distractors are randomly indexed and shuffled. Crucially, the system prompts strictly forbid any mention of grid indices, coordinates, row/column positions, or adjacent item references (allowing only internal structural descriptions of the items). Utterances are capped at 50 words per turn, with a maximum budget of 10 interaction turns per item. This setup strips away all positional crutches, forcing the models to rely exclusively on fine-grained visual-linguistic feature synthesis and collaborative dialogue pragmatics.
2. Graded Information Availability Protocols: Disentangling visual perception from proactive information seeking
To cleanly isolate perceptual shortcomings from interactive reasoning failures, the benchmark establishes four tiered communication protocols:
- STATIC: The Matcher receives a preconfigured, human-authored full description one target at a time and must select immediately without dialogue, measuring pure static visual matching.
- FULL: The Matcher receives the same informative description but is allowed optional follow-up questions, testing whether clarification can verify, refine, or repair initial impressions.
- PARTIAL: The Matcher receives only the least informative attribute extracted from the human description via an LLM-as-a-judge procedure (validated with 0.998 attribute support rate and 0.895 description coverage), testing whether the model can recognize descriptive underspecification and actively elicit discriminating details.
- NONE: No initial description is provided. The Director acts as a restricted oracle returning only "Yes", "No", or "Unclear", forcing the Matcher to formulate hypothesis-testing questions from scratch and construct an internal target representation.
These four protocols form an explicit capability gradient, exposing exactly where the grounding process breaks down across models.
3. Dynamic Self-Repair and Confidence Calibration: Diagnosing meta-cognitive awareness and error propagation Errors in multi-turn interaction compound if an agent lacks the ability to reconsider early commitments. The framework introduces a formal tracking mechanism for "Self-Repairs": in any interactive turn under FULL or PARTIAL, the Matcher can revise an earlier target selection if later dialogue evidence invalidates its previous decision. Furthermore, whenever the Matcher selects a target item, it must provide an explicit confidence estimate \(c_j \in [0, 100]\). The evaluation partitions these estimates into 10 uniformly spaced bins \(B_\ell \in [0, 10), \dots, [90, 100]\) to construct calibration curves and compute the Expected Calibration Error (ECE): $\(ECE = \sum_{\ell=1}^{10} \frac{|B_\ell|}{N} \left| \text{acc}(B_\ell) - \text{conf}(B_\ell) \right|\)$ This joint behavioral and meta-cognitive tracking reveals whether LVLMs possess genuine awareness of uncertainty or merely suffer from ungrounded overconfidence.
Interactive Visual Grounding and Multi-Turn Evaluation Protocol¶
The benchmark is structured across rigorously filtered human reference datasets, simulated interlocutors, and unified evaluation metrics:
1. Human-Grounded Visual Contexts: Four diverse image sets originating from cognitive communication studies are curated:
- Dogs: 13 candidates (10 targets, 3 distractors), spoken dialogue transcripts, real photography, highly nameable items.
- Baskets 1: 13 candidates (10 targets, 3 distractors), spoken dialogue transcripts, real woven objects, weakly nameable fine-grained textures.
- Baskets 2: 18 candidates (12 targets, 6 distractors), text-based dialogue transcripts, real woven objects, weakly nameable textures.
- Tangrams: 12 candidates (12 targets, 0 distractors), text-based dialogue transcripts, abstract geometric silhouettes, highly variable human conceptualizations.
Dialogues are filtered to retain only participants achieving \(\ge 80\%\) (Baskets 2) or \(\ge 90\%\) (Tangrams) initial accuracy without inter-round performance drops, yielding 48 complete dialogues and 536 canonical item-description pairs.
2. Simulation Protocol:
- The Director is simulated by a fixed frontier model (GPT-5.4, achieving ~85% initial accuracy with human matchers), ensuring comparable information delivery across conditions.
- The Matcher is instantiated by 8 leading LVLMs across proprietary and open-weight families, interacting strictly via structured JSON schemas (utterance, selections containing nth_item, target_item_index, and confidence).
- Matchers must withhold target selection while seeking clarification; selecting an item signals readiness to advance. A forced selection is triggered if the 10-turn budget expires.
3. Evaluation Metrics: Primary metrics comprise Accuracy (%) (percentage of correctly matched target items per round; random baseline is \(1/N\)) and Question Rate (average clarification questions asked per item; 0 under STATIC by definition). Secondary metrics track calibration alignment curves and self-repair trajectories (frequency, successful repairs, error-inducing revisions, and confidence drift).
Key Experimental Results¶
Main Results¶
The primary benchmark evaluates 8 state-of-the-art LVLMs across all four interaction protocols, totaling 17,152 main target decisions (expanding to 38,848 across follow-up studies).
The table below presents micro-averaged accuracy (%) across the four datasets alongside paired mean differences relative to the STATIC baseline:
| Model | STATIC (Baseline) | FULL (Clarification) | PARTIAL (Recovery) | NONE (Question-Driven) |
|---|---|---|---|---|
| Gemini-3.1-Pro | 87.5 | 89.9 (+2.9) | 89.0 (+2.3) | 56.7 (-28.6*) |
| GPT-5.5 | 65.8 | 69.5 (+3.6) | 71.4 (+5.5) | 59.7 (-6.1) |
| GPT-5.4 | 66.1 | 69.4 (+3.4*) | 65.5 (-0.4) | 43.2 (-23.6*) |
| Gemini-3.1-Flash | 67.8 | 65.3 (-2.7) | 54.5 (-12.9*) | 37.9 (-30.4**) |
| Gemma-4-31B-IT | 56.0 | 60.0 (+4.3) | 58.4 (+2.6) | 26.4 (-29.0**) |
| Qwen3-VL-32B | 48.8 | 50.5 (+2.8) | 44.0 (-4.0) | 23.9 (-25.5*) |
| GPT-5.4-Mini | 47.9 | 49.0 (+0.6) | 37.1 (-11.5*) | 13.5 (-35.2*) |
| Qwen3-VL-8B | 33.6 | 35.0 (+1.6) | 35.3 (+1.2) | 21.6 (-12.7*) |
LVLM-Human Performance Gap¶
To quantify how far current models lag behind human conversational proficiency, the following table details the paired mean accuracy gap (%) between each model's best-performing protocol and task-level human matchers (negative values indicate deficits; asterisks denote significance from paired t-tests):
| Model | Dogs (10/13) | Baskets 1 (10/13) | Baskets 2 (12/18) | Tangrams (12/12) | All Datasets |
|---|---|---|---|---|---|
| Gemini-3.1-Pro | -7.0* | 0.0 (Parity) | -8.3* | -6.0 | -6.8** |
| GPT-5.5 | -14.0** | -15.0** | -18.5*** | -40.5*** | -25.3*** |
| GPT-5.4 | -26.0*** | -16.0*** | -20.8*** | -42.9*** | -27.3*** |
| Gemini-3.1-Flash | -18.0*** | -23.0** | -22.6*** | -47.0*** | -28.9*** |
| Gemma-4-31B-IT | -37.0*** | -31.0*** | -27.4*** | -44.6*** | -36.7*** |
| Qwen3-VL-32B | -25.0*** | -26.0** | -41.7*** | -72.6*** | -46.2*** |
| GPT-5.4-Mini | -40.0*** | -30.0*** | -32.1*** | -70.8*** | -47.7*** |
| Qwen3-VL-8B | -42.0*** | -54.0*** | -61.9*** | -72.0*** | -61.4*** |
Ablations and Robustness Studies¶
- Test-Time Reasoning Scaling (Tangrams):
- On STATIC, increasing test-time reasoning yields no noticeable benefit for GPT-5.4 (50.6% at None \(\to\) 52.4% at Medium \(\to\) 48.8% at High).
- On FULL, reasoning compute significantly enhances dialogue utility: GPT-5.4 jumps from 52.4% (None) to 60.7% (+8.3 at Medium) and 61.3% (+8.9 at High). GPT-5.4-Mini improves from 17.3% to 30.4% (+13.1) and 31.0% (+13.7).
-
Gains plateau rapidly between Medium and High effort, indicating that test-time search alone cannot overcome grounding bottlenecks.
-
Self-Repair Dynamics and Pathological Overconfidence:
- Revisions are rare, occurring in less than 2% of trials (49 total across all experiments, strictly limited to FULL and PARTIAL; 0 under NONE).
- Among the 49 repairs, 53.1% (26) successfully flipped an incorrect choice to correct, 10.2% (5) flipped a correct choice to incorrect, and 36.7% (18) replaced one incorrect answer with another.
-
While Gemini-3.1-Pro displayed high repair precision (13 of 17 repairs succeeded, 76.5%), GPT-5.4 corrected 0 out of 6 revisions (0.0% success rate) while paradoxically boosting its stated confidence by an average of +11.2 points.
-
Alternative Director and Domain Transfer:
- Swapping the Director to Gemini-3.1-Flash across Basket 2 and Tangrams changed mean Matcher accuracy by only -1.1% in FULL and -3.5% in PARTIAL, maintaining near-identical relative Matcher rankings (Spearman \(\rho = 0.929\) and \(0.976\)).
- Evaluated on 6 fine-grained FACES sets (12 facial expressions per identity), the protocol-level performance hierarchy remained perfectly consistent: STATIC (48.7%) \(\approx\) FULL (47.8%) > PARTIAL (41.9%) > NONE (30.7%).
Key Findings¶
- Clarification helps, but cannot substitute for initial guidance: Optional clarification (FULL) improves upon static matching (STATIC) across 7 of 8 models. However, when initial descriptions are entirely removed (NONE), all models collapse significantly below STATIC (e.g., GPT-5.4 drops by -23.6%, Mini by -35.2%), despite asking the highest number of questions.
- Question volume does not equal conversational grounding competence: Models like GPT-5.4-Mini rarely ask questions and suffer severely under underspecification. Conversely, asking many questions is insufficient: Gemini-3.1-Pro poses more questions than GPT-5.5 under NONE, yet GPT-5.5 achieves higher accuracy (59.7% vs. 56.7%) due to superior evidence synthesis across turns.
- Pervasive overconfidence and meta-cognitive miscalibration: Although models adjust confidence downward as information scarcity increases (NONE < PARTIAL < FULL), empirical accuracy consistently lags behind stated confidence, plotting well below the ideal calibration diagonal.
Highlights & Insights¶
- Clean Cognitive Factorization: By decomposing grounding across four explicit information regimes, this work decisively decouples visual perceptual alignment from communicative pragmatics and multi-turn inquiry.
- Illuminating the Fragility of Self-Repair: The discovery that models like GPT-5.4 can fail 100% of the time during self-repair while increasing confidence provides crucial empirical caution against naive multi-agent reflection loops.
- Standardized Human-Grounded Testbed: Repurposing gold-standard cognitive science datasets into structured vision-language benchmarks provides a lasting resource for evaluating collaborative embodied agents.
Limitations & Future Work¶
- Simulated Dialogue Partners: While Director substitutions proved ranking stability, LLM Directors lack the full multimodal subtlety (e.g., gaze, hesitation, tone) of human conversational partners.
- Isolated Object Settings: The evaluation operates on object-level grid arrays rather than complex 3D cluttered indoor scenes or dynamic video environments.
- Future Directions: Extending protocols to open-ended conversational responses in the NONE regime, incorporating reinforcement learning with verification rewards (RLVR) for question generation, and deploying policies on embodied physical platforms.
Related Work & Insights¶
- vs. Static Visual Grounding (RefCOCO, Flickr30k Entities, Ref-Adv): Traditional benchmarks evaluate single-turn pattern matching from complete descriptions; this work studies dynamic reference under communicative uncertainty and progressive clarification.
- vs. Interactive Grounding Games (GuessWhat?!, INGRESS, InViG): Prior interactive setups focused on single-target identification with unconstrained information delivery; this benchmark introduces multi-target matching under controlled information gradients.
- vs. Multimodal Multi-Turn Benchmarks (MMDU, MMRC, Chatterbox): While existing multi-turn benchmarks focus on general instruction following or dialogue comprehension, this work targets collaborative reference resolution and common ground formation.
Rating¶
- Novelty: โญโญโญโญโญ (Pioneering benchmark disentangling perceptual matching from proactive clarification)
- Experimental Thoroughness: โญโญโญโญโญ (8 leading LVLMs, 38,848 selections, extensive ablations on reasoning, directors, and face domains)
- Writing Quality: โญโญโญโญโญ (Exemplary clarity, mathematically disciplined definitions, and tight integration with cognitive linguistics)
- Value: โญโญโญโญโญ (Essential benchmark for developing reliable, self-aware conversational vision agents)