Skip to content

InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/maevehutch/insight
Area: LLM Agent
Keywords: Interactive Visualization Verification, Multimodal Agents, Sequential Evidence Acquisition, Interaction Efficiency Score, Interaction Trace Auditing

TL;DR

InSight introduces an ecologically valid benchmark of 21,349 analytical claims grounded in interactive, multi-view Vega-Lite environments, requiring multimodal agents to actively uncover occluded visual evidence via pure pixel inputs and proposing the Interaction Efficiency Score (IES) to penalize passive perception and ungrounded confabulation.

Background & Motivation

Vision-Language Models (VLMs) have exhibited remarkable perceptual capabilities across static chart comprehension and one-shot visual question answering. However, authentic real-world data analysis is fundamentally dynamic, exploratory, and interactive. Analysts routinely explore complex datasets within faceted browsers and interactive multi-view dashboards where critical data values are not presented upfront on a flat visual canvas. Instead, essential evidence is conditionally revealed through user agencyโ€”concealed behind hover-triggered tooltips, distributed across linked coordinated views, or accessible solely through zooming, panning, and interval filtering. Existing visualization benchmarks predominantly evaluate models over single-view static chart images paired with templated question-answer pairs or summarization tasks, completely bypassing the sequential decision-making and epistemic requirements of real-world exploratory data workflows.

This discrepancy between perceived fluency and genuine visual grounding is increasingly alarming. Recent empirical findings reveal that state-of-the-art VLMs consistently lack foundational visualization literacy, often failing to ground their answers in actual graphical encodings when confronted with modified or decontextualized charts. Instead, models default to confabulating answers based on linguistic plausibility or parametric training priors. Furthermore, conventional fact-checking and static visual QA paradigms evaluate only final task accuracy, treating internal reasoning as an unobservable black box or relying on post-hoc, untrustworthy chain-of-thought explanations. There has been no mechanism to audit whether a model actively engaged with the necessary visual evidence required to justify its verdict.

To bridge this critical divide, this work reframes visual reasoning over visualizations as a sequential decision-making process under partial observability. Core idea: formalize visual claim verification as an agentic, sequential evidence acquisition task over interactive multi-view environments, requiring agents to actively explore occluded visual evidence via mouse and navigation primitives to assign True/False/NEI labels, and audit the reasoning fidelity of action trajectories using a novel Interaction Efficiency Score (IES) that penalizes ungrounded parametric guessing.

Method

Overall Architecture

InSight operationalizes interactive visual claim verification as an agentic decision process over a partially observable environment \(E\). The environment consists of fully functional, web-based analytical notebooks authored in Vega-Lite, featuring an average of 4.5 views and 3.7 distinct visualization specifications per notebook. Because the rendered page size deliberately exceeds the fixed browser viewport, the initial observation exposes only a partial slice of the multi-view layout, rendering every evaluation claim impossible to verify from the initial screenshot alone. Crucially, the web page is treated strictly as a pure visual surface: the agent receives only the RGB screenshot of the current viewport \(s_t\) alongside the textual claim \(c\), without any access to the underlying DOM tree, HTML source code, or accessibility tree.

At each interaction turn \(t \le T_{\max}\), the agent selects an action \(a_t\) from a standardized high-level action space \(A\), which encompasses navigation controls (pixel scrolling and directional paging), mouse interactions (hover, click, shift-click, and drag), and a termination action answer(label) where \(\text{label} \in \{\text{True}, \text{False}, \text{NEI}\}\). The browser environment executes \(a_t\), updates its visual state (e.g., exposing a tooltip with exact numerical readings or scrolling to an off-screen chart), and returns the updated screenshot \(s_{t+1}\). The interaction loop terminates when the agent submits its verdict or exhausts its action budget \(T_{\max}\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Analytical Claim + Initial Viewport Screenshot"] --> B["Multi-Stage Analytical Claim Construction<br/>Cross-run consensus extraction -> atomic decomposition -> controlled mutation"]
    B --> C["Pure-Visual Environment & Action Space<br/>Headless Chromium via Playwright, strictly RGB without DOM access"]
    C --> D["Sequential Evidence Acquisition Loop<br/>Active navigation & hover interaction under turn budget Tmax"]
    D --> E["Interaction Efficiency Score IES<br/>Joint evaluation of answer accuracy and State Change Ratio SCR"]

Key Designs

1. Multi-Stage Analytical Claim Construction: Grounding claims in authentic human narratives with controlled semantic mutations

Rather than relying on templated questions or synthetic charts, InSight derives its dataset from 297 capstone project notebooks authored by analysts formally trained in data visualization. The construction pipeline executes across four distinct stages: First, candidate text spans corresponding to Lundgard and Satyanarayan's semantic levels 2 (statistical/numerical insights) and 3 (visual/perceptual patterns) are extracted via Gemini 2.5 Flash across three independent runs and consolidated using a word trigram F1 overlap matching criterion, retaining tokens with at least two-run consensus to eliminate hallucinations. Second, compound statements are decomposed into atomic, fully decontextualized propositions with resolved references (e.g., pronouns and implicit comparative subjects) and verified via an NLI entailment check, forming the ground-truth True set. Third, contrastive False and Not Enough Information (NEI) claims are generated via minimal, semantics-preserving perturbations: antonym substitution targets scalar and directional language; in-lexicon argument substitution replaces categorical values or numeric figures with valid alternatives from the same dataset; out-of-lexicon argument substitution replaces entities with unobserved attributes or values to create NEI claims. Every mutated claim is validated using a pretrained DeBERTaV3 NLI model requiring bidirectional contradiction \(> 0.9\) for False claims and both entailment and contradiction \(< 0.2\) for NEI claims. Fourth, human validation with 13 expert annotators across 294 sampled claims achieved 81.3% raw agreement and a Bayesian Dawid-Skene MAP accuracy of 77.6%, confirming the high reliability of the ground-truth labels.

2. Pure-Visual Environment & Action Space: Eliminating structural text leakage and enforcing realistic GUI interaction

To ensure models genuinely reason over visual representations rather than shortcutting via underlying tabular data embedded in HTML or JSON code, InSight executes environments in headless Chromium via Playwright, providing only RGB pixel screenshots. The agent must predict exact continuous pixel coordinates \((x, y)\) for interactions. The defined action space \(A\) spans navigation commands (scroll(dx, dy), page_up(n), page_down(n), arrow_right(n)), mouse interactions (hover(x, y) for tooltip inspection, click(x, y) for point selection, shift_click(x, y) for multi-selection, drag((x1, y1), (x2, y2)) for interval filtering), and terminal submission (answer(label)).

3. Sequential Evidence Acquisition Loop: Hypothesis testing and evidence gathering across multi-view topologies

Because visualizations in InSight are multi-view and inherently dynamic, static perception is guaranteed to fail. The agent must operate as an active investigator: it must formulate exploratory hypotheses, scroll across the page canvas to locate the specific visual view pertinent to the claim, and trigger hover states over graphical marks to inspect exact data readings. Crucially, verifying an NEI claim requires an exhaustive traversal establishing that no visual evidence exists, whereas verifying a True claim demands pinpoint evidence confirmation, and False claims necessitate uncovering contradictory evidence that disproves a linguistically plausible statement.

4. Interaction Efficiency Score (IES): Penalizing passive guesses and auditing active visual engagement

Standard agentic benchmarks measure task success purely based on final goal-state reachability, allowing lucky guesses or parametric biases to masquerade as robust reasoning. InSight introduces the Interaction Efficiency Score (IES) to explicitly couple answer accuracy with interaction quality. For an episode \(i\), let \(A_i \in \{0, 1\}\) represent accuracy, \(T_i\) represent total actions executed, and \(E_i\) represent effective actions that induce an observable visual state change: $\(E_i = \sum_{t=1}^{T_i} \mathbf{1}[s_{t+1} \neq s_t]\)$ The per-episode and dataset-level IES metrics are formulated as: $\(\text{IES}_i = A_i \cdot \frac{E_i}{T_i}, \quad \text{IES} = \frac{1}{N} \sum_{i=1}^N A_i \cdot \frac{E_i}{T_i}\)$ where \(E_i / T_i\) denotes the State Change Ratio (SCR). Critically, an agent that submits an immediate answer at turn 1 without executing any exploratory action has \(T_i = 0\) and \(E_i = 0\); the metric explicitly assigns \(E_i / T_i = 0\), yielding \(\text{IES} = 0\) regardless of whether the prediction was correct. This mathematical formulation penalizes passive perception and ungrounded confabulation while rewarding agents whose interaction trajectories actively acquire visual evidence.

Key Experimental Results

Main Results

The primary benchmark evaluation was conducted on a stratified subset of 500 claims (167 True, 167 False, 166 NEI) across 14 model configurations from four prominent families at the standard action budget of \(T_{\max} = 10\):

Model \(T_{\max}\) Actions/Ep \(T/N\) Overall Accuracy \(\text{Acc}_{\text{False}}\) \(\text{Acc}_{\text{NEI}}\) \(\text{Acc}_{\text{True}}\) Effective Ratio \(E/T\) Overall IES
GPT-5.5 10 3.13 57.2% 59.6% 59.0% 53.0% 45.4% 26.98%
Gemma 4 31B 10 3.16 47.2% 40.4% 63.9% 37.5% 43.8% 20.57%
Qwen 3.5 27B 10 1.57 45.0% 38.0% 66.3% 31.0% 17.7% 7.57%
Qwen 3.6 35B 10 1.78 44.8% 42.8% 62.7% 29.2% 21.9% 9.07%
Gemini 3.5 Flash 10 6.27 41.6% 45.8% 45.8% 33.3% 61.3% 24.93%
Gemma 4 E4B 10 0.89 43.2% 18.7% 88.0% 23.2% 18.3% 6.87%
Qwen 3.5 9B 10 0.50 38.0% 50.6% 26.5% 36.9% 9.56% 3.52%
Qwen 3.5 2B 10 0.24 37.0% 30.7% 51.8% 28.6% 4.10% 1.17%
Qwen 3.5 0.8B 10 0.00 34.0% 21.7% 54.8% 25.6% 0.00% 0.00%
Gemma 4 E2B 10 0.16 31.2% 7.8% 71.7% 14.3% 5.00% 0.90%

Ablation Study

The interaction budget ablation examines the effect of removing interaction entirely (\(T_{\max} = 1\), representing a static visual question answering baseline) versus expanding the interaction budget to \(T_{\max} = 25\):

Model Config Budget \(T_{\max}\) Actions/Ep \(T/N\) Overall Accuracy \(\text{Acc}_{\text{False}}\) \(\text{Acc}_{\text{NEI}}\) \(\text{Acc}_{\text{True}}\) Effective Ratio \(E/T\) Overall IES Note
Gemini 3.5 Flash 25 9.33 50.0% 54.8% 50.6% 44.6% 56.7% 31.00% Highest accuracy and IES when allowed full exploration
Gemini 3.5 Flash 10 6.27 41.6% 45.8% 45.8% 33.3% 61.3% 24.93% Forced cutoff mid-exploration drops accuracy below baseline
Gemini 3.5 Flash 1 - 44.2% 33.1% 73.5% 26.2% - - Zero interaction baseline, heavily biased toward NEI guesses
Qwen 3.5 27B 10 1.57 45.0% 38.0% 66.3% 31.0% 17.7% 7.57% Reluctant interaction with low effective ratio
Qwen 3.5 27B 1 - 46.2% 35.5% 79.5% 23.8% - - Static baseline, dominates via NEI prior
Gemma 4 31B 10 3.16 47.2% 40.4% 63.9% 37.5% 43.8% 20.57% Substantial gains across True/False when interacting
Gemma 4 31B 1 - 45.4% 26.5% 84.9% 25.0% - - Without interaction, falsification collapses to 26.5%

Key Findings

  • Interactive verification is far from solved: Even the frontier model GPT-5.5 achieves only 57.2% overall accuracy, far below human expert reliability (Bayesian MAP estimate 77.6%), while small models barely exceed random chance (33.3%).
  • Falsification is uniquely interaction-critical: Under the no-interaction baseline (\(T_{\max} = 1\)), models disproportionately default to predicting NEI (e.g., Gemma 4 31B achieves 84.9% on NEI but only 26.5% on False). Active navigation enables agents to uncover contradictory visual evidence, raising Gemma 4 31B's falsification accuracy to 40.4%.
  • Exploration budget exhibits non-monotonic behavior: Gemini 3.5 Flash achieves 44.2% accuracy at \(T_{\max}=1\), drops to 41.6% at \(T_{\max}=10\), and rises to 50.0% (with an IES of 31.00%) at \(T_{\max}=25\). Under a restricted budget of 10, Gemini's active exploration is often truncated after abandoning its prior but before finding conclusive evidence; expanding the budget to 25 allows it to complete its multi-view trajectory.
  • Sub-billion models exhibit total behavioral collapse: Qwen 3.5 0.8B executes zero actions per episode, receiving an IES of 0.00%. Mid-sized models (e.g., Qwen 3.5 9B) show modest scroll/click activity but almost zero hover actions, systematically failing to access evidence conditionally revealed via tooltips.

Highlights & Insights

  • Reframing chart reasoning as embodied evidence gathering: By transitioning from static single-image QA to multi-view interactive dashboards, InSight restores the true ecological validity of data analysis workflows.
  • Interaction Efficiency Score as an anti-cheating metric: IES gracefully decouples lucky guesses from grounded reasoning by penalizing zero-interaction episodes and weighting accuracy by the observable State Change Ratio (SCR).
  • Distinct behavioral signatures among frontier models: Evaluation exposes stark architectural personalitiesโ€”GPT-5.5 adopts an economical, high-precision search strategy (3.13 actions/ep, 57.2% acc), while Gemini 3.5 Flash operates as an exhaustive visual searcher (6.27 actions/ep, 61.3% effective ratio).
  • Rigorous multi-stage data provenance: Extracting candidate spans from authentic master's level analytical notebooks, decomposing compound insights into atomic claims, and validating perturbations with bidirectional NLI offers a robust methodology for future benchmark design.

Limitations & Future Work

  • High-level action space abstractions: The action space simplifies continuous human gestures into discrete coordinates and basic mouse/keyboard events, omitting complex multi-touch gestures or application-specific keyboard shortcuts.
  • Traces serve as necessary but non-exhaustive proxies: While IES verifies whether an action produced observable pixel changes, it does not confirm whether the model's internal attention correctly processed the newly exposed graphical mark.
  • Unweighted state transitions: Currently, any pixel-altering action is credited equally as effective, without distinguishing between exploratory scrolling and the pivotal hover action that reveals the exact target number.
  • vs FEVER / FEVEROUS (Textual Fact-Checking): Textual fact-checking benchmarks rely on discrete token retrieval over Wikipedia; InSight extends the epistemic rigor of fact-checking (including NEI supervision) to continuous, partially observable 2D pixel environments.
  • vs ChartQA / PlotQA (Static Chart QA): Prior chart QA benchmarks present all visual tokens simultaneously in a single frame; InSight hides evidence across multi-view layouts and interactive tooltips, penalizing passive perception.
  • vs OSWorld / VisualWebArena (General Web/GUI Agents): While general GUI benchmarks focus on task goal-state reachability, InSight scopes specifically to claim verification, enabling controlled tracking of how multimodal agents gather, synthesize, and falsify evidence.

Rating

  • Novelty: โญโญโญโญโญ Formulates interactive visual claim verification as sequential evidence acquisition and establishes interaction traces and IES as primary audit proxies.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 14 models, 4 model families, multiple budget settings, and behavioral action breakdowns.
  • Writing Quality: โญโญโญโญโญ Clear problem motivation, rigorous task formalization, transparent data pipeline, and insightful behavioral analysis.
  • Value: โญโญโญโญโญ Provides a vital benchmark for detecting parametric hallucinations and advancing truly grounded multimodal GUI agents.