From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP¶

Conference: ECCV2026
arXiv: 2606.26535
Code: https://github.com/iiyamayuki/CRISP-Bench
Area: Multimodal VLM / Evaluation Benchmark
Keywords: Spatial Reasoning, 3D Scene Graph, Perception-Reasoning Decoupling, Semantic Shortcut, Consistency

TL;DR¶

CRISP is a structural diagnostic benchmark for visual spatial intelligence. Through a dual-task paradigm of Spatial QA + 3D Scene Graph combined with a cross-task consistency protocol, it reveals whether VLM spatial reasoning represents genuine 3D geometric understanding or merely semantic shortcuts relying on language priors.

Background & Motivation¶

Current vision-language models have excelled in semantic recognition—identifying static semantic relationships like "cup on the table" or "table next to the chair" is no longer a challenge. However, true spatial intelligence supporting embodied AI and autonomous driving requires much more than semantic labels: robots must judge relative distances between cups and table edges, estimate whether an object is blocking the way, and "rotate" perspectives in their minds to comprehend spatial layouts from the other side. These capabilities demand that models possess implicit 3D spatial cognition to reconstruct the intrinsic geometric structure of a scene from a single 2D image. Yet, existing evaluation standards typically use classification-style QA to assess spatial reasoning, e.g., asking "Is the chair to the left or right of the table?" with a correct answer being considered a pass. The risk of this approach lies in the fact that models can easily guess the correct answer through language priors (statistically, "chairs are often near tables" and "frames are usually on walls") without needing to construct any 3D geometry of the scene. Many current VLMs exhibit severe internal inconsistencies behind seemingly correct answers—they output correct results while their intermediate spatial modeling is flawed, a phenomenon termed "hallucination of spatial intelligence" in this work.

The key challenge of this problem is that traditional QA formats cannot distinguish "geometric understanding" from "linguistic statistical knowledge." A model can answer a vast number of spatial questions correctly while having almost zero 3D perception, as long as sufficient semantic co-occurrence patterns exist in the training data. Even worse, even if a model does possess partial perception capabilities, it is impossible to determine whether it is reasoning based on visual signals or compensating with language priors. Existing probing methods are also inadequate: textual rationales themselves contain hallucinations, 2D cognitive maps lack 3D metric precision, and attention maps only show that a model looked at a region, not that it understood depth and scale. A more rigorous diagnostic tool is needed.

The key insight of this paper is that to genuinely diagnose spatial intelligence, models must be forced to explicitly externalize their implicit 3D modeling, leaving no escape route through language priors. CRISP’s dual-task paradigm is designed precisely for this: models are required to both answer spatial QA and generate structured 3D Scene Graphs (SGC) containing precise metric estimates (object sizes, distances) and topological relationships (semantic relations like front, back, left, right). Cross-task consistency is then used to verify whether the model’s reasoning is genuinely anchored in its own geometric perception. Core Idea: A model that truly understands a scene must produce QA answers consistent with its own generated 3D Scene Graph; this cross-task consistency itself serves as a reliable diagnostic signal for spatial understanding, rather than simple accuracy in answering questions.

Method¶

Overall Architecture¶

CRISP is a structural diagnostic evaluation benchmark designed to make models answer two "test sheets"—one for traditional Spatial QA and another for 3D Scene Graph Construction (SGC)—and then inspect whether the two sets of answers are self-consistent. The overall architecture consists of four modules: Data Construction, Dual-Task Design, Evaluation Metrics, and Intervention Diagnostic Protocols.

The data is sourced from nuScenes (outdoor driving) and ScanNet++ (indoor scenes), with 1,162 high-quality static single-views selected (1:1 indoor to outdoor ratio). After rigorous visibility filtering (occlusion rate \(< 0.8\), shortest side of 2D bounding boxes \(\ge 40\) pixels) and diversity filtering, 9,839 QA questions are programmatically generated from 3D ground truth using a deterministic logical engine. An ingenious design choice is the exclusive use of numeric IDs instead of semantic labels to refer to objects—terms like "chair" or "table" are completely avoided in the questions, thereby blocking any leaks of semantic co-occurrence priors.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image"] --> B["Spatial QA Task<br/>7 Ability QA Types"]
    A --> C["SGC Task<br/>Output 3D Scene Graph"]
    B --> D["QA Score"]
    C --> E["SGC Score<br/>Metric Estimation + Relations"]
    B --> F["Self-Consistency<br/>Cross-Task Consistency"]
    C --> F
    F --> G["Dual Intervention Experiments"]
    G --> H["Feed GT 3D SG<br/>Test Upper Limit of Reasoning"]
    G --> I["Feed Pred 3D SG<br/>Test Modality Conflict"]
    H --> J["Diagnostic Conclusion"]
    I --> J

Evaluation is performed at three levels: QA Score (using MCQ for accuracy, and MRA for numerical answers), SGC Score (mean of metric estimation \(S_{est}\) and relationship accuracy \(S_{rel}\)), and the most diagnostic Self-Consistency Score—which uses a deterministic logical engine to derive QA answers from the model's own generated SGC, followed by calculating the alignment score against the model’s direct QA answers.

Key Designs¶

1. Dual-Task Structural Diagnosis: Leaving No Place for Implicit Modeling to Hide

Traditional spatial evaluations only utilize QA, allowing models to slip through by relying on language priors. The core insight of CRISP is that to diagnose something, it must be exposed. To this end, CRISP simultaneously designs two complementary tasks: Spatial QA and 3D Scene Graph Construction (SGC). QA tests the "explicit reasoning results," while SGC tests the "implicit perceptual modeling." The combination of these two tasks provides a four-quadrant diagnosis: High QA + Flawed SGC = Semantic Shortcut (the model guesses the correct answer via language priors but its internal geometric modeling collapses); Low QA + Reasonable SGC = Perception-Reasoning Disconnection (a good scene graph is built, but is not utilized during reasoning); both consistent and correct = Genuine Understanding; both consistent and incorrect = Consistent Hallucination (strong language priors bias even SGC). The SGC task employs a star-topology generation strategy—given a central object, the model only generates the relation subgraph between it and other objects. This not only keeps the output size manageable but, crucially, experiments demonstrate that the format compliance rate under this structure is \(>99\%\) for all models, proving that low SGC scores stem from perceptual deficiencies rather than format failure.

2. Self-Consistency: The Most Rigorous Test of "Genuine Understanding"

This is the most core diagnostic tool of CRISP, essentially "using one's own other answer sheet to verify oneself." Specifically, it involves two steps: first, a deterministic logical engine parses the 3D Scene Graph output by the model to symbolically derive answers to corresponding QA questions (Derived QA) (e.g., deriving "Object 5 is to the left of Object 3" from the SGC entry "Object 5 is 1.2m to the left of Object 3"). Then, the consistency between the Derived QA and the model’s direct QA answer (\(A_{QA}\)) is evaluated. Note that this does not check correctness against the ground truth—a model can be consistently "wrong" (so-called "consistent hallucination"), which still does not qualify as understanding. What truly matters is that if the model's spatial reasoning is genuinely anchored in geometric perception, \(A_{QA}\) must be consistent with the SGC derived result. Indeed, experiments reveal approximately 10-12% consistent hallucination samples: the models' SGC-derived answers are self-consistent with their direct QA answers, but both deviate from the ground truth, showing that the pull of language priors is powerful enough to override visual perception signals.

3. Oracle Intervention Experiments: Decoupling Perception from Reasoning

To answer whether the model suffers from poor perception or poor reasoning, CRISP designs dual intervention experiments. The first condition feeds both the image and the Ground-Truth 3D Scene Graph to the model—simulating the upper limit of reasoning capabilities under "ideal perception." The second condition feeds the image and the model's own predicted SGC (Pred SG)—simulating reasoning robustness under "imperfect but realistic perception." By comparing the QA performance under these two conditions against the baseline (image-only), one can precisely pinpoint where the bottleneck lies.

The results are striking. After injecting GT 3D SG, almost all models' QA scores jump dramatically, with Gemini 2.5 Flash improving by 30.3 percentage points, approaching the upper limit of pure symbolic reasoning (~80%). This offers two pieces of evidence: first, it proves that current top VLMs possess robust latent reasoning engines capable of complex multi-hop spatial reasoning; second, since reasoning capability is not the issue, the bottleneck must lie on the perception side—specifically, in the "alignment" step from perceptual output to the reasoning engine. Conversely, when fed Pred SG, most models' QA performance drops, indicating a trust bias towards their own corrupted textual output, preferring to believe it over correcting it using visual signals. Gemini 2.5 Flash is the sole exception (where Pred SG improves QA), suggesting its SGC quality has crossed a critical threshold—when the self-predicted SGC is good enough, the explicit structure itself becomes a cognitive scaffold for reasoning.

4. Three-Mode Diagnostic Classification: From Identifying Issues to Guiding Improvement

Based on this analysis, CRISP summarizes three diagnostic modes and provides corresponding improvement paths. The first is the "Semantic Shortcut" type (High QA + Low SGC/Low Consistency): the model bypasses geometry and answers via language priors, requiring reinforcement of the visual encoder to enhance primitive metric perception (e.g., injecting depth priors). The second is the "Perception-Reasoning Disconnection" type (Medium-High SGC + Low QA): the visual encoder can extract rough structures, but the reasoning engine fails to anchor to them, requiring a redesign of multimodal alignment strategies to bridge the internal utilization gap. The third is the "Modality Trust Bias" type (where QA drops after feeding self-predicted SGC): the model prioritizes its own incorrect textual generation over visual signals, requiring upgrades to the LLM backbone's conflict resolution capabilities and composition reasoning depth. The diagnostic criteria for all three modes can be directly read from CRISP's three-dimensional scores (QA / SGC / Consistency), providing clear, actionable optimization directions for subsequent research on spatial intelligence.

Key Experimental Results¶

Main Results¶

CRISP evaluates 13 SOTA VLMs, including closed-source models (Gemini 2.5 Flash/Pro, Gemini 3 Flash, GPT-5 Mini/5.2), open-source models (Qwen2.5/3-VL, InternVL3.5, LLaVA-OneVision-1.5), and specialized spatial models (Cambrian-S, VG LLM), all under zero-shot settings with thought/reasoning modes disabled for most models to isolate the impact of test-time training computation.

Model	QA	S_est	S_rel	SGC	Consistency
Gemini-2.5-Pro	58.93	57.79	71.36	64.58	57.28
Gemini-3-Flash	53.41	65.25	72.33	68.79	55.20
GPT-5-Mini	54.15	61.39	72.01	66.70	56.75
Qwen3-VL-8B	55.34	57.09	67.62	62.35	52.16
InternVL3.5-38B	53.98	53.12	64.68	58.90	48.70
LLaVA-OV-1.5-8B	47.85	44.53	56.12	50.33	38.98
Cambrian-S (Specialized Spatial)	45.95	46.76	47.50	47.13	34.22

Key findings: The absolute scores of all models are far lower than those reported on traditional VQA, indicating that true spatial intelligence is far from solved. All models perform significantly better in topological relations (\(S_{rel}\)) than metric estimation (\(S_{est}\)), showing that precise 3D metric perception remains the biggest shared bottleneck. Specialized visual-spatial models actually lag behind general-purpose baselines, implying that current spatial post-training methods may only activate local structural cues without ensuring that the reasoning engine utilizes them. Notably, the performance of the Qwen3-VL-8B open-source model is close to GPT-5-Mini, showing that the gap between open-source and closed-source models is shrinking rapidly.

Ablation Study¶

Blind tests (removing visual inputs, using only numeric IDs in questions) provide a critical baseline for understanding each model’s visual contribution.

Model	QA Blind (Drop)	SGC Blind (Drop)	Key Interpretation
Gemini-3-Flash	7.94 (-45.47)	39.47 (-29.32)	Visual contribution is the largest, but safety policies led to a high rate of refusal to answer under the Blind setting
Gemini-2.5-Pro	26.90 (-32.03)	37.05 (-27.53)	Visual signal significantly improves SGC, showing solid perception capabilities
Qwen2.5-VL-7B	28.47 (-16.31)	39.46 (-3.24)	⚠️ QA improves significantly but SGC is nearly unchanged—revealing a semantic-geometric gap
Qwen3-VL-8B	31.29 (-24.05)	34.75 (-27.60)	Visual gains are balanced across QA and SGC, showing no gap
LLaVA-OV-1.5-8B	22.39 (-25.46)	43.20 (-7.13)	Minimal SGC gains; vision is primarily utilized for semantic recognition rather than geometric understanding

Key Findings¶

The Semantic-Geometric Gap is the most insightful discovery: With visual input, Qwen2.5-VL’s QA score increases by 16.31 points, but the SGC score only rises by 3.24 points, almost returning to the baseline. This means its visual encoder successfully identifies objects in the frame (recovering semantic context) but fails to translate this into a geometric understanding of "where they are"—the model knows there is a chair, but does not know its concrete location or distance. This strongly demonstrates that traditional QA evaluations conflate semantic recognition with spatial understanding.
Perception-Reasoning Disconnection is a typical issue for closed-source models: Gemini 2.5 Flash’s SGC score is almost on par with the Pro version (only 0.34 points lower), yet its QA score lags by 11.25 points—it builds a reasonable scene graph but fails to utilize it during reasoning. Intervention experiments further confirm this: once GT 3D SG is fed, the QA score surges by 30 percentage points, showing that the reasoning engine itself is fully intact; the issue lies in the alignment of perceptual outputs to reasoning inputs.
Cognitive Scaffolding Effect: Even if self-predicted SGC has metric errors, most models still benefit from it in compositional logic tasks. This indicates that explicit structured representation itself possesses a cognitive offloading function—even with high metric noise, structural topology relieves the model's computational burden of maintaining spatial relationships, allowing limited resources to be focused on high-level reasoning.

Highlights & Insights¶

Replacing Accuracy with Self-Consistency as an Understanding Metric: This is the most valuable conceptual contribution of the work. Traditional evaluations rely solely on whether final answers are correct. In contrast, CRISP introduces "cross-task internal consistency" as the benchmark for understanding—answering correctly does not prove understanding, but answering correctly while remaining consistent with one's own generated scene graph does. This paradigm can be extended to any scenario requiring diagnosis of whether a model genuinely understands or is cheating.
Extremely Clean Design of Oracle Intervention Experiments: By solely changing input conditions (GT vs. Pred SGC), the causal relationship between perception and reasoning is cleanly isolated, mimicking the "controlled variable method" in scientific studies. This elegantly proves the core conclusion—the reasoning engine is fine, the issue is alignment. This experimental paradigm can be directly extended to other tasks requiring the decoupling of perception and reasoning (e.g., navigation, object manipulation).
Anti-leakage Design with Numeric IDs: This seemingly minor adjustment blocks major shortcuts. Referencing objects by numeric IDs instead of semantic labels in the SGC task fundamentally prevents the leakage of semantic co-occurrence priors. Many spatial evaluations claim to measure "spatial understanding," yet their benchmarks leak substantial semantic information (e.g., the questions themselves contain object names, which naturally carry positional priors). This design detail of CRISP is highly instructional for all spatial evaluation benchmarks.
A Complete Loop from Diagnosis to Action: CRISP does not just identify three diagnostic modes; it proposes specific improvement paths for each. This is rare for purely evaluative works and directly translates benchmark results into actionable R&D directions.

Limitations & Future Work¶

The paper itself acknowledges that terms like "perception-reasoning disconnection" describe empirically observed behavioral syndromes rather than causally verified neural mechanisms. The current evidence is correlational; more refined causal experiments (such as ablation-by-intervention on intermediate representations) are needed to confirm if a true "disconnection" exists.
CRISP focuses on static single-view 3D geometric understanding, which is a necessary but insufficient condition. True spatial intelligence requires handling dynamic multi-view scenes (e.g., parallax, motion occlusion), which is left to future work.
The data is sourced from nuScenes and ScanNet++, restricting scene types to urban roads and standard indoor environments. Extreme environments such as highly cluttered scenes, transparent objects, specular reflections, or water bodies are hardly covered, potentially introducing domain bias.
In the Oracle intervention experiments, the GT 3D SG is injected in text form (natural language descriptions), and the model's own SGC is also textual. This raises a competing hypothesis: textual structures might inherently be easier to reason over than raw visual signals. If a rendered 3D Bird's-Eye View (BEV) map were injected instead, would GPT-5 still maintain its reasoning upper limit? Distinguishing the contribution of "text as a reasoning medium" from "structural information itself" is a promising future direction.

vs. VSI-bench / OmniSpatial / SpatialEval: These spatial VLM benchmarks exclusively adopt pure QA formats, making them unable to distinguish semantic shortcuts from genuine geometric understanding. CRISP introduces SGC generation tasks and Consistency metrics, fundamentally enhancing diagnostic granularity.
vs. SpatialRGPT / SpatialCLIP: These methods enhance VLM spatial perception by explicitly injecting depth maps. CRISP's diagnostic results provide the most direct justification for such methods: if perception-reasoning alignment is the main bottleneck, depth maps acting as an explicit structural prior are highly likely to be an effective solution.
vs. SPACE / Whatsup: These early works identified spatial deficiencies in VLMs but only pointed out that "problems exist" without answering "whether it is a perceptual or reasoning issue." CRISP’s intervention experiments and three diagnostic modes address this gap.
vs. SpatialReasoner / SpaceR: These works employ RL to enhance spatial reasoning in VLMs. CRISP's finding that open-source models lack multi-hop compositional reasoning (rather than perceptual capabilities) provides a concrete optimization target for RL training strategies—multi-hop composition should be prioritized over perception enhancement.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Replacing accuracy with cross-task consistency to diagnose spatial understanding is brilliant. The design of the Oracle intervention experiments is exceptionally clever, and the three findings hold independent value.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 13 models \(\times\) 4 dimensions (QA/SGC/Consistency/Intervention). Blind tests, interventions, and algorithmic breakdowns are cleanly and robustly designed.
Writing Quality: ⭐⭐⭐⭐ Structuring and argumentation are clear, though some sections are terminologically dense and require repeated reading to fully digest.
Value: ⭐⭐⭐⭐⭐ Provides an entirely new diagnostic paradigm for spatial VLM evaluation, with the three diagnostic modes and their corresponding improvement paths directly guiding future research directions.