Object Presence Is Not Relation Evidence: Relation-Grounded Faithful Reasoning for Multimodal Large Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Relation Hallucination, Relation Evidence Amnesia, Counterfactual Causal Probing, Training-Free Decoding Intervention
TL;DR¶
Addressing the Relation Evidence Amnesia pathology where MLLMs fabricate ungrounded interactions from mere entity co-presence, RGFR introduces a training-free inference-time decoding intervention that dynamically routes decoupled geometric constraints and counterfactual causal probes, lifting relation recognition accuracy by over 8.1% while maintaining general task utility.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) have established remarkable visual perception capabilities across complex scene understanding and open-vocabulary grounding tasks. Nonetheless, hallucinated generation driven by linguistic inertia remains a fundamental bottleneck that impairs their deployment in high-stakes reasoning applications. In conventional object-centric hallucination benchmarks, failures are predominantly framed as perceptual blindness where models fail to localize occluded, small, or absent entities. In structural relationship comprehension, however, empirical evidence uncovers an entirely distinct failure paradigm: models accurately identify and localize all participating visual entities (e.g., recognizing both a "man" and a "dog"), yet their deep generative layers actively rationalize non-existent physical interactions (e.g., claiming the man is "feeding" the dog and inventing fabricated narrative evidence such as "holding a bag of food").
This pervasive cognitive override is systematically formalized in this work as Relation Evidence Amnesia (REA). Rather than stemming from deficient sensory inputs, REA originates from semantic rationalization hijacking the generative trajectory, manifesting in three orthogonal modes: Coordinate Blindness, wherein the model ignores spatial arrangement rules despite precise entity bounding boxes; Co-presence Bias, wherein the mere visual co-occurrence of multiple entities is treated as definitive evidence of an interaction; and Textual Prior Overconfidence, wherein strong language priors override conflicting or ambiguous visual signals. Layer-wise cross-modal attention tracking reveals a critical Perception-Cognition Gap: while shallow perception layers faithfully attend to physical interaction boundaries and contact interfaces, attention in deep causal layers rapidly collapses back to salient entity centers, cognitively suppressing the actual relational evidence.
Existing mitigation paradigms primarily rely either on massive supervised fine-tuning or training-free inference-time interventions. Fine-tuning struggles to eliminate deep-rooted language priors without distorting general capabilities, while inference-time contrastive methods such as Visual Contrastive Decoding (VCD) rely on injecting indiscriminate global Gaussian noise into the entire image. Although global perturbation successfully penalizes object-level existence hallucinations, it inevitably blurs the fine-grained spatial boundaries and localized contact surfaces essential for physical interaction verification. Core idea: Decouple relation verification into explicit symbolic geometric constraints and targeted counterfactual causal probing, surgically blinding relation evidence masks in deep attention layers, and mediating unfaithful language priors via a prior-calibrated schema-aware rectifier.
Method¶
Overall Architecture¶
Relation-Grounded Faithful Reasoning (RGFR) is an inference-time decoding intervention framework that operates on frozen model parameters to intercept prior-driven rationalization. As illustrated in the pipeline, RGFR comprises three cooperative components: First, a Semantic-Driven Dynamic Router utilizes the LLM's pre-trained token embedding space to project queried predicate semantics into continuous soft-gating weights. Second, a Parallel Dual-Branch Verifier simultaneously deploys an external bounding-box-level Geometric Verifier to check spatial topologies and an internal Causal Probing Branch using a Relation Evidence Schema (RES) to surgically mask predicate-relevant interaction areas across deep attention layers. Finally, a Unified Schema-Aware Rectifier quantifies visual epistemic uncertainty via factual-counterfactual KL divergence, incorporates unconditional textual priors, and dynamically adjusts the output logits during autoregressive generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Input<br/>Image + Queried Relation Predicate r"] --> B["Semantic-Driven Dynamic Router<br/>Static Unembedding Cosine Projection to Soft-Gating Weights"]
B --> C["Parallel Dual-Branch Verifier<br/>Symbolic Spatial Checking and RES Attention Probing"]
C --> D["Unified Schema-Aware Rectifier<br/>KL Divergence Uncertainty and Textual Prior Logit Rectification"]
D --> E["Relation-Grounded Faithful Prediction Output"]
Key Designs¶
1. Semantic-Driven Dynamic Router: Continuous Soft-Gating via Unembedding Projection
To circumvent brittle heuristic rules and rigid classification thresholds, the router functions as a zero-shot continuous gating mechanism built upon the LLM's static unembedding space \(E(x)\). During an offline preparation phase, compact anchor sets for spatial prepositions \(A_{\text{geo}}\) and physical action verbs \(A_{\text{cau}}\) are mapped to their respective semantic centroids:
For an incoming queried relation predicate \(r\), the router calculates the cosine similarity \(s_k = \cos(E(r), e_k)\) and derives smooth routing weights via a temperature-scaled softmax projection:
This soft formulation seamlessly assigns dominant geometric weights (\(\omega_{\text{geo}} \to 1\)) to directional and structural prepositions, prioritizes causal probing (\(\omega_{\text{cau}} \to 1\)) for physical interactions, and adaptively balances composite relations without any manual intervention.
2. Parallel Dual-Branch Verifier: Decoupled Symbolic Geometry and Counterfactual RES Masking
To tackle Coordinate Blindness and Co-presence Bias simultaneously, RGFR constructs two orthogonal verification pathways. The Geometric Verifier branch receives localized object bounding boxes \(b_A\) and \(b_B\) from Grounding DINO and enforces deterministic spatial rules outside the neural network. If the relative coordinates directly contradict the queried spatial relation, an infinite penalty \(P_{\text{geo}} = \infty\) is assigned; otherwise, \(P_{\text{geo}} = 0\), operating with zero GPU compute overhead.
Concurrently, the Causal Probing Branch maps predicate \(r\) into canonical sub-masks \(M(s)\) governed by the Relation Evidence Schema (RES): Contact (physical proximity and touching boundaries), Spatial (vertical/horizontal support intervals), and Directional (trajectory vectors). During deep causal decoding layers (e.g., layers 22 to 32), a bifurcated counterfactual forward pass with shared KV-cache surgically blinds the pre-softmax attention matrix \(S^{(l)}\):
Governed by the Weakest Link Principleโwhere spatial alignment cannot compensate for an absent physical contact interfaceโthe branch aggregates schema-specific sensitivities \(e_s^{(l)}\) using a non-compensatory geometric mean, ensuring that the absence of any critical interaction evidence invalidates the hallucinated relation.
3. Unified Schema-Aware Rectifier: Prior-Calibrated Dynamic Logit Rectification
To neutralize strong textual priors that persist even when visual cues are absent, the rectifier measures the model's visual epistemic uncertainty using the Kullback-Leibler (KL) divergence between factual prediction \(p\) and counterfactual prediction \(p_{\text{cf}}\):
A low \(\Delta_{\text{KL}}\) indicates that surgically removing visual interaction evidence causes negligible change in the model's output distribution, exposing an ungrounded hallucination driven by linguistic inertia. The rectifier computes an unconditional text-only prior \(p_{\text{prior}, r}\) and conditionally activates a causal penalty \(P_{\text{cau}}\) whenever \(\Delta_{\text{KL}}\) drops below threshold \(\tau\):
Combining the geometric and causal constraints via the soft-gating weights, the final rectified logit \(\tilde{z}_r\) is computed as:
This formula forcefully penalizes ungrounded generation candidates and allows visually supported relations to surface naturally.
Loss & Training¶
RGFR is strictly a training-free test-time intervention framework requiring zero parameter updates or fine-tuning loss optimizations. All evaluations are conducted across frozen pre-trained checkpoints (LLaVA-1.5-7B, InstructBLIP-7B, Qwen2.5-VL-7B, and InternVL3.5-8B). Entity bounding boxes are extracted using an off-the-shelf Grounding DINO detector. Counterfactual forward passes share KV-caches across all early-to-mid layers, limiting the additional forward computation strictly to the designated late intervention layers.
Key Experimental Results¶
Main Results¶
On two major relation hallucination benchmarksโMMRel (11,871 samples across VG and DALL-E) and R-Bench (7,787 image-level samples)โRGFR achieves substantial performance gains over vanilla decoding and competitive training-free baselines (VCD, CAP, Calibrate):
| Model | Method | MMRel Acc (%) | MMRel Prec (%) | MMRel F1 (%) | R-Bench Acc (%) | R-Bench Prec (%) | R-Bench F1 (%) |
|---|---|---|---|---|---|---|---|
| LLaVA-1.5 | Vanilla | 57.50 | 55.15 | 69.53 | 69.72 | 63.26 | 76.10 |
| LLaVA-1.5 | VCD | 60.05 | 57.16 | 70.79 | 71.23 | 64.59 | 76.97 |
| LLaVA-1.5 | CAP | 58.08 | 55.52 | 69.97 | 70.14 | 63.59 | 76.38 |
| LLaVA-1.5 | Calibrate | 58.41 | 56.03 | 69.67 | 74.76 | 74.66 | 83.85 |
| LLaVA-1.5 | RGFR (Ours) | 81.97 | 80.69 | 83.16 | 77.96 | 77.83 | 84.41 |
| InstructBLIP | Vanilla | 58.54 | 56.10 | 70.18 | 76.72 | 80.91 | 82.65 |
| InstructBLIP | VCD | 58.62 | 55.84 | 70.17 | 80.48 | 83.59 | 86.98 |
| InstructBLIP | CAP | 62.44 | 59.75 | 71.47 | 80.18 | 81.28 | 85.79 |
| InstructBLIP | Calibrate | 60.88 | 57.80 | 71.41 | 80.75 | 84.34 | 85.61 |
| InstructBLIP | RGFR (Ours) | 84.14 | 82.99 | 84.99 | 83.67 | 83.89 | 87.99 |
| Qwen2.5-VL | Vanilla | 77.85 | 74.60 | 80.55 | 76.64 | 91.58 | 79.59 |
| Qwen2.5-VL | VCD | 79.78 | 76.71 | 81.96 | 81.53 | 89.66 | 85.20 |
| Qwen2.5-VL | CAP | 82.67 | 79.49 | 84.12 | 80.83 | 87.42 | 85.06 |
| Qwen2.5-VL | Calibrate | 79.72 | 77.01 | 81.86 | 81.64 | 89.55 | 85.17 |
| Qwen2.5-VL | RGFR (Ours) | 87.44 | 89.73 | 87.22 | 82.20 | 91.59 | 85.31 |
| InternVL3.5 | Vanilla | 77.01 | 71.91 | 80.84 | 83.87 | 85.33 | 88.21 |
| InternVL3.5 | VCD | 76.45 | 72.74 | 79.74 | 82.39 | 86.75 | 86.65 |
| InternVL3.5 | CAP | 80.06 | 75.43 | 82.72 | 84.44 | 87.32 | 88.03 |
| InternVL3.5 | Calibrate | 77.22 | 71.52 | 81.24 | 84.08 | 86.80 | 88.16 |
| InternVL3.5 | RGFR (Ours) | 87.85 | 86.98 | 88.31 | 86.35 | 86.41 | 89.87 |
Ablation Study¶
On the DALL-E subset of MMRel, component ablations highlight the necessity of decoupling geometric verification from causal probing across Spatial and Action subsets:
| Model | Configuration | Spatial Acc (%) | Action Acc (%) | Overall Acc (%) |
|---|---|---|---|---|
| LLaVA-1.5 | Vanilla | 52.34 | 48.65 | 50.49 |
| LLaVA-1.5 | Geometric Only | 76.51 | 49.12 | 62.81 |
| LLaVA-1.5 | Causal Only | 51.08 | 71.43 | 61.25 |
| LLaVA-1.5 | RGFR (Full Dual-Branch) | 77.15 | 72.38 | 74.76 |
| Qwen2.5-VL | Vanilla | 75.20 | 71.30 | 73.25 |
| Qwen2.5-VL | Geometric Only | 88.45 | 70.55 | 79.50 |
| Qwen2.5-VL | Causal Only | 74.82 | 85.60 | 80.21 |
| Qwen2.5-VL | RGFR (Full Dual-Branch) | 89.12 | 86.45 | 87.78 |
| InternVL3.5 | Vanilla | 72.85 | 69.45 | 71.15 |
| InternVL3.5 | Geometric Only | 85.10 | 68.20 | 76.65 |
| InternVL3.5 | Causal Only | 73.05 | 82.55 | 77.80 |
| InternVL3.5 | RGFR (Full Dual-Branch) | 85.88 | 83.20 | 84.54 |
Key Findings¶
- Orthogonal Verification Dynamics: Symbolic geometric checking sharply boosts Spatial accuracy (e.g., +24.17% on LLaVA-1.5) but fails to impact physical Action predictions. Conversely, causal counterfactual masking lifts Action accuracy (+22.78%) while leaving spatial orientation uncalibrated. Full RGFR effectively unifies both dimensions via dynamic soft-gating.
- Intervention Layer Localization: Evaluating LLaVA-1.5 under different intervention depths indicates that late-stage intervention (L22โL32) achieves the best accuracy (81.97%) and precision (80.69%), whereas early-stage intervention (L0โL10) yields only 59.85%. Full-layer intervention drops to 74.65%, confirming that linguistic suppression occurs predominantly in the deepest reasoning layers.
- Targeted Masking vs. Global Noise: On Qwen2.5-VL, targeted RES attention masking delivers 87.8% accuracy compared to 78.4% for global noise (VCD) and 73.3% for vanilla decoding. Global noise blurs fine-grained interaction boundaries, whereas targeted masking isolates true visual dependencies.
- Utility and Latency Trade-Off: Testing general multimodal performance on MM-Vet with Qwen2.5-VL shows that RGFR preserves strong utility (62.4 score, 5.637s latency) compared to sharp utility drops in VCD (54.4 score, 7.121s) and CAP (57.0 score, 6.159s).
Highlights & Insights¶
- Mechanistic Pathology Framing: Identifies and conceptualizes Relation Evidence Amnesia (REA), proving that relationship hallucination stems from cognitive suppression driven by textual priors rather than basic perceptual failure.
- Surgical Relation-Evidence Masking: Replaces indiscriminate image-level noise with predicate-guided attention masking across contact, spatial, and directional primitives, establishing a precise causal dependency baseline.
- Non-Compensatory Evidence Fusion: Formalizes the Weakest Link Principle via geometric mean aggregation, preventing models from compensating for missing physical contact using spatial co-presence alone.
Limitations & Future Work¶
- Binary Relation Scope: The current framework is developed for binary relation verification under single-image settings and has not yet been extended to dense scene graph generation, multi-turn dialogues, or video temporal reasoning.
- Detector Error Propagation: Relies on Grounding DINO for bounding box extraction. Error analysis indicates that missing localization and imprecise bounding boxes account for 36.8% of remaining failure cases.
- Out-of-Schema Fallback: Rare or complex predicates outside the three canonical primitives fall back to vanilla model decoding without active causal intervention.
Related Work & Insights¶
- vs VCD (Visual Contrastive Decoding): VCD applies global Gaussian noise across the whole visual field to counter object-level hallucination. While effective for object presence, uniform noise destroys boundary-level interaction signals; RGFR maintains context integrity and surgically probes relation-bearing attention tokens.
- vs CAP (Constraint-Aware Prompting): CAP prepends spatial and logical rules to prompts. However, text prompts cannot intercept internal attention disorientation in deep causal layers; RGFR acts directly upon pre-softmax attention matrices and logit distributions.
- vs Calibrate (Reefknot): Calibrate adjusts confidence post-hoc through calibration heads or statistical heuristics; RGFR directly couples external symbolic verification with an internal counterfactual causal probe.
Rating¶
- Novelty: โญโญโญโญโญ Formulates Relation Evidence Amnesia with convincing attention analysis and designs a principled, training-free dual-branch intervention.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 4 open-source MLLMs and commercial APIs (GPT-5.5, Gemini 3.5 Flash), spanning MMRel and R-Bench with extensive ablations.
- Writing Quality: โญโญโญโญโญ Clear theoretical motivation, coherent structure, and mathematically sound formulation.
- Value: โญโญโญโญโญ Delivers an efficient, plug-and-play inference-time framework that significantly improves structural reasoning trustworthiness.