Skip to content

Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Language Prior Interference, Visual Perception, Dynamic Prior Induction, Model-Aware Prior Softening

TL;DR

Addressing the critical flaw where multimodal LLMs favor internalized text priors over direct visual evidence ("believing rather than seeing"), this paper introduces ViSiP-Benchโ€”a 4,500-sample rare-but-real benchmark with model-aware dynamic prior induction and continuous Visual Information Gain (VIG)โ€”and proposes MAPS, a targeted knowledge-softening method that significantly boosts visual fidelity without degrading general reasoning capabilities.

Background & Motivation

Multimodal Large Language Models (MLLMs) have established remarkable breakthroughs in multimodal reasoning and vision-language integration, leveraging vast pre-trained linguistic knowledge to reason about complex scenes. These internalized language priors encode commonsense and world structure that fundamentally underpin semantic understanding. However, when visual evidence sharply contradicts established commonsense, these language priors regularly hijack the perceptual pipeline, causing models to report what they expect rather than what they actually observe. A classic illustration occurs when an image displays an elementary arithmetic mistake like \(37 + 77 = 110\); leading MLLMs overwhelmingly adhere to their addition prior and insist on \(114\). Such ungrounded perceptual failures pose grave risks in safety-critical deployments, such as radiology anomaly detection and autonomous industrial fault inspection.

Despite growing interest, existing language prior benchmarks (such as ROME, HallusionBench, and VLind-Bench) suffer from two foundational methodology flaws. First, they operate under the rigid assumption that all models share an identical commonsense prior, testing them against a fixed set of synthetic anomalies; yet cross-model prior agreement analysis reveals that prior alignment among prominent architectures drops below 80%. Second, existing evaluations rely solely on raw accuracy, ignoring that more than 15% of items can be answered correctly without any image whatsoever. Dissecting these image-free correct responses reveals three distinct confounders: Chance Success (weak priors leading to random guessing), Low-quality Framing (leading or ambiguous prompts leaking the anomaly), and Anomalous Prior (the model natively favoring the atypical option). Furthermore, many prior datasets employ physically impossible counterfactuals rather than rare-but-real instances, hopelessly conflating prior resistance with reasoning over impossible worlds.

Current mitigation paradigms are similarly trapped in severe trade-offs. Inference-time strategies such as Chain-of-Thought (CoT) or prompt steering either hit the ceiling of frozen representations or actively worsen prior adherenceโ€”extended reasoning steps systematically amplify language shortcuts. Conversely, existing fine-tuning methods directly penalize prior-aligned logits, indiscriminately eroding beneficial world knowledge and causing catastrophic degradation across general reasoning benchmarks. To resolve this impasse, this paper pursues a realistic, model-calibrated evaluation paradigm coupled with constructive knowledge enrichment rather than destructive suppression. Core idea: benchmark genuine prior resistance on rare-but-real scenarios via model-aware dynamic prior induction and continuous log-odds VIG metrics, and introduce MAPS to probe model-specific strong priors and soften them through factually grounded rare knowledge fine-tuning without impairing general multimodal competence.

Method

Overall Architecture

The proposed methodology encompasses two tightly integrated components: the ViSiP-Bench diagnostic suite with its dynamic evaluation protocol, and the MAPS (Model-Aware Prior Softening) training framework. ViSiP-Bench comprises 4,500 carefully curated instances across six commonsense domains: Physical Properties, Time & Space, Scientific & Mathematic, Customs & Culture, Psychology Cognition, and Function Cognition. Every image captures a rare-but-real physical phenomenon (collected from authentic web scenes or generated across diverse artistic styles via GPT-Image-2 and Nano Banana 2). During evaluation, each target model is queried first under the image-free Blank condition to elicit its inherent prior \(a_B\), and subsequently under the Visual condition to obtain \(a_V\). Confounders where \(a_B = a^*\) are strictly filtered out to yield pristine decision metrics (PAR and Filtered Accuracy) and continuous confidence metrics (VIG). On the mitigation side, MAPS queries the model with probe questions to identify hyper-confident priors, synthesizes factually grounded rare counter-statements, and applies LoRA fine-tuning to soften prior rigidity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Evaluation Inputs<br/>Rare-but-Real Scenes Across 6 Domains"] --> B["Dynamic Prior Induction Protocol<br/>Blank and Visual Paired Probing"]
    B --> C["Multi-Dimensional Resistance Metrics<br/>PAR Adherence + Filtered Acc + VIG Shift"]
    C --> D["Model-Aware Prior Softening MAPS<br/>High-Confidence Prior Probe & Filter"]
    D --> E["Factually Grounded Knowledge Injection<br/>Synthesize Rare Facts & LoRA Supervision"]
    E --> F["Prior-Resistant MLLM<br/>Low PAR / High VIG / Zero General Degradation"]

Key Designs

1. Dynamic Prior Induction Protocol & Filtered Decision Metrics: Eliminating Confounded Resistance

To overcome the invalid assumption of shared universal priors and the contamination of image-free guessing, ViSiP-Bench implements dynamic prior induction. For any target model, the evaluation runs in two stages: under the Blank condition, the model receives only the text question and four candidate choices, yielding its prior probability distribution \(p_B\) and chosen answer \(a_B\); under the Visual condition, the image is paired with the identical prompt template to produce \(p_V\) and \(a_V\) (with ground-truth visual label \(a^*\)). To purify the test bed from random chance hits (\(a_B = a^*\)) and idiosyncratic anomalies, all items where the model correctly guesses \(a^*\) without the image are removed. The Prior Adherence Rate (PAR) is then computed strictly on genuine conflict instances where \(a_B \neq a^*\), quantifying how often the model clings to its text prior despite seeing conflicting visual reality: $$ \text{PAR} = \frac{\sum_{i: a_{B,i} \neq a^i} \mathbb{I}[a \neq a^} = a_{B,i}]}{\sum_{i} \mathbb{I}[a_{B,i_i]} $$ Filtered Accuracy is evaluated concurrently on the identical conflict subset (\(a_{B,i} \neq a^*_i\)). By anchoring the evaluation to the model's own elicited beliefs rather than an arbitrary external expectation, this protocol isolates true perceptual resilience from spurious text reasoning.

2. Visual Information Gain (VIG): Measuring Continuous Confidence Shifts in Log-Odds Space

Discrete categorical metrics (PAR and accuracy) fail to measure subtle internal distribution shifts when visual evidence nudges model confidence without overturning the top-1 argmax choice. To provide continuous, fine-grained diagnostic sensitivity, the paper introduces Visual Information Gain (VIG) for models with accessible output token probabilities. VIG measures the change in log-odds assigned to the ground-truth visual answer \(a^*\) upon receiving the visual input: $$ \text{VIG} = \ln \frac{p_V(a^) + \epsilon}{1 - p_V(a^) + \epsilon} - \ln \frac{p_B(a^) + \epsilon}{1 - p_B(a^) + \epsilon} $$ where \(\epsilon = 10^{-7}\) prevents numerical instability (yielding a bounded dynamic range of \((-33, 33)\)). A large positive VIG confirms that the visual evidence decisively shifts probability mass toward the authentic percept; a near-zero or negative VIG reveals complete perceptual neglect. Crucially, tracking VIG across parameter scales uncovers the VIG Polarization effect: larger models do not uniformly resist priors, but split into extreme behavioral polesโ€”either overriding priors decisively or clinging to them with even greater stubbornness than smaller models.

3. Model-Aware Prior Softening (MAPS): Mitigating Prior Interference via Constructive Knowledge Enrichment

Existing training-based mitigations (such as Post-hoc probability penalties and NaPO preference tuning) directly penalize the logits of prior-driven tokens, which devastates the model's broader commonsense repository (as evidenced by a catastrophic surge in filtered-out questions FC and steep drops on general benchmarks). MAPS replaces destructive penalization with constructive world-knowledge expansion through a five-step pipeline: 1. Prior Probing Generation: For each base commonsense prior, Claude Opus 4.7 formulates a 10-choice question designed to probe the model's bias (e.g., "What is the color of crows?" with ten plausible color options); 2. Prior Probability Extraction: The target model is queried on these probe questions under text-only mode to extract its 10-way softmax distribution; 3. Targeted Selection: Questions where any single option achieves a probability exceeding threshold \(\alpha\) (default \(\alpha = 0.90\)) are selected, precisely isolating the priors held with dogmatic certainty; 4. Factual Rare Knowledge Generation: For each selected prior, Claude Opus 4.7 generates 20 factual single-sentence descriptions of rare-but-real phenomena that violate the prior (e.g., "Young crows have brownish plumage", "Albino crows are entirely snow-white due to a genetic lack of melanin"); prompt templates are kept strictly disjoint from ViSiP-Bench items to prevent test leakage; 5. Supervised LoRA Fine-Tuning: The generated rare-knowledge sentences are compiled into a model-specific corpus to fine-tune the model via LoRA for 2 epochs. By teaching the model that physical reality permits valid exceptions, MAPS softens excessive prior rigidity while leaving its foundational knowledge matrix intact.

Benchmark & Evaluation Protocol

ViSiP-Bench incorporates 4,500 high-quality question-image instances spanning 1,200 unique commonsense priors across six domains. Imagery combines authenticated real web photographs with balanced multi-style synthetic images (photorealistic, anime, oil painting, and sci-fi) from GPT-Image-2 and Nano Banana 2. All data underwent double-blind review requiring unambiguous visual ground truth, plausible non-eliminable distractors, and neutral non-leading phrasing. Experiments evaluate 21 open-source checkpoints across six model families (InternVL3.5, InternVL3, Qwen2.5-VL, Qwen3-VL, Qwen3.5, and LLaVA-OneVision) and 3 proprietary frontier models (GPT-5.5, Claude Sonnet 4.6, Gemini-3-flash-preview). Generalization is comprehensively tracked across reasoning-intensive suites (MMMU-Pro, VisuLogic) and general vision benchmarks (MMStar, RealWorldQA).

Key Experimental Results

Main Results: ViSiP-Bench Evaluation & Intervention Comparison

Table 1 summarizes the benchmark evaluation across model scales and architectures, while Table 2 contrasts inference-time and training-based interventions evaluated on Qwen3-VL-8B-Instruct.

Model / Intervention Category Model / Method Name PAR (%) โ†“ Filtered Acc (%) โ†‘ FC (Filtered Count) \(\text{VIG}_{\text{avg}}\) โ†‘ \(\text{VIG}_{\text{med}}\) โ†‘
Open-Source Small (2B-3B) InternVL3.5-2B 51.51 42.94 182 4.41 4.11
Open-Source Small (2B-3B) Qwen2.5-VL-3B-Instruct 63.18 28.84 236 3.21 2.27
Open-Source Small (2B-3B) llava-onevision-qwen2-0.5b 72.11 21.45 381 2.81 1.99
Open-Source Mid (7B-9B) InternVL3.5-8B 47.17 44.64 156 7.13 7.48
Open-Source Mid (7B-9B) Qwen3-VL-8B-Instruct (Base) 42.67 51.29 147 10.64 10.82
Open-Source Mid (7B-9B) Qwen3.5-9B 51.24 39.36 137 5.24 5.24
Open-Source Large (27B-78B) InternVL3.5-38B 30.04 63.01 102 10.12 10.31
Open-Source Large (27B-78B) InternVL3-78B 36.95 61.26 81 12.84 12.93
Open-Source Large (27B-78B) Qwen2.5-VL-72B-Instruct 37.03 54.86 119 9.64 9.87
Proprietary Frontier GPT-5.5 26.01 69.94 39 โ€” โ€”
Proprietary Frontier Claude Sonnet 4.6 30.44 65.87 22 โ€” โ€”
Proprietary Frontier Gemini-3-flash-preview 28.64 68.00 48 โ€” โ€”
Inference-Time Strategy + VFP (Visual-Focused Prompt) 40.60 53.01 147 10.78 10.90
Inference-Time Strategy + CoT (Chain-of-Thought) 47.71 43.11 147 8.11 8.32
Inference-Time Strategy + AS (Attention Steering) 42.11 52.08 147 10.88 11.22
Inference-Time Strategy + VCD (Contrastive Decoding) 39.15 55.48 147 12.11 13.08
Inference-Time Strategy + PAI (Paying Attention to Image) 39.01 55.23 147 12.39 13.97
Training-Based Baseline + Post-hoc (Logit Penalization) 32.05 61.94 385 19.81 18.92
Training-Based Baseline + NaPO (Preference Optimization) 37.09 57.34 209 16.33 17.12
Training-Based (Ours) + MAPS 35.09 59.01 154 18.84 18.90

Ablation Study: Probing Threshold Sensitivity & Generalization Preservation

Table 3 details the sensitivity of the prior selection threshold \(\alpha\) (with total training tokens strictly normalized), and Table 4 tracks capability preservation on external reasoning and general vision benchmarks.

Evaluation Stream Configuration / Method ViSiP PAR (%) โ†“ ViSiP Acc (%) โ†‘ FC MMMU-Pro โ†‘ VisuLogic โ†‘ MMStar โ†‘ RealWorldQA โ†‘
Generalization Benchmarks Qwen3-VL-8B Base 42.67 51.29 147 55.90 28.14 70.90 71.50
Generalization Benchmarks + Post-hoc 32.05 61.94 385 41.88 (-14.02) 21.50 (-6.64) 56.18 (-14.72) 53.08 (-18.42)
Generalization Benchmarks + NaPO 37.09 57.34 209 49.51 (-6.39) 24.93 (-3.21) 66.51 (-4.39) 66.00 (-5.50)
Generalization Benchmarks + MAPS (Ours) 35.09 59.01 154 56.84 (+0.94) 27.94 (-0.20) 69.27 (-1.63) 71.68 (+0.18)
Threshold Ablation \(\alpha = 0.50\) (Broad / Loose) 33.09 60.93 218 50.90 โ€” 63.84 โ€”
Threshold Ablation \(\alpha = 0.80\) 34.24 60.10 181 54.17 โ€” 66.81 โ€”
Threshold Ablation \(\alpha = 0.90\) (Default Optimal) 35.09 59.01 154 56.84 โ€” 69.27 โ€”
Threshold Ablation \(\alpha = 0.95\) (Ultra-Restrictive) 38.41 54.09 150 56.10 โ€” 71.02 โ€”
Threshold Ablation \(\alpha = 1.00\) (Base Un-tuned) 42.67 51.29 147 55.90 28.14 70.90 71.50

Key Findings

  • Model Scaling Fails to Cure Prior Adherence: Scaling parameters from 2B to 72B/78B across InternVL3, Qwen2.5-VL, and Qwen3.5 families reduces PAR modestly, yet adherence consistently plateaus above 30%~37%. In Scientific & Mathematic categories, larger models demonstrate equal or higher prior dogmatism.
  • CoT Reasoning Exacerbates Prior Vulnerability: Chain-of-Thought prompting systematically backfires, worsening Qwen3-VL-8B's PAR from 42.67% to 47.71% and crashing Filtered Accuracy down to 43.11%. Step-by-step token generation in the textual domain allows linguistic priors to dominate hidden activations, suppressing visual attention.
  • The Hidden Destructiveness of Prior Penalization: While Post-hoc logit penalization reports an attractive PAR of 32.05%, its FC metric jumps from 147 to 385 (revealing widespread collapse of basic text-only knowledge), and scores drop catastrophic 14.02 points on MMMU-Pro and 18.42 points on RealWorldQA. In contrast, MAPS holds FC steady at 154 and preserves all general benchmark scores within a fractional point of the base model.
  • Domain-Intrinsic Resistance Hierarchy: Prior susceptibility is fundamentally governed by semantic domain: Scientific & Mathematic triggers the highest PAR across all 24 models (exceeding 79% in smaller open models and 35% in GPT-5.5), whereas Physical Properties is the most receptive to visual correction (PAR 12%~40%).

Highlights & Insights

  • De-biasing the Benchmark via Dynamic Prior Induction: By demonstrating that cross-model prior agreement falls below 80% and over 15% of questions are solved without images, the authors replace flawed shared-prior assumptions with individualized blank-condition probing, preventing illusory claims of robustness.
  • Discovery of VIG Polarization: Through continuous log-odds tracking, the study identifies that scaling up models does not evenly increase perceptual adherence; instead, larger models bifurcate into extreme responses, either overpowering priors with massive confidence or fiercely defending their text priors.
  • Constructive Knowledge Enrichment Over Destructive Forgetting: Rather than forcing models to unlearn or penalizing prior logits, MAPS enriches the model's factual knowledge of rare real-world scenarios. Teaching the model that "exceptions exist in reality" allows it to gracefully defer to visual evidence when rare phenomena occur.

Limitations & Future Work

  • Static Taxonomy Coverage: The six commonsense domains are anchored to a standard ontology and do not encompass highly specialized scientific disciplines (e.g., medical pathology, specialized legal/technical schematics) or rapidly evolving real-time knowledge.
  • Token-Level Log-Odds Inaccessibility for Closed APIs: The continuous VIG metric requires token probability extraction, making it inapplicable to proprietary black-box APIs (such as GPT-5.5 and Claude Sonnet 4.6), leaving their internal confidence dynamics unverified.
  • Multi-Turn Contextual Drift: The investigation focuses exclusively on single-turn visual QA; how language priors dynamically reassert themselves over lengthy multi-turn conversational trajectories remains an unaddressed research question.
  • vs ROME / HallusionBench / VLind-Bench: Prior suites assumed fixed, universal counterfactuals and reported raw accuracy. ViSiP-Bench grounds stimuli in rare-but-real physical occurrences, adopts dynamic prior induction to evaluate models relative to their own beliefs, and filters out the >15% no-image correct artifacts.
  • vs Post-hoc / NaPO: Conventional debiasing methods penalize prior token probabilities, incurring severe capability damage across general benchmarks. MAPS employs a non-destructive knowledge enrichment strategy via targeted rare-fact fine-tuning, maintaining general benchmark fidelity.
  • vs Test-Time Interventions (VCD / PAI / AS): Inference-time logit subtracting or attention steering provides only marginal gains bounded by frozen representations, and CoT actively degrades prior resistance. MAPS achieves far more decisive prior softening with zero runtime inference overhead.

Rating

  • Novelty: โญโญโญโญโญ [Pioneering dynamic prior induction protocol, continuous VIG metric, and constructive knowledge softening paradigm.]
  • Experimental Thoroughness: โญโญโญโญโญ [Exhaustive evaluation across 24 models, 6 architectural families, controlled threshold ablations, and 4 general vision/reasoning benchmarks.]
  • Writing Quality: โญโญโญโญโญ [Extremely clear analytical exposition, rigorous breakdown of error sources, and elegant conceptual clarity.]
  • Value: โญโญโญโญโญ [Provides an essential diagnostic benchmark and an actionable, non-destructive alignment recipe for faithful multimodal perception.]