EGVLR: Evidence-Grounded Vision–Language Reinforcement for Anomaly Reasoning¶

Conference: ECCV 2026
Paper: ECCV Official
Area: LLM Reasoning
Keywords: Industrial Anomaly Reasoning, Multimodal Large Language Models, Reinforcement Learning, Visual Grounding, Preference Optimization

TL;DR¶

Addressing language-prior shortcuts, contradictory reasoning traces, and hallucinated defect boxes on normal samples in industrial MLLMs, this paper proposes EGVLR, a reinforcement framework unified under an Evidence-Driven Diagnostic Protocol (EDDP) across progressive visual pre-alignment, knowledge-grounded instruction tuning, decoupled preference optimization, and box-guided segmentation rendering, achieving state-of-the-art anomaly reasoning while slashing spatial false positives.

Background & Motivation¶

Industrial anomaly inspection is transitioning from closed-set defect classification and pixel-level heatmap scoring toward interactive, natural-language-driven anomaly understanding. In conventional automated inspection pipelines, detectors simply output an anomaly score or a rough saliency map; human inspectors must still determine the defect category, verify whether highlighted regions are false alarms, and scrutinize how the query deviates from a golden normal sample. Recent multimodal large language models (MLLMs) open up a far more interactive paradigm: they can receive inspection images, accept open-ended queries, compare a test product against normal references, and provide structured diagnostic reasoning. Benchmarks such as MMAD have codified this demand by evaluating models across defect classification, localization, description, and root-cause analysis.

However, deploying general MLLMs to industrial quality inspection reveals a fundamental reliability gap. Industrial defects—such as microscopic hairline cracks, subtle missing solder joints, tiny contaminants, or minor surface abrasions—are inherently sparse, local, and visually subtle, occupying only a tiny fraction of the overall image. Because general foundation models are pre-trained primarily on web-scale image-text datasets dominated by coarse global semantics, they lack fine-grained spatial grounding and domain-specific inspection logic. Consequently, when asked to diagnose defects, MLLMs often rely on language priors and shortcut heuristics. This creates severe "evidence inconsistency" across answers, locations, and rationales: models frequently guess the correct defect category while predicting an invalid bounding box, describe imaginary scratches simply because the prompt asks about flaws, or hallucinate defect bounding boxes on completely normal products. In production lines, such spatial false positives on normal items flood human operators with invalid alarms and undermine operational trust.

Standard supervised fine-tuning (SFT) fails to resolve this issue because token-level imitation cannot penalize contradictory outputs, such as a normal classification coupled with an abnormal bounding box. Conversely, directly applying reinforcement learning on base MLLMs triggers training instability, format collapse, and spatial reward hacking. The core idea of this paper is: unify the entire diagnostic process under a structured Evidence-Driven Diagnostic Protocol (EDDP), systematically decoupling the alignment into progressive visual evidence pre-alignment, knowledge-grounded instruction tuning, geometry-semantic decoupled preference optimization (GS-DPO), and box-guided segmentation rendering, thereby eliminating spatial hallucinations and logical contradictions at their root.

Method¶

Overall Architecture¶

EGVLR is built upon the unified Evidence-Driven Diagnostic Protocol (EDDP), which enforces a shared structural contract across all trainable stages. Every response generated by the MLLM is parsed into four explicit fields: visual evidence <evidence>, diagnostic logic <logic>, spatial location <location>, and the final choice <answer>. For comparative inspection, a strict paired-image convention is maintained throughout: the first image is always the query/test image, and the second image is the normal reference image from the identical product class. Crucially, the <location> field exclusively stores normalized bounding boxes \([x_1, y_1, x_2, y_2]\) in the \([0, 1000]\) coordinate space; for normal samples, decoy negative regions, or non-spatial queries, the model is strictly required to output the empty list <location>[]</location>. This prevents syntactic confusion between textual grid labels, continuous coordinates, and null decisions.

The comprehensive pipeline progresses through four sequential stages: first, Progressive Visual-Evidential Fine-Tuning (PVE-FT) establishes spatial grounding and null-hypothesis retention on purely synthetic visual anomalies without textual shortcuts; second, Knowledge-Grounded Instruction Tuning (KG-IT) injects domain-specific terminology, inspection QA, and comparative reasoning while preserving the exact EDDP schema; third, Geometry-Semantic Decoupled Preference Optimization (GS-DPO) leverages a five-channel GRPO objective to eliminate logic-location-answer contradictions; finally, Box-Guided Segmentation Rendering (BGSR) passes verified bounding boxes to an off-the-shelf segmentation backend (such as SAM3) to produce dense pixel masks without altering MLLM reasoning.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Image + Normal Reference<br/>Strict EDDP Schema Contract"] --> B["Stage I: Progressive Visual-Evidential Fine-Tuning<br/>Synthetic Defect & Grid Pre-Alignment"]
    B --> C["Stage II: Knowledge-Grounded Instruction Tuning<br/>Domain Knowledge & Comparative QA"]
    C --> D["Stage III: Geometry-Semantic Decoupled Preference Optimization<br/>Multi-Channel GRPO Alignment"]
    D --> E["Generate Structured Diagnostic Response<br/>Evidence + Logic + Normalized Box + Answer"]
    E -->|Non-Empty Box| F["Stage IV: Box-Guided Segmentation Rendering<br/>Frozen SAM3 Backend Generates Pixel Mask"]
    E -->|Empty Box []| G["Classified as Normal: Suppress Mask Rendering"]

Key Designs¶

1. Progressive Visual-Evidential Fine-Tuning: Grounding Localized Visual Evidence and Null-Hypothesis Calibration

To overcome MLLM vulnerability to language-prior shortcuts on subtle defects, this stage constructs a vision-centric pre-alignment task free from external semantic leakage. Starting from pristine industrial normal images, localized defects are synthesized using CutPaste patch replacement and DTD texture insertion. The generator programmatically logs ground-truth defect masks, normalized boxes, coarse \(3 \times 3\) grid indices, and fine subregion identifiers, synthesizing five programmatic visual QA families: anomaly detection, \(3 \times 3\) grid localization, fine subregion identification, local verification with positive and decoy patches, and null-hypothesis calibration on normal references. For all normal images and negative decoy patches, the target label is rigorously constrained to <location>[]</location>. By training via autoregressive cross-entropy \(\mathcal{L}_{\mathrm{PVE}}\), the visual backbone and coordinate tokens internalize fine-grained visual discrepancy and develop the instinct to predict empty locations when visual evidence is absent.

2. Knowledge-Grounded Instruction Tuning: Internalizing Domain Semantics and Comparative Logic

Following visual-evidential pre-alignment, KG-IT transitions the model from basic spatial perception to high-level domain diagnostic cognition. The model is supervised on a tripartite data blend: domain-specific QA derived from industrial operational standards, single-image normal product QA, and query-reference comparative inspection QA, mixed in a 50:35:15 ratio. Crucially, the EDDP structural schema is strictly preserved: queries lacking bounding-box annotations still output <location>[]</location> with an internal non-spatial flag. This prevents schema drift between training stages and equips the model with domain terminology, failure-mode categorization, and reference-guided comparative reasoning without degrading coordinate formatting.

3. Geometry-Semantic Decoupled Preference Optimization: Eliminating Contradictions and Spatial Reward Hacking

Token-level imitation in SFT cannot explicitly penalize subtle contradictions, such as predicting a correct option using an invalid bounding box or generating reasoning that contradicts the final answer. GS-DPO adopts a critic-free GRPO reinforcement learning framework, sampling a group of \(G\) candidate outputs per prompt and evaluating group-normalized advantages over a decoupled five-channel reward function:

\[R = \lambda_f R_{\mathrm{fmt}} + \lambda_a R_{\mathrm{ans}} + \lambda_{\mathrm{box}} R_{\mathrm{box}} + \lambda_{\mathrm{bge}} R_{\mathrm{bge}} + \lambda_{\mathrm{cpl}} R_{\mathrm{cpl}}\]

Here, \(R_{\mathrm{fmt}}\) rewards parseable EDDP tags, and \(R_{\mathrm{ans}} = \mathbb{I}(a_{\mathrm{pred}} = a_{\mathrm{gt}})\). The spatial reward \(R_{\mathrm{box}}\) evaluates IoU/DIoU matching against ground truth with penalties for redundant boxes on abnormal samples, while returning \(+1\) for empty predictions and \(-1\) for any hallucinated box on normal samples. The semantic rationale reward \(R_{\mathrm{bge}}\) encodes the generated <evidence> and <logic> strings with BAAI/bge-small-en-v1.5, computing maximum cosine similarities \(s_N\) and \(s_A\) against normal and abnormal prototype banks to supply a smooth directional reward via \(R_{\mathrm{bge}} = \tanh(\gamma(s_A - s_N))\). Finally, the coupling reward \(R_{\mathrm{cpl}} = \mathbb{I}(a_{\mathrm{pred}} = a_{\mathrm{gt}}) \cdot \mathbb{I}(\mathrm{LocValid}) \cdot \mathbb{I}(\mathrm{SemValid})\) activates only when the answer is correct, the spatial prediction is valid (non-empty and accurate for abnormal; strictly empty for normal), and the semantic rationale direction fully corroborates the choice. This multi-channel design prevents spatial reward hacking via oversized bounding boxes and penalizes self-contradictory reasoning traces.

4. Box-Guided Segmentation Rendering: Decoupling Cognitive Reasoning from Pixel-Level Masking

To provide dense pixel-level defect masks for industrial human-machine interfaces without encumbering the language model with heavy segmentation tokens, BGSR treats dense mask generation strictly as an external rendering task. The MLLM is tasked purely with cognitive reasoning and normalized bounding-box localization. If the predicted location is empty, the rendering pipeline terminates immediately, ensuring zero false-positive masks on normal products. If non-empty boxes are predicted, they are rescaled to original image dimensions and fed as prompt boxes into a frozen, off-the-shelf segmentation backend (such as SAM3). This architectural separation preserves MLLM training efficiency, avoids attributing mask generation quality to cognitive reasoning, and keeps the inspection trace clean and inspectable.

Key Experimental Results¶

Main Results¶

On the comprehensive industrial multimodal anomaly benchmark MMAD (evaluating 1-shot anchor-guided defect classification, localization, description, and root-cause analysis), EGVLR instantiated on Qwen3-VL-8B-Instruct is compared against leading commercial models, open-source general MLLMs, and specialized anomaly baselines.

Model	Scale	Anomaly Discrim.	Defect Class.	Defect Local.	Defect Desc.	Defect Analysis	Object Class.	Object Analysis	Average
GPT-4o	-	68.63	65.80	55.62	73.21	83.41	94.98	82.80	74.92
Gemini 2.5 Pro	-	83.07	73.86	67.20	79.97	86.27	94.88	83.08	81.19
Qwen2.5-VL-Instruct	7B	71.39	54.35	61.17	65.81	79.32	91.44	84.43	72.56
LLaVA-OneVision-1.5	8B	60.85	56.29	55.25	74.73	83.21	91.03	89.14	72.93
Qwen3-VL-Instruct (Base)	8B	69.75	59.34	59.08	73.54	81.29	90.30	89.07	74.62
AnomalyR1	7B	60.93	64.81	70.72	79.06	85.52	93.12	86.91	77.29
OmniAD	7B	68.80	78.80	75.50	67.20	86.40	96.00	86.40	79.90
JUDO	7B	64.51	72.17	75.95	84.38	87.76	94.24	86.07	80.73
EGVLR (Ours)	8B	71.50	78.75	76.85	84.42	87.82	93.30	85.21	82.55

Ablation Study¶

The ablation evaluation dissects the impact of progressive training stages and individual GS-DPO reward components on average accuracy (Avg. Acc.), defect localization accuracy (Loc. Acc.), format adherence (Format), normal image-level false-positive rate (\(\mathrm{FPR}_{\mathrm{img}} \downarrow\)), and normal average false-positive box count (\(\mathrm{FPBox} \downarrow\)).

Config	Avg. Acc.	Loc. Acc.	Format (%)	\(\mathrm{FPR}_{\mathrm{img}}\) (%) \(\downarrow\)	\(\mathrm{FPBox} \downarrow\)	Note
Stage Ablation
Base MLLM (Qwen3-VL-8B)	74.62	59.08	58.7	42.6	0.78	Raw base model with weak localization and high hallucinations
+ PVE-FT	77.84	71.90	94.6	29.4	0.47	Pre-alignment activates localized spatial perception and null calibration
+ PVE-FT + KG-IT	80.63	75.42	97.3	23.8	0.34	Injects domain-specific QA and comparative inspection logic
+ PVE-FT + KG-IT + GS-DPO	82.55	76.85	98.6	13.9	0.18	Full pipeline suppresses contradictions and false-positive boxes
Reward Channel Ablation
Answer-only GRPO	80.73	72.34	94.8	31.7	0.53	Answer-only reward reinforces language-prior shortcuts
Standard GRPO	81.62	75.02	97.5	24.6	0.37	Undecoupled baseline achieves suboptimal consistency
w/o box reward	80.41	71.72	98.2	30.5	0.49	Removing spatial reward causes severe localization degradation
w/o BGE rationale reward	81.86	75.91	98.4	20.7	0.31	Removing semantic embedding guidance increases reasoning drift
w/o semantic coupling	81.94	76.18	98.3	19.3	0.28	Removing coupling reward triggers answer-location-logic detachment
Full GS-DPO	82.55	76.85	98.6	13.9	0.18	Optimal performance across accuracy, localization, and hallucination

Key Findings¶

Progressive Staging Provides Strong Complementary Gains: Progressing from Base to PVE-FT, KG-IT, and GS-DPO continuously improves localization accuracy from 59.08% to 76.85%, while drastically decreasing image-level false-positive rate \(\mathrm{FPR}_{\mathrm{img}}\) on normal samples from 42.6% down to 13.9%. This underscores that visual evidence pre-alignment is indispensable before high-level domain QA tuning.
Null Calibration and Coupling Rewards are Essential to Suppress Hallucinations: Without null-hypothesis calibration or semantic coupling rewards, models default to predicting non-empty boxes to exploit recall gains. GS-DPO successfully reduces the average false-positive box count on normal images from 0.78 to 0.18.
Compensating for Native Base Model Weaknesses: The raw Qwen3-VL-8B base model exhibited lower native localization accuracy (59.08%) than Qwen2.5-VL-7B (61.17%). However, post-training with EGVLR elevates its localization performance to 76.85%, proving that evidence-grounded alignment rectifies the underlying foundation model's spatial deficiencies.

Highlights & Insights¶

Unified EDDP Structural Protocol: Enforcing <evidence>, <logic>, <location>, and <answer> alongside a canonical <location>[]</location> representation for normal/non-spatial samples establishes a clean, unified contract across pre-alignment, instruction tuning, RL reward formulation, and downstream mask rendering.
Decoupled Reward with Semantic Prototypes: Moving beyond scalar answer rewards, the combination of BGE prototype cosine similarities \(\tanh(\gamma(s_A - s_N))\) and the multi-field coupling indicator reward prevents spatial reward hacking via oversized bounding boxes while ensuring strict trace consistency.
Decoupling Cognitive Reasoning from Pixel Rendering: Entrusting the MLLM strictly with evidence-based reasoning and bounding-box prediction while delegating mask generation to SAM3 circumvents the burden of fitting dense spatial tokens inside language model architectures.

Limitations & Future Work¶

Omission on Low-Contrast, Microscopic Defects: The authors acknowledge that extremely minute defects or subtle blemishes blending seamlessly into natural surface textures can still be missed.
Salience Bias in Multi-Instance Anomaly Scenarios: In images containing multiple disparate anomalies, the model tends to focus on the single most visually salient defect, occasionally missing secondary localized flaws.
Inherent Limits of Embedding-Based Semantic Rewards: While BGE cosine scoring outperforms simplistic keyword matching, it does not perform full natural-language inference (NLI) and can struggle with nuanced syntactical negations or ambiguous rationales.

vs AnomalyGPT / FabGPT: Early industrial MLLMs relied primarily on prompt tuning or LoRA to project visual features into conversational tokens, lacking explicit spatial grounding contracts and null-hypothesis calibration; EGVLR introduces rigorous visual pre-alignment and decoupled RL to guarantee verifiable spatial-semantic traces.
vs OmniAD / JUDO: While recent competitive methods often blend defect localization and domain QA into a single SFT/RL stage, risking language-prior dominance over visual perception, EGVLR decouples the curriculum into visual grounding, domain QA, and multi-channel preference optimization, establishing a robust framework for mission-critical industrial inspection.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Proposes a clean, principled EDDP schema and a decoupled five-channel GS-DPO alignment framework specifically tailored to eliminate industrial spatial hallucinations.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducts extensive evaluations across all MMAD task axes, detailing stage contributions, reward ablations, false-positive metrics, and qualitative failure analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Rigorous organization, transparent problem framing, concise mathematical formulations, and compelling technical motivation throughout.
Value: ⭐⭐⭐⭐⭐ Offers an actionable blueprint for deploying multimodal foundation models to high-stakes industrial inspection and safety-critical vision-language reasoning.