UC-VLM: Consistency-Driven Learning for AI-Generated Image Detection with Vision-Language Large Models

Conference: ECCV 2026
Paper: ECCV Official
PDF: ECCV PDF
Area: Multimodal VLM
Keywords: AI-Generated Image Detection / Vision-Language Large Models / Weakly Supervised Learning / Prompt Optimization / Forensic Consistency

TL;DR

Addressing the dual bottlenecks of existing vision-language models in AI-generated image detection—namely, over-reliance on human rationale annotations and neglect of low-level visual forensic cues—UC-VLM introduces a unified multi-stage framework trained solely on binary real/fake supervision. By combining stochastic cropping and patch shuffling for visual adaptation, LLM-driven instruction evolution, and label-conditioned text regeneration on a frozen vision tower, it achieves state-of-the-art accuracies of 96.1% on GenImage and 77.9% on Chameleon.

Background & Motivation

Rapid advancements in generative models, particularly diffusion models and generative adversarial networks, have enabled the synthesis of photorealistic images that convincingly mimic real-world photographs. While transforming digital content creation, this capability poses severe threats to digital media authenticity, intellectual property, and public trust. Conventional AI-generated image detectors formulate the problem as a purely visual binary classification task, relying on hand-crafted or specialized forensic indicators such as frequency artifacts, local texture distortions, lossless coding discrepancies, or reconstruction errors. However, these specialized discriminative models cannot interact via natural language or produce explanatory rationales, and their generalizability often degrades severely when confronted with unseen generators or high-resolution real-world test distributions.

Recent developments in Vision-Language Large Models (VLLMs) offer a promising avenue by providing unified multimodal perception and natural language generation. Nonetheless, current VLLM-based forensic systems (e.g., FakeVLM, FakeReasoning, AIGI-Holmes) face two fundamental limitations. First, because their vision towers are pre-trained on large-scale web data to capture macro semantic concepts, they inherently overlook subtle, non-semantic low-level forensic traces. Second, prior methods heavily depend on handcrafted prompts and labor-intensive human-annotated rationales, which severely restricts scalability across emerging generative pipelines. Furthermore, fine-tuning visual and linguistic modules in isolation yields brittle outcomes: vision-only tuning overfits to specific generator artifacts, language-only tuning improves textual fluency without reinforcing grounding on forensic evidence, and naive joint end-to-end tuning induces catastrophic inter-modal optimization conflict.

To break this impasse, the central inquiry of this paper is whether weak image-level binary authenticity labels (Authentic vs. Generated) alone can be transformed into a unified, shared supervisory driver across both visual perception and textual generation. Core idea: reuse the same binary authenticity labels across a multi-stage framework by enforcing local non-semantic visual adaptation via stochastic cropping and patch shuffling, discovering robust structured instructions through LLM-driven evolutionary search, and supervising textual outputs via label-conditioned regeneration on a frozen vision tower, thereby attaining superior discrimination and stable reasoning without any human rationales.

Method

Overall Architecture

UC-VLM is built upon the open-source vision-language model Qwen2.5-VL-7B. Given an input image \(I\), the objective is to predict its authenticity label \(y \in \{\text{Authentic}, \text{Generated}\}\) and produce an accompanying structured forensic rationale, relying strictly on binary image-level labels. The overall learning framework consists of three tightly coupled stages: first, input images undergo stochastic cropping and patch shuffling before being fed to the vision encoder, which is fine-tuned with LoRA under a binary cross-entropy loss to suppress global semantics in favor of local forensic artifacts; second, an offline evolutionary prompt search guided by an LLM (GPT-4o) evaluates candidate instruction variations on a validation set and extracts an optimal four-step structured prompt (Prompt-B); third, the model generates pseudo-rationales, prepending an explicit ground-truth directive whenever an initial prediction is erroneous to re-sample correct explanations, and fine-tunes the language module via autoregressive loss while freezing the adapted vision tower.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Binary Authenticity Label"] --> B["Visual Adaptation<br/>Stochastic Cropping + Patch Shuffling + BCE Loss"]
    B --> C["Instruction Optimization<br/>Candidate Pool Init → Validation Evaluation → Mutation Rewriting"]
    C --> D["Optimal Evaluation Query Prompt-B"]
    D --> E["Label-Conditioned Regeneration<br/>Correction Resampling → Pseudo-Rationale Dataset → Autoregressive Tuning"]
    E --> F["Final Authenticity Decision & 4-Step Rationale"]

Key Designs

1. Visual Adaptation: Suppressing Global Semantics and Elevating Microscopic Forensic Cues Standard VLLM vision encoders naturally prioritize high-level semantic abstractions (e.g., identifying object categories), which leaves them vulnerable when evaluating synthetic images where semantic composition is plausible but microscopic synthesis traces are aberrant. To redirect the visual encoder toward low-level non-semantic cues, UC-VLM subjects the input image \(I \in \mathbb{R}^{H \times W \times 3}\) to two spatial disruption operations. First, a local sub-region \(I'\) of size \(h' \times w'\) is stochastically cropped, where \(h' \sim U(\frac{1}{8}H, \frac{1}{4}H)\) and \(w' \sim U(\frac{1}{8}W, \frac{1}{4}W)\), stripping away broad contextual layouts. Second, the crop is partitioned into \(N\) non-overlapping patches \(\{p_i\}_{i=1}^N\) and reassembled into a shuffled image \(\tilde{I}' = \text{Reassemble}(\{p_{\pi(i)}\}_{i=1}^N)\) using a random permutation \(\pi \in \mathcal{P}(N)\). Patch shuffling shatters spatial semantic continuity, forcing the network to extract discriminative signals exclusively from patch boundary discontinuities, unnatural local textures, and high-frequency discrepancies. The vision tower (configured with LoRA rank 16) is optimized directly using binary cross-entropy: $\(\mathcal{L}_{vis} = - \left[ y \log \hat{y} + (1 - y) \log (1 - \hat{y}) \right]\)$

2. Instruction Optimization: Discovering Robust Structured Prompts via Evolutionary Search Empirical investigation demonstrates that VLLM predictions exhibit pronounced sensitivity to prompt phrasing; for example, merely changing "Authentic/Generated" to "Real/Fake" causes a 4.3% shift in zero-shot accuracy, rendering manual prompt design sub-optimal and fragile. UC-VLM overcomes this with an offline evolutionary optimization algorithm. Starting from a base template, an LLM initializes an instruction pool of \(N=5\) candidates \(P_0\). In each iteration \(t\), candidates are evaluated on a VLM validation subset to record authenticity accuracy, and the Top-\(k\) (\(k=2\)) performers are retained. Randomly selected parent prompts then undergo either semantic rewriting (preserving meaning while varying vocabulary) or reasoning-expression modification (altering structural flow) with equal probability (0.5) to populate child instructions. After \(M=10\) evolution rounds, the search converges onto an optimal prompt Prompt-B, which automatically establishes a disciplined four-step forensic protocol: - [Step-1] Visual Examination: Identify 2-3 aspects of unnatural lens distortions/glitches alongside convincing realistic details; - [Step-2] Statistical Analysis: Examine anomalies in the frequency domain that suggest synthetic manipulation or corroborate authenticity; - [Step-3] Semantic & Physics Consistency: Inspect the scene for depth perception conflicts, misaligned perspectives, or lighting/shadow discrepancies; - [Step-4] Final Assessment: Output a single-word conclusion ('Generated' or 'Authentic') and summarize pivotal evidence.

3. Label-Conditioned Regeneration: Aligning Language-Side Generation Under Binary Supervision Without costly human-annotated chain-of-thought rationales, querying a VLLM for explanations frequently triggers severe hallucination. UC-VLM resolves this by repurposing the ground-truth binary label as an automated error corrector. For each sample \(x_j\), the model generates an initial rationale and prediction \((r_j, \hat{y}_j)\) using Prompt-B. If \(\hat{y}_j\) matches the ground truth \(y_j\), the rationale is directly adopted as a reliable pseudo-rationale. If incorrect, a corrective label directive \(d_j\) is prepended (e.g., "This image is 'Generated' (AI-generated). Explain why this image is AI-generated, not authentic." + Prompt-B), constraining the model to regenerate a plausible explanatory chain \(r'_j\) conditioned on the true label. The resulting regeneration dataset is compiled as: $\(\mathcal{D}_{regen} = \{(x_j, r_j, y_j) \mid \hat{y}_j = y_j\} \cup \{(x_j, r'_j, y_j) \mid \hat{y}_j \neq y_j\}\)$ Crucially, the visual encoder remains completely frozen, and only the language module parameters (LoRA rank 8) are updated using an autoregressive next-token objective: $\(\mathcal{L}_{text} = - \sum_{(x, r, y) \in \mathcal{D}_{regen}} \log p_\theta(r \mid x, y, p_{best})\)$ The authors transparently clarify that these generated rationales serve as internal consistency regularizers rather than ground-truth physical verifications, yet they effectively bridge binary supervision to language space.

Loss & Training

The formal objective is denoted as \(\mathcal{L} = \mathcal{L}_{vis} + \mathcal{L}_{text}\). However, empirical ablations reveal that joint end-to-end tuning of visual and language components induces cross-modal optimization interference, causing performance on Chameleon to plunge to 52.0%. Consequently, UC-VLM adopts a strictly decoupled multi-stage protocol: 1. Stage 1 (Visual Adaptation): LoRA (rank 16) is attached to the vision tower of Qwen2.5-VL-7B and trained for 1 epoch using SGD (learning rate \(1 \times 10^{-4}\), batch size 32) on patch-shuffled crops; 2. Stage 2 (Instruction Optimization): Offline evolutionary search runs for \(M=10\) rounds with GPT-4o to identify and freeze Prompt-B; 3. Stage 3 (Label-Conditioned Regeneration Tuning): With the vision tower frozen, LoRA (rank 8) is attached to the language model and fine-tuned on \(\mathcal{D}_{regen}\) for 1 epoch using AdamW with learning rate \(1 \times 10^{-4}\). All experiments execute efficiently on a single NVIDIA A100 GPU (FP16).

Key Experimental Results

Main Results

UC-VLM is evaluated on two prominent benchmarks: GenImage (comprising 2.68 million images across eight distinct diffusion and GAN generators) and Chameleon (a challenging test-only suite containing 26,000 high-resolution images from 720P to 4K). Classification accuracy (ACC %) is reported following standardized protocols.

Method	Venue	GenImage 8-Model Avg (%)	Chameleon (ProGAN Trained) (%)	Chameleon (SDV1.4 Trained) (%)
ResNet-50	CVPR'16	72.1	-	-
DeiT-S	ICML'21	71.6	-	-
Swin-T	ICCV'21	74.8	-	-
CNNSpot	CVPR'20	64.2	56.9	60.1
DIRE	ICCV'23	72.7	58.2	59.7
FatFormer	CVPR'24	87.4	-	-
NPR	CVPR'24	88.6	57.3	58.1
DRCT	ICML'24	89.5	-	-
VIB-Net	CVPR'25	88.9	-	-
AIDE	ICLR'25	86.9	58.4	62.6
DEUA	ICCV'25	91.5	-	-
FIND	AAAI'26	88.4	-	-
UC-VLM (Ours)	ECCV 2026	96.1	69.6	77.9

Across the eight generator subsets of GenImage, UC-VLM demonstrates exceptional cross-generator generalizability: Midjourney (90.1%), SDV1.4 (99.9%), SDV1.5 (99.8%), ADM (87.0%), GLIDE (98.5%), Wukong (96.9%), VQDM (98.1%), and BigGAN (98.8%), consistently establishing top-tier performance. On the challenging Chameleon zero-shot benchmark, UC-VLM surpasses the strongest baseline AIDE by +11.2% under ProGAN training and +15.3% under SDV1.4 training.

Ablation Study

The ablation experiments systematically isolate the contributions of visual adaptation, instruction optimization, and label-conditioned regeneration:

Config	GenImage Accuracy (%)	Chameleon Accuracy (%)	Note
Baseline Backbone (Qwen2.5-VL-7B Zero-Shot)	51.9	55.5	Generic VLLM lacks innate forensic sensitivity
Baseline (Visual Tuning Only)	76.8	50.9	Vision-only tuning lacks semantic-language anchoring
Baseline (LLM Tuning Only)	80.3	71.0	Exploits language reasoning but bounded by vision cues
Baseline (Naive Joint Visual + LLM Tuning)	77.6	52.0	Simultaneous joint tuning triggers optimization conflict
Frozen Backbone + Instruction Optimization	74.2	68.3	Structured prompt unlocks latent forensic capacity
Visual Baseline + Visual Adaptation	91.7	55.4	Shuffled crops force attention to micro-textures (+14.9%)
LLM Baseline + Label-Conditioned Regeneration	82.4	65.1	Error-corrected resampling enhances language supervision
UC-VLM (Full Multi-Stage Framework)	96.1	77.9	Optimal synergy across vision, prompt, and language

Key Findings

• Collapse of Naive Joint Fine-Tuning: Simultaneously updating both the visual encoder and language module degrades Chameleon accuracy to 52.0% (inferior to the 55.5% zero-shot baseline). This empirically validates the paper's multi-stage decoupling design: adapting the vision pathway first and subsequently freezing it during language refinement avoids gradient interference.
• Outsized Dividends from Instruction Search: Without modifying any model parameters, evolutionary prompt optimization boosts zero-shot Qwen2.5-VL-7B accuracy from 51.9% to 74.2% (+22.3%) on GenImage and to 68.3% (+12.8%) on Chameleon. This underscores how heavily multimodal reasoning models depend on structured task querying.
• Visual Representation Realignment: Introducing stochastic cropping and patch shuffling propels the visual baseline from 76.8% to 91.7% on GenImage (+14.9%), demonstrating that disrupting global coherence is an exceptionally cost-effective technique to reorient general-purpose vision backbones toward forensic-grade artifact detection.

Highlights & Insights

• Maximizing Utility of Weak Binary Supervision: Bypasses the costly bottleneck of manual chain-of-thought rationales and manipulation masks by reusing the ubiquitous binary authenticity label across three coordinated phases: patch-shuffled visual adaptation, validation-driven prompt evaluation, and error-directed text regeneration.
• Systematic LLM-Driven Prompt Evolution: Successfully transitions the prompt from a disjointed list of generic forensics (PRNU, FFT shifts, JPEG blocks) to a coherent, prioritized four-step protocol targeting lens aberrations, frequency anomalies, perspective depth, and summary judgments.
• Pragmatic Assessment of Explanatory Faithfulness: The authors explicitly caution that natural language rationales supervised solely by binary labels are auxiliary internal consistency mechanisms rather than infallible physical proofs, demonstrating rare scientific rigor regarding model hallucination.

Limitations & Future Work

• Vulnerability to Hyper-Realistic and Stylized Artwork (Failure Cases): Qualitative failure analysis indicates that when synthetic portraits exhibit near-flawless facial micro-textures and lighting geometry, the model can mistake them for authentic photographs. Conversely, authentic artistic paintings and illustrations lacking standard perspective geometry are frequently misclassified as AI-generated.
• Offline Prompt Evolution Overhead: The evolutionary prompt search requires 10 iterations of candidate mutations via GPT-4o and iterative validation evaluations, which incurs noticeable offline latency and cost when rapidly adapting to newly released generative architectures.
• Potential Destruction of Global Physical Context: While patch shuffling effectively filters out high-level semantic distractions, it inherently disrupts scene-wide lighting trajectories and long-range shadow projections, suggesting that future iterations could benefit from a multi-scale hybrid scheme retaining topological illumination cues.

Related Work & Insights

• vs. Conventional Forensics Detectors (NPR, DIRE, DEUA): Traditional detectors utilize frequency transforms, diffusion inversion errors, or upsampling discrepancies. While potent on matched distributions, they degrade steeply on unseen generators and output only an uninterpretable scalar probability. UC-VLM harnesses rich multimodal priors to achieve superior 96.1% accuracy while delivering structured, human-readable forensic analyses.
• vs. Existing VLM Forensics Frameworks (FakeVLM, FakeReasoning, AIGI-Holmes): FakeVLM and AIGI-Holmes depend heavily on manual prompt engineering, expert-annotated rationale datasets, or DPO preference alignment; FakeReasoning requires dual visual backbones (CLIP + DINO). UC-VLM demonstrates that a single VLLM backbone trained with parameter-efficient LoRA under pure binary supervision can outperform complex multi-encoder and heavily annotated baselines.

Rating

• Novelty: ⭐⭐⭐⭐☆ (Ingenious closed-loop reuse of binary labels across visual adaptation and language regeneration)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive validation across GenImage's 8 generators and Chameleon's 26K high-resolution samples)
• Writing Quality: ⭐⭐⭐⭐⭐ (Lucid exposition, rigorous empirical motivation, and transparent boundary definitions)
• Value: ⭐⭐⭐⭐⭐ (Provides a highly practical, low-cost engineering blueprint for scaling multimodal image forensics)

Related Papers

• [ICCV 2025] AIGI-Holmes: Towards Explainable and Generalizable AI-Generated Image Detection via Multimodal Large Language Models
• [ECCV 2026] Token-Based Affordance Grounding with Large Vision-Language Models
• [ECCV 2026] Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models
• [ECCV 2026] Dual-Generalization-aware Minimization for Continual Fine-Tuning of Vision-Language Models
• [ECCV 2026] PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided VLM