Skip to content

title: >- [Paper Note] Break Visual-Linguistic Asymmetry: Unleashing VLM's Cross-Modal Potential for General Face Forgery Detection description: >- [ECCV 2026][Multimodal VLM][Deepfake Detection] Alleviating visual-linguistic feature evolution asymmetry in VLMs via multi-level feature alignment and asynchronous gradient interaction. tags: - ECCV 2026 - Multimodal VLM - Deepfake Detection - Face Forgery Detection - Representation Learning date: 2026-09-19 content_hash: 258a3e96c0828ef8

Break Visual-Linguistic Asymmetry: Unleashing VLM's Cross-Modal Potential for General Face Forgery Detection

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Face Forgery Detection, Vision-Language Models, Cross-Modal Asymmetry, Parameter-Efficient Fine-Tuning, Cross-Modal Interaction

TL;DR

Addressing the inherent cross-modal and intra-modal visual-linguistic asymmetry of vision-language models (VLMs) in face forgery detection, this paper introduces Asy-Det—an asymmetry-guided parameter-efficient detection framework combining a two-phase Meta Forgery Adapter (MFA) and an Asymmetry-Guided Interactor (AGI) to achieve multi-granularity bidirectional cross-modal alignment, delivering an image-level AUC of 92.94% on unseen Celeb-DF-v2 with only 4.41M trainable parameters.

Background & Motivation

Face manipulation techniques driven by generative adversarial networks (GANs) and diffusion models have achieved unprecedented visual fidelity, producing synthetic face images that pose critical threats to personal privacy, media authenticity, and digital security. To counter the proliferation of malicious manipulations, diverse face forgery detection (FFD) approaches have been developed. However, conventional detectors predominantly rely on purely visual cues—focusing on high-frequency signals or low-level spatial artifacts such as blending boundaries and unnatural textures. Consequently, they suffer from severe generalization bottlenecks, deteriorating drastically when encountering unseen manipulation algorithms or generation pipelines in real-world scenarios.

To overcome this generalization bottleneck, state-of-the-art methods have turned toward adapting large pre-trained Vision-Language Models (VLMs, e.g., CLIP). Given the limited size and diversity of typical face forgery training corpora, existing VLM-based detectors mostly adopt parameter-efficient adaptation paradigms—such as prompt tuning or lightweight adapters—keeping the underlying VLM backbone frozen. Nonetheless, these approaches either restrict adaptation exclusively to the visual modality or optimize vision and language branches completely independently prior to final contrastive classification, ignoring cross-modal interplay at intermediate representations. Crucially, directly applying off-the-shelf symmetric and implicit multi-modal interaction frameworks (e.g., CoCoOp, MaPLe, MMA) yields suboptimal results that even lag behind unimodal vision detectors.

The root cause of this failure lies in the overlooked visual-linguistic asymmetry inherent to pre-trained VLMs. This asymmetry manifests along two dimensions: cross-modal asymmetry, where visual and linguistic latent spaces at identical layer depths exert opposite impacts on generalizability (shallow text representations foster generalization whereas shallow vision features harm it, and vice versa in deeper layers); and intra-modal asymmetry, where representations at different semantic levels display inverse trade-offs between fine-grained discriminability and cross-domain generalizability. The core idea is to uncover and exploit this visual-linguistic asymmetry via an asymmetry-guided detector (Asy-Det), orchestrating phased intra-modal meta-adaptation alongside multi-granularity asynchronous cross-modal mutual learning to unleash the latent potential of VLMs for general face forgery detection.

Method

Overall Architecture

Asy-Det builds on top of a frozen pre-trained CLIP backbone, keeping all visual and text encoder parameters fixed during training to prevent the degradation of foundational cross-modal representations. To mitigate intra-modal asymmetry, both the vision and linguistic encoders are partitioned into two equal phases equipped with Meta Forgery Adapters (MFAs): the first phase specializes in capturing fine-grained forgery-specific clues from shallow layers to maximize discriminability, while the second phase excites global forgery-general semantics in deep layers to foster cross-domain transferability. Concurrently, the Asymmetry-Guided Interactor (AGI) aggregates multi-granularity features via an asynchronous registration mechanism and executes explicit bidirectional cross-attention alignment, re-injecting balanced multimodal context back into the backbone feature streams via residual connections.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Category Prompts"] --> B["Stage 1: Shallow Feature Extraction & MFA Discrimination Boosting<br/>Capture fine-grained local forgery cues"]
    B --> C["Multi-Granularity Asynchronous Registration & Aggregation<br/>Construct shallow shared latent space"]
    C --> D["Stage 2: Asymmetry-Guided Interactor AGI<br/>Bidirectional cross-attention explicit alignment"]
    D --> E["Stage 3: Deep Feature Encoding & MFA Generalization Boosting<br/>Excite forgery-invariant high-level semantics"]
    E --> F["Shared Latent Space Cosine Similarity Binary Classification"]

Key Designs

1. Meta Forgery Adapter (MFA): Decoupling Discrimination and Generalization Across Phased Adapters To resolve the intra-modal conflict where low-level artifact discrimination and high-level domain generalization inherently oppose each other, MFA is integrated across the divided encoder phases with two parallel branches: a forgery adaptation branch and a meta generation branch. In the forgery adaptation branch, intermediate visual features \([f_g^q, f_l^q]\) pass through a lightweight adapter \(\mathcal{FA}\) formed by grouped fully-connected layers and ReLU activations, balanced by a learnable scaling factor \(\alpha\): $\([\hat{f}_g^q, \hat{f}_l^q] = [f_g^q, f_l^q] + \alpha \cdot \mathcal{FA}(V_q([f_g^q, f_l^q]))\)$ This controlled residual integration prevents catastrophic alteration of pre-trained multimodal weights while tuning shallow stages specifically toward local boundary and frequency discrepancies, and orienting deep stages toward semantic, forgery-invariant patterns.

2. Multi-Granularity Asynchronous Registration: Building Layer-Aligned Cross-Modal Spaces Standard VLM adaptation schemes only project representations into a single shared embedding space at the terminal layer, which precludes cross-modal information sharing throughout intermediate representations. The meta generation branch in MFA bridges this gap by applying convolutions and an asynchronous register \(\Pi\) on intermediate patch tokens and prompt embeddings, generating shared visual tokens \(f_{ssv}^q = \Pi(\text{ReLU}(\text{Conv}(f_l^q)))\) and shared linguistic tokens \(w_{sst}^\ell = \Pi(\text{ReLU}(\text{Conv}(w_\ell)))\). By sampling and registering tokens across distinct granularities (e.g., stages \(Q/6, Q/3\) and \(L/6, L/3\)), it establishes multi-granularity shared spaces \(C_{vt}^s\) tailored for asymmetric cross-modal exchange.

3. Asymmetry-Guided Interactor (AGI): Explicit Mutual Learning Across Asymmetric Modalities Generic multimodal adaptation architectures rely on symmetric or implicit feature pooling, worsening cross-modal misalignment due to the opposing generalization evolutions of vision and language spaces. Within each shared space \(C_{vt}^s\), AGI aggregates intra-modal tokens into representative embeddings \(f_{ssv}\) and \(w_{sst}\), and calculates explicit bidirectional cross-attention matrices: $\(\text{Attn}_{v2t} = w_{sst} \cdot (f_{ssv})^T, \quad \text{Attn}_{t2v} = f_{ssv} \cdot (w_{sst})^T\)$ Softmax-normalized attention maps yield aligned features \(\hat{f}_{ssv}\) and \(\hat{w}_{sst}\), which are then projected via learnable weight matrices and added back to the original modality streams via residual connections. This bidirectional flow allows linguistic concepts to ground and constrain visual anomaly localization while guiding textual embeddings with fine-grained visual artifacts.

Loss & Training

During the training process, all original CLIP vision and text encoder weights remain strictly frozen. Only the parameters of the MFA modules, asynchronous registration projections, and AGI cross-attention blocks are updated. At inference, classification relies on the cosine similarity between the final projected image representation \(f_g'\) and the textual category embeddings \(b_c'\) (\(c \in \{0, 1\}\) representing real and fake): $\(P_c = \frac{\exp(\text{cos}(f_g', b_c') / \tau)}{\sum_{k \in \{0, 1\}} \exp(\text{cos}(f_g', b_k') / \tau)}\)$ where \(\tau\) is a learnable temperature parameter. Optimization is driven end-to-end using standard binary cross-entropy loss.

Key Experimental Results

Main Results

The framework is evaluated under rigorous cross-dataset generalization protocols. All detectors are trained exclusively on FaceForensics++ (FF++-c23) and evaluated on unseen target benchmarks including DeepFakeDetection (DFD), Celeb-DF-v2 (CDF-v2), and DFDC-Large (DFDC-L) using image-level AUC (%).

Dataset Metric Ours (Asy-Det) Prev. SOTA (UDD / AAAI 2025) Gain
DFD AUC (%) 93.80 91.00 +2.80%
Celeb-DF-v2 AUC (%) 92.94 86.90 +6.00%
DFDC-Large AUC (%) 83.50 75.80 +7.70% (+6.20% margin)
Average (Avg) AUC (%) 90.10 84.60 +5.50%

When compared against generic multimodal interaction paradigms (e.g., CoCoOp, MaPLe, MMA, TCP, MMRL), Asy-Det demonstrates decisive superiority: outperforming the strongest baseline MMA (85.40% on CDF-v2 and 83.80% on average) by +7.50% on CDF-v2 and +6.30% overall average AUC.

Ablation Study

Ablation experiments conducted on FF++-c23 and evaluated across DFD, CDF-v2, and DFDC-L highlight the contribution of each module along with trainable parameter counts (T-Param):

Config MFA AGI DFD (AUC/AP/EER) CDF-v2 (AUC/AP/EER) DFDC-L (AUC/AP/EER) Trainable Params Note
M1 (Baseline) \(\times\) \(\times\) 80.36 / 97.38 / 26.17 75.03 / 85.14 / 33.14 72.16 / 72.60 / 33.69 1.38 M Independent dual-modality prompt baseline
M2 \(\checkmark\) \(\times\) 92.69 / 99.12 / 13.77 87.94 / 93.76 / 20.79 80.67 / 84.15 / 26.78 2.76 M MFA decoupling discrimination & generalization
M3 \(\times\) \(\checkmark\) 91.24 / 98.94 / 15.98 87.92 / 93.45 / 20.79 80.45 / 83.38 / 26.57 1.64 M AGI explicit asymmetric cross-attention only
M4 (Full Model) \(\checkmark\) \(\checkmark\) 93.85 / 99.25 / 12.95 92.94 / 96.43 / 15.16 83.60 / 86.50 / 24.79 4.41 M Synergistic full Asy-Det model

Key Findings

  • Adding MFA alone (M2) surges CDF-v2 AUC by 12.91%, illustrating that phase-decoupled adaptation substantially suppresses intra-modal overfitting.
  • Integrating AGI alone (M3) enhances CDF-v2 AUC by 12.89%, validating that explicit cross-modal mutual alignment effectively counteracts unimodal perceptual bias.
  • The combination of MFA and AGI (M4) achieves a dramatic cumulative improvement of 17.91% over baseline on CDF-v2 (75.03% \(\to\) 92.94%), cutting EER by more than half (33.14% \(\to\) 15.16%) with merely 4.41M trainable parameters.
  • In cross-manipulation transfer (Table 2), when trained on FaceShifter and evaluated on DeepFakes, Asy-Det reaches 91.63% AUC (surpassing previous SOTA SSG at 89.99%) and attains an average cross-manipulation AUC of 94.02%.
  • Robustness benchmarks under four corruption types across five severity levels demonstrate that under level-4 Gaussian noise, where standard detectors like NPR plunge below 65% AUC, Asy-Det maintains robust detection performance near 97% AUC.

Highlights & Insights

  • Theoretical Grounding of Visual-Linguistic Asymmetry: Systematically uncovers that depth-dependent feature spaces in VLMs impact discrimination and generalization in opposing directions across modalities, pinpointing why standard symmetric multi-modal fusion pipelines fail in fine-grained forensics.
  • Highly Parameter-Efficient yet Resilient: Achieves state-of-the-art detection accuracy and unprecedented robustness against severe input corruptions while fine-tuning only 4.41M parameters on top of a frozen foundation model.
  • Transferable Asymmetric Paradigm: The concept of asymmetric cross-modal registration and phased discrimination-generalization decoupling offers an actionable template for other fine-grained multimodal tasks burdened by domain shift and subtle class discrepancies.

Limitations & Future Work

  • Static Text Prompts: Currently, textual guidance is rooted in fixed category prompts ("real" / "fake"); leveraging large language models to generate rich, descriptive facial anomaly prompts (e.g., blending boundaries, illumination mismatches) could further boost interpretability.
  • Temporal Dynamics in Video Forgery: The current framework operates on image-level inputs; extending the asynchronous registration and cross-modal attention mechanisms to the spatio-temporal video domain represents a promising research avenue.
  • vs UDD (AAAI 2025): UDD confines token shuffling and debiasing entirely to the visual encoder; Asy-Det transcends unimodal boundaries by enabling explicit intermediate cross-modal mutual learning, leading UDD by 6.00% AUC on Celeb-DF-v2.
  • vs MMA (CVPR 2024): MMA enforces symmetric unimodal projections and top-layer adapters, ignoring depth-wise functional divergence; Asy-Det resolves this via phased asymmetric adaptation and multi-granularity alignment, exceeding MMA by 6.30% in average cross-dataset AUC.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First work to uncover visual-linguistic asymmetry in VLM-based face forensics and propose an asymmetry-guided interaction paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 6 benchmarks spanning cross-dataset, cross-manipulation, corruption robustness, and granular ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear motivational narrative, sound theoretical backing with Rademacher complexity analysis, and rigorous empirical validation.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational perspective for parameter-efficient adaptation of vision-language foundation models in security-sensitive forensic tasks.