Skip to content

DiscoVL: Unveiling Disentangled Cross-Modal Representation Learning via Orthogonal Adversarial Regularization for Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Vision-Language Models, Cross-Modal Disentanglement, Orthogonal Adversarial Regularization, Prompt Learning, Few-Shot Generalization

TL;DR

DiscoVL introduces a disentangled cross-modal representation learning framework that decouples shared transferable semantics from task-specific cues via a multi-branch low-rank residual aligner and employs orthogonal adversarial triplet regularization to prevent centroid collapse, achieving superior generalization across 15 benchmarks.

Background & Motivation

Pre-trained vision-language models (VLMs) such as CLIP capture comprehensive joint visual-textual semantics across hundreds of millions of web image-text pairs, demonstrating extraordinary zero-shot transfer capabilities. To tailor these foundation models to specific downstream tasks without incurring prohibitive re-training costs, parameter-efficient prompt learning methods—such as CoOp, CoCoOp, PromptSRC, and MMRL—have emerged as the prevailing paradigm. By freezing the foundational vision-language backbone and fine-tuning only a lightweight collection of prompt vectors or cross-modal representation tokens, these strategies mitigate the risk of catastrophic forgetting while adapting to domain-specific objectives.

However, existing VLM adaptation techniques inherently suffer from a critical flaw: representation entanglement, wherein class-specific discriminative cues and class-agnostic transferable knowledge are tightly conflated within a unified latent space. This entanglement triggers three interrelated failure modes. First, aggressive adaptation toward target downstream classes contaminates the class-agnostic semantic manifold, causing learned features to drift away from the foundational pre-trained priors and severely deteriorating zero-shot transfer to unseen novel classes. Second, conventional alignment mechanisms rely on rigid joint-space projections that overlook the intrinsic cross-modal distribution gaps between vision and text, amplifying representation discrepancies across deeper network layers. Third, standard contrastive or cross-entropy objectives inevitably encourage representation tokens to collapse prematurely onto class centroids, destroying the fine-grained semantic geometry indispensable for novel-class generalization.

To resolve this fundamental bottleneck, this work proposes an explicit disentanglement framework that factorizes the adapted representation space into a shared, class-agnostic subspace that preserves CLIP's transferable semantics and a task-specific subspace dedicated to downstream discriminative cues. Core idea: DiscoVL explicitly factorizes vision-language representations into shared and task-specific subspaces via a multi-branch low-rank residual aligner with bidirectional cross-modal feedback, while enforcing orthogonal adversarial triplet regularization to eliminate feature redundancy and prevent centroid collapse.

Method

Overall Architecture

DiscoVL is built upon a frozen CLIP dual-tower architecture comprising an image encoder and a text encoder. Starting from a chosen intermediate Transformer layer \(Q\) (set to layer 6 in practice), a set of learnable representation tokens is injected into the forward stream. The adaptation pipeline is driven by two cooperative components: the Disentangled Cross-modal Representation Aligner (DCRA) decomposes representations into dynamic low-rank subspaces and facilitates bidirectional cross-modal exchange across layers; meanwhile, the Orthogonal Adversarial Representation Learning (OARL) module enforces orthogonal penalties and adversarial triplet constraints on the output embeddings to preserve feature diversity and suppress representation degradation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Text Prompts"] --> B["Multi-branch Low-rank Residual Aligner<br/>Gated decomposition into E low-rank subspaces"]
    B --> C["Bidirectional Cross-modal Feedback<br/>Lower-layer text-to-vision / upper-layer vision-to-text"]
    C --> D["Feature Projection & Prediction<br/>Frozen path yields fk / adapted path yields fa"]
    D --> E["Adversarial Triplet Representation Constraint<br/>Pulls intra-class representation / pushes inter-class visual"]
    D --> F["Orthogonal Feature Regularization<br/>Frobenius-norm orthogonality penalty suppressing collapse"]
    E --> G["Total Multi-objective Joint Optimization"]
    F --> G

Key Designs

1. Multi-branch Low-rank Residual Aligner: Dynamically capturing diverse cross-layer subspaces Prior prompt-learning and representation aligners (such as MMRL++) typically decompose projection weights into a single globally shared kernel and a single low-rank residual matrix, lacking the capacity to adaptively select appropriate subspaces conditioned on dynamic input features. DCRA extends this paradigm to a multi-branch low-rank residual structure across \(E\) subspaces. For modality \(m \in \{i, l\}\) at layer \(\ell\), the projection weights are formulated as the summation of a globally shared kernel and dynamically weighted residuals:

\[F_\ell^m = F_{\text{shared}}^m + \sum_{e=1}^{E} o_e^\ell J_e^\ell D_e^\ell\]

where \(J_e^\ell \in \mathbb{R}^{d_a \times a_1}\) and \(D_e^\ell \in \mathbb{R}^{a_1 \times d_m}\) denote branch-specific low-rank matrices with rank \(a_1 = 4\), and the scalar attention weights \(o_e^\ell\) are dynamically predicted by a lightweight gating network followed by Softmax: \(o^\ell = \text{Softmax}(g(F_\ell^m))\). This gating mechanism allows each layer to selectively activate the most task-relevant geometric subspace based on input data characteristics, capturing complex and diverse semantic patterns while maintaining parameter compactness.

2. Bidirectional Cross-modal Feedback: Bridging modality distribution gaps across depths To alleviate representation discrepancies caused by unidirectional information transfer, DCRA establishes an adaptive bidirectional feedback loop across transformer depths. In the lower transformer layers, visual representation tokens absorb high-level textual context through an auxiliary projection \(Z_{T2I}\); in the upper layers, the information flow is reversed, injecting visual summary representations into the text stream via projection \(Z_{I2T}\). This exchange is modulated by a learned fusion gate \(\rho(\ell)\):

\[A_\ell^{i'} = (1 - \rho(\ell)) F_\ell^i(A_\ell^i) + \rho(\ell) Z_{T2I}(F_\ell^l(A_\ell^l))\]
\[A_\ell^{l'} = (1 - \rho(\ell)) F_\ell^l(A_\ell^l) + \rho(\ell) Z_{I2T}(F_\ell^i(A_\ell^{i'}))\]

The fusion gate \(\rho(\ell)\) dynamically arbitrates the balance between preserving modality-specific identity and integrating complementary cues from the other modality at each layer \(\ell\). This progressive bidirectional interaction harmonizes visual and textual feature distributions and mitigates cross-layer representation drift.

3. Adversarial Triplet Representation Constraint: Aligning class semantics while isolating discriminative cues To prevent representation tokens from drifting aimlessly during downstream adaptation, OARL incorporates a category-level adversarial triplet loss. Considering the image class feature \(f_k = Z_{\text{img}}^k(k_L)\) for class \(k\) as the anchor, the aligned representation feature \(f_a = Z_{\text{img}}^a(a_L)\) from the same class acts as the positive sample, while an image feature \(f_{k'}\) from a different class \(k' \neq k\) serves as the negative sample. The triplet constraint is defined with margin \(m\):

\[\mathcal{L}_{\text{atr}}^k = \max\left(0, \|f_k - f_a\|_2^2 - \|f_k - f_{k'}\|_2^2 + m\right)\]

This objective forces the adapted representation \(f_a\) to stay closely aligned with the anchor visual feature of the same category while pushing away representations of dissimilar categories, thereby decoupling class-specific adaptations from general image features and preventing inter-class semantic contamination.

4. Orthogonal Feature Regularization: Suppressing feature redundancy and centroid collapse Although adversarial triplet learning helps separate class semantics, unconstrained optimization frequently leads representation tokens to collapse onto isolated class centroids, destroying the underlying manifold structure necessary for novel-class transfer. To counteract this collapse, OARL imposes an orthogonal regularization penalty via the Frobenius norm, enforcing mutual independence among representation dimensions:

\[\mathcal{L}_{\text{ortho}}^k = \zeta \cdot \left\| I - \frac{X^k (X^k)^T}{\|X^k\|_F^2} \right\|_F^2\]

where \(X^k \in \mathbb{R}^{M \times d_i}\) represents the stacked representation feature matrix, and \(I\) is the identity matrix. By heavily penalizing off-diagonal correlation terms, the orthogonal penalty ensures that each representation dimension encodes distinct, non-redundant semantic information. This structural independence prevents adversarial perturbations from exploiting correlated features and guarantees that the representation space retains a rich geometric structure.

Loss & Training

The entire framework is trained end-to-end by minimizing a multi-task objective combining task classification, foundational knowledge retention, and orthogonal adversarial constraints:

\[\mathcal{L}_{\text{total}} = \alpha \mathcal{L}_{\text{ce}}^k + (1 - \alpha)\mathcal{L}_{\text{ce}}^a + \beta(\mathcal{L}_{\text{cos}}^i + \mathcal{L}_{\text{cos}}^l) + \gamma(\mathcal{L}_{\text{atr}}^k + \mathcal{L}_{\text{atr}}^{k'} + \mathcal{L}_{\text{ortho}}^k + \mathcal{L}_{\text{ortho}}^{k'})\]

where \(\mathcal{L}_{\text{cos}}^i\) and \(\mathcal{L}_{\text{cos}}^l\) penalize deviations from the original frozen CLIP image and text features via cosine distance, ensuring foundational zero-shot capabilities remain uncorrupted. The hyper-parameters are configured as \(\alpha = 0.7, \beta = 0.5, \gamma = 0.3\), token count \(M=5\), injection layer \(Q=6\), and low-rank dimensions \(a_1=4, a_2=64\). At inference time: - Base class prediction: Combines class logits and representation logits via weighted interpolation: \(p(y=k \mid x) = \alpha p(y=k \mid f_k) + (1-\alpha) p(y=k \mid f_a)\); - Novel class prediction: Evaluates solely the class token logits: \(p(y=k \mid x) = p(y=k \mid f_k)\), preventing task-specific biases from compromising novel category generalization.

Key Experimental Results

Main Results

Under the 16-shot base-to-novel generalization benchmark evaluated across 11 diverse visual recognition datasets, DiscoVL demonstrates consistent gains over competitive prompt-learning and representation-learning methods.

Method Base Acc (%) Novel Acc (%) Harmonic Mean HM (%) Note
CoOp (IJCV 2022) 82.69 63.22 71.66 Vanilla prompt tuning; severe novel degradation
CoCoOp (CVPR 2022) 80.47 71.69 75.83 Image-conditional prompts mitigate overfitting
MaPLe (CVPR 2023) 82.28 75.14 78.55 Dual-branch multi-modal prompt tuning
PromptSRC (ICCV 2023) 84.26 76.10 79.97 Self-regularized prompts preserve unsupervised priors
TCP (CVPR 2024) 84.13 75.36 79.51 Textual class-aware prompt tuning
CoPrompt (ICLR 2024) 84.00 77.23 80.48 Consistency-guided dual-head tuning
Skip Tuning (CVPR 2025) 85.04 77.53 81.11 Backbone skip-layer adaptation
MMRL (CVPR 2025) 85.68 77.16 81.20 Multi-modal representation learning baseline
DiscoVL (Ours) 85.98 77.61 81.58 Base +0.30%, Novel +0.45%, HM +0.38%

In the cross-dataset transfer evaluation (trained on ImageNet with 16 shots and tested across 10 out-of-domain datasets without fine-tuning), DiscoVL delivers the highest average performance:

Method Source: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 Aircraft SUN397 DTD EuroSAT UCF101 Average
CoOp 71.51 93.70 89.14 64.51 68.71 85.30 18.47 64.15 41.90 46.39 66.55 63.88
CoCoOp 71.02 94.43 90.14 65.32 71.88 86.06 22.94 67.36 45.73 45.37 68.21 65.74
MaPLe 70.72 93.53 90.49 65.57 72.23 86.20 24.74 67.01 46.49 48.06 68.69 66.30
PromptSRC 71.27 93.60 90.25 65.70 70.25 86.15 23.90 67.10 46.87 45.50 68.75 65.81
MMRL 72.03 94.67 91.43 66.10 72.77 86.40 26.30 67.57 45.90 53.10 68.27 67.25
DiscoVL (Ours) 72.10 94.73 91.90 66.30 72.87 86.70 26.47 67.80 46.13 53.07 68.80 67.48

Ablation Study

1. Component-wise Ablation Analysis (Average over 11 datasets)

Configuration Base Acc (%) Novel Acc (%) HM (%) Gain over Baseline
Baseline (MMRL) 85.68 77.16 81.20 -
+ Multi-branch residual aligner 85.70 77.23 81.24 +0.04% HM
+ Bidirectional feedback (full DCRA) 85.77 77.30 81.32 +0.12% HM
+ Adversarial representation learning 85.84 77.50 81.46 +0.26% HM
+ Orthogonal regularization (full DiscoVL) 85.98 77.61 81.58 +0.38% HM

2. Loss Function Contribution (StanfordCars dataset)

Configuration \(\mathcal{L}_{\text{atr}}^k\) \(\mathcal{L}_{\text{atr}}^{k'}\) \(\mathcal{L}_{\text{ortho}}\) Base (%) Novel (%) HM (%)
Baseline \(\times\) \(\times\) \(\times\) 81.30 75.07 78.06
+ Anchor adversarial triplet \(\checkmark\) \(\times\) \(\times\) 81.57 75.20 78.26
+ Bidirectional adversarial triplet \(\checkmark\) \(\checkmark\) \(\times\) 81.70 75.23 78.33
+ Orthogonal regularization (full loss) \(\checkmark\) \(\checkmark\) \(\checkmark\) 81.80 75.30 78.42

Key Findings

  • Orthogonal regularization unlocks novel-class generalization: Adversarial triplet objectives alone offer modest gains, but adding the Frobenius-norm orthogonal penalty \(\mathcal{L}_{\text{ortho}}\) yields an extra bump in novel accuracy (77.50% \(\rightarrow\) 77.61%), confirming that breaking feature correlations effectively prevents centroid collapse.
  • Over 55% parameter reduction with enhanced throughput: Compared with MMRL (4.99M parameters), DiscoVL utilizes compact multi-branch low-rank factorization to reduce trainable parameters to 2.21M (-55.7%), decreasing latency from 6.31 ms to 6.11 ms and cutting ImageNet training time from 156 min to 147 min while outperforming previous models on HM.
  • Robust performance under extreme few-shot regimes: In 1-shot and 2-shot settings, DiscoVL reaches 73.48% and 76.63% average accuracy, outperforming PromptSRC and Skip Tuning by significant margins and proving that disentangled features preserve crucial classification boundaries even with sparse supervision.

Highlights & Insights

  • Subspace Factorization versus Entangled Projections: Rather than optimizing a single conflated representation space, DiscoVL explicitly isolates class-agnostic shared semantics from task-specific cues, guaranteeing that downstream adaptation does not corrupt pre-trained foundation knowledge.
  • Synergy of Orthogonality and Adversarial Constraints: Standard adversarial triplets suffer from feature redundancy and centroid collapse; coupling an orthogonal penalty forces the adversarial objective to operate across genuinely independent dimensions, creating robust decision boundaries.
  • Broad Transferability to Open-Vocabulary Pipelines: The combination of low-rank residual aligners and orthogonal regularization is model-agnostic, readily extensible to open-vocabulary object detection, segmentation, and multimodal agent action grounding.

Limitations & Future Work

  • Admitted Limitations: The construction of mini-batch positive/negative category triplets requires sufficient category diversity within batches, which can become memory-intensive on datasets with very large label sets.
  • Observed Caveats: Inference relies on a rigid bifurcation strategy (dual-branch for base classes, single class token for novel classes). In practical open-world deployment, identifying whether a test sample belongs to a known or unknown category is non-trivial without an automated novelty gate.
  • Future Directions: Exploring dynamic test-time prompt adaptation mechanisms and extending orthogonal adversarial disentanglement to streaming temporal video understanding.
  • vs MMRL / MMRL++ (CVPR 2025 / IJCV 2026): MMRL relies on a static shared kernel and single residual projection within an entangled space; DiscoVL introduces multi-branch low-rank aligners and orthogonal adversarial regularization, cutting parameter count by 55.7% while delivering superior generalization.
  • vs PromptSRC (ICCV 2023): PromptSRC utilizes self-regulating consistency losses to combat forgetting; DiscoVL directly regulates the algebraic geometry of representation tokens via orthogonality, achieving better transferability with simpler loss formulations.
  • vs AAPL (CVPR 2024): AAPL employs adversarial triplets for attribute-guided prompts but suffers from token collapse due to feature correlation; DiscoVL demonstrates that orthogonal regularization is the missing ingredient for stabilizing adversarial representation learning.

Rating

  • Novelty: ⭐⭐⭐⭐ [Addresses representation entanglement in VLM prompt learning via clean subspace factorization and orthogonal regularization]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 11 base-to-novel datasets, cross-dataset transfer, 4 ImageNet domain shifts, few-shot setups, and detailed ablations]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, well-motivated structural design, and intuitive visualizations]
  • Value: ⭐⭐⭐⭐⭐ [Offers a robust, parameter-efficient adaptation paradigm with strong theoretical and empirical justifications]