Skip to content

When Modality Gap Reduction Fails: Prediction-Level Misalignment in Vision-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Modality Gap, Zero-Shot Classification, Prediction-Level Hubness, CLIP, Decision Margins

TL;DR

Reveals why reducing the modality gap in CLIP fails to consistently improve zero-shot accuracy, demonstrating that gap correction induces class-wise score biases that collapse predictions toward a few classes (prediction-level hubness), backed by margin analysis and causal intervention.

Background & Motivation

Contrastive Language-Image Pre-training (CLIP) maps visual and textual representations into a shared high-dimensional embedding space, serving as a cornerstone for modern multimodal foundation models and zero-shot transfer. However, CLIP representations are known to suffer from a systemic modality gap, where image and text embeddings segregate into distinct clusters on the hypersphere. This geometric separation has spurred a plethora of gap-reduction techniques, including post-hoc geometric translation via an estimated gap vector (Linear correction), post-pre-training alignment (e.g., CLIPRefine), and pre-training objectives with explicit gap constraints (e.g., AlignCLIP). These approaches share a common intuition: monotonically shrinking the average Euclidean displacement between image and text embeddings should improve cross-modal alignment and downstream accuracy.

Despite this intuitive foundation, reducing the average modality gap does not consistently lead to downstream zero-shot classification gains, and aggressive linear correction can even precipitate catastrophic accuracy degradation. This empirical mismatch exposes a fundamental contradiction: the modality gap is defined as the squared Euclidean distance between global mean image and text embeddings, which is a coarse, dataset-wide average summary statistic. In stark contrast, downstream zero-shot image classification depends on local inner-product rankings and fine-grained decision margins between a test image and all competing class-text prototypes. When a uniform translation vector is applied across a non-uniform set of prototypes, it warps the relative decision margins across classes, undermining the discriminative balance preserved by the original pre-trained model.

This paper demystifies the gap-performance divergence by shifting focus from embedding-space nearest neighbors to the output distribution of predicted labels. The analysis reveals that excessive gap correction causes predictions on previously correct samples to be systematically redirected and concentrated into a tiny subset of candidate classes. Core idea: modality gap reduction induces an asymmetric class-wise score bias aligned with text prototypes, triggering an output-space collapse toward a few dominant classes—termed "prediction-level hubness"—which degrades downstream zero-shot decision margins and accuracy.

Method

Overall Architecture

The proposed framework formulates a comprehensive analytical and diagnostic pipeline addressing the failure modes of modality gap reduction in zero-shot image classification. In standard zero-shot transfer, an input image is mapped by the frozen visual encoder into an \(L_2\)-normalized embedding, while candidate class names are converted into normalized text prototypes via dataset-specific prompt templates. Predictions are obtained by selecting the class prototype that achieves the highest cosine similarity with the query image.

Under linear gap correction, image embeddings are shifted against the estimated gap vector, while text prototypes are shifted along it before renormalization. The framework decomposes the resulting inner-product score into sample-dependent terms, global shift constants, and an explicit first-order class-wise bias term. It then tracks prediction concentration across classes via the Predicted-Class Gini coefficient and normalized prediction entropy, maps transition-level failure trajectories, and introduces a score-space bias subtraction intervention to verify the causal impact of the gap-induced bias.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Candidate Text Prototypes"] --> B["Modality Gap Shift & Score Bias Induction<br/>Decompose inner products to isolate class-wise bias"]
    B --> C["Margin Distortion & Hubness Diagnosis<br/>Quantify Predicted-Class Gini and transition flows"]
    C --> D["Score-Space Bias Subtraction Intervention<br/>Subtract class-wise bias to restore decision margins"]
    D --> E["Zero-Shot Output Class Prediction"]

Key Designs

1. Modality Gap Shift & Score Bias Induction: Global gap reduction introduces asymmetric class-wise score drift

To uncover how global mean displacement reduction alters local decision geometry, the paper uses analytically tractable post-hoc Linear correction. Given image embeddings \(\{x_i\}_{i=1}^N \subset \mathbb{R}^d\) and text embeddings \(\{t_j\}_{j=1}^N \subset \mathbb{R}^d\), the modality gap vector is estimated as the difference between the empirical modality centroids: $\(g = \frac{1}{N}\sum_{i=1}^N x_i - \frac{1}{N}\sum_{j=1}^N t_j\)$ The average modality gap is defined by its squared Euclidean norm: \(\text{MG} = \|g\|_2^2\). Linear correction shifts embeddings in opposite directions using a strength parameter \(\alpha\): \(x_i^{(\alpha)} = x_i - \alpha g\) and \(t_c^{(\alpha)} = t_c + \alpha g\). Expanding the unnormalized inner-product score yields: $\(s_{i,c}^{(\alpha)} = \langle x_i - \alpha g, t_c + \alpha g \rangle = \langle x_i, t_c \rangle + \alpha \langle x_i, g \rangle - \alpha \langle g, t_c \rangle - \alpha^2 \|g\|_2^2\)$ For a fixed image \(i\), the baseline similarity \(\langle x_i, t_c \rangle\) provides the original evidence, while \(\alpha \langle x_i, g \rangle\) and \(-\alpha^2 \|g\|_2^2\) are class-independent offsets that do not alter the argmax ranking across classes. Crucially, the only class-dependent first-order term is: $\(b_c = -\langle g, t_c \rangle\)$ This reveals that whenever a prototype \(t_c\) forms an obtuse angle with the gap vector \(g\) (i.e., \(\langle g, t_c \rangle < 0\)), it automatically receives a positive score boost \(b_c > 0\). Because prototypes are non-uniformly distributed over the unit hypersphere, this uniform shift imparts heterogeneous score gains across classes, systematically warping the decision margins between the true class \(y_i\) and rival classes \(c\).

2. Margin Distortion & Hubness Diagnosis: Decision margin collapse causes output-space label concentration

The paper connects this class-wise bias to downstream accuracy degradation via decision margin analysis. Let the decision margin between the ground-truth class \(y_i\) and a competing class \(c\) after step \(k\) be \(\Delta_{i,c}^{(k)} = s_{i,y_i}^{(k)} - s_{i,c}^{(k)}\). Defining the score shift as \(\delta_{i,c}^{(k)} = s_{i,c}^{(k)} - s_{i,c}^{(0)}\), the updated margin satisfies: $\(\Delta_{i,c}^{(k)} = \Delta_{i,c}^{(0)} + \delta_{i,y_i}^{(k)} - \delta_{i,c}^{(k)}\)$ A previously correct sample is erroneously redirected to class \(c\) whenever \(\delta_{i,c}^{(k)} - \delta_{i,y_i}^{(k)} > \Delta_{i,c}^{(0)}\). When competing prototypes enjoy substantial bias gains \(b_c\), this condition is readily fulfilled across many samples, creating prediction hubs.

Distinct from classical high-dimensional nearest-neighbor hubness in representation space, the authors introduce prediction-level hubness as an output-space phenomenon. It is diagnosed by evaluating the predicted-class count vector \(m^{(k)} = (m_1^{(k)}, \dots, m_C^{(k)})\) where \(m_c^{(k)} = \sum_{i=1}^N \mathbb{I}[\hat{y}_i^{(k)} = c]\). Concentration is quantified using the Predicted-Class Gini coefficient \(G^{(k)}\) and normalized prediction entropy \(H_{\text{norm}}^{(k)}\). Furthermore, a transition-level destination analysis separates prediction changes into wrong-to-correct (W\(\to\)C) and correct-to-wrong (C\(\to\)W) paths. Under over-correction, C\(\to\)W transitions surge, with the majority absorbed by the top-1 and top-5 hub classes.

3. Score-Space Bias Subtraction Intervention: Causal verification of the hubness mechanism

To establish that prediction-level hubness is causally driven by the gap-induced bias \(b_c\) rather than an artifact of embedding normalization, the authors implement a score-space intervention: $\(s'_{i,c} = s_{i,c}^{(\alpha)} - \lambda \alpha b_c\)$ Here \(\lambda\) modulates intervention strength. Under the over-correction regime (\(\alpha = 0.5\)), setting \(\lambda = 1\) approximately cancels the first-order class-dependent distortion \(\alpha b_c\).

Predictions are obtained via \(\hat{y}'_i = \arg\max_c s'_{i,c}\). The experimental results demonstrate that subtracting \(\lambda \alpha b_c\) consistently depresses Predicted-Class Gini, redistributes predictions across non-hub classes, and recovers the accuracy lost to over-correction. This provides interventional proof that gap-induced class-wise bias is the active mechanism driving prediction-level hubness.

Loss & Training

The study examines inference-time adjustments and learned alignment checkpoints: 1. Linear Correction: Evaluated across correction strengths \(\alpha \in [-0.5, 0.5]\) with step size \(0.05\) on frozen CLIP ViT-B/32 encoders. Embeddings are shifted and \(L_2\)-renormalized prior to cosine similarity evaluation. 2. CLIPRefine: An additional-training profile using post-pre-training contrastive alignment on ViT-B/32, evaluated across sequential checkpoint releases. 3. AlignCLIP: A pre-training-based alignment approach built upon a ViT-B/16 backbone, evaluated via public checkpoints. 4. Scoring Variants & Ablations: To rule out metric-specific artifacts, models are evaluated under Cross-domain Similarity Local Scaling (CSLS with \(k=10\)): $\(\text{CSLS}(x_i, t_c) = 2 \cos(x_i, t_c) - \frac{1}{k}\sum_{t \in \mathcal{N}_T(x_i)} \cos(x_i, t) - \frac{1}{k}\sum_{x \in \mathcal{N}_X(t_c)} \cos(x, t_c)\)$ and modality-wise centering, where embeddings are centered post-shift and re-normalized.

Key Experimental Results

Main Results

The framework is evaluated across 10 benchmark image classification datasets: Caltech101, CIFAR-10, CIFAR-100, DTD, EuroSAT, FGVC-Aircraft, Flowers102, Food-101, ImageNet-1K, and Oxford-IIIT Pet.

Table 1: Spearman rank correlations between class-wise bias \(b_c = -\langle g, t_c \rangle\) and prediction metrics under Linear over-correction (\(\alpha = 0.5\))
(\(\rho_m\): correlation with post-correction count \(m_c^{(\alpha)}\); \(\rho_{\Delta m}\): correlation with count increase \(\Delta m_c^{(\alpha)}\); \(\rho_e\): correlation with correct-to-wrong transitions \(e_c^{(\alpha)}\))

Dataset \(\rho_m\) \(\rho_{\Delta m}\) \(\rho_e\)
Caltech101 0.821 0.824 0.792
CIFAR-10 0.891 0.903 0.819
CIFAR-100 0.869 0.863 0.833
DTD 0.586 0.918 0.770
EuroSAT 0.612 0.976 0.796
FGVC-Aircraft 0.570 0.531 0.585
Flowers102 0.747 0.616 0.802
Food-101 0.747 0.817 0.842
ImageNet-1K 0.835 0.812 0.748
Oxford-IIIT Pet 0.749 0.783 0.793
Mean 0.743 0.804 0.778

Table 2: Within-dataset Spearman rank correlations between zero-shot accuracy and Predicted-Class Gini

Method Caltech101 CIFAR-10 CIFAR-100 DTD EuroSAT FGVC-Aircraft Flowers102 Food-101 ImageNet-1K Oxford-IIIT Pet Avg. Exp. Direction
Linear -0.819 -0.998 -0.997 -0.996 -0.725 -0.766 -0.934 -0.995 -1.000 -0.993 -0.922 10/10
CLIPRefine 0.799* -0.923 -0.913 -0.268 -0.630 -0.431 -0.122 -0.900 -0.711 -0.750 -0.485 9/10

*Note: The positive correlation on Caltech101 under CLIPRefine is driven by high label imbalance (ground-truth label Gini = 0.402); normalizing predicted counts by class prior frequencies reverses the correlation to -0.979.

Ablation Study

The evaluation investigates destination concentration across best and worst accuracy settings, alongside mitigations including CSLS scoring and modality-wise centering.

Table 3: Destination concentration of prediction transitions (Across all 10 datasets)

Correction Profile Transition Type Transition Count Top-1 Destination Share Top-5 Destination Share
Linear-best Correct \(\to\) Wrong (C\(\to\)W) 1,556 0.116 0.391
Linear-best Wrong \(\to\) Correct (W\(\to\)C) 3,487 0.187 0.396
Linear-worst (\(\alpha=0.5\)) Correct \(\to\) Wrong (C\(\to\)W) 33,804 0.294 0.672
Linear-worst (\(\alpha=0.5\)) Wrong \(\to\) Correct (W\(\to\)C) 5,238 0.272 0.585
CLIPRefine-best Correct \(\to\) Wrong (C\(\to\)W) 3,547 0.156 0.460
CLIPRefine-best Wrong \(\to\) Correct (W\(\to\)C) 6,430 0.201 0.475
CLIPRefine-worst Correct \(\to\) Wrong (C\(\to\)W) 1,630 0.145 0.388
CLIPRefine-worst Wrong \(\to\) Correct (W\(\to\)C) 1,294 0.121 0.360

Table 4: Mitigation analysis using CSLS scoring and modality-wise centering

Configuration Cosine Acc. Delta (\(\alpha=0.5\)) CSLS Acc. Delta (\(\alpha=0.5\)) Sweep Acc. Range Sweep Gini Range
Linear Baseline -15.1 pt -7.4 pt 18.5 pt 0.30
Linear + Centering (+Center) - - 0.2 pt 0.004

Key Findings

  • Severe output collapse under excessive linear correction: In the Linear-worst setting (\(\alpha = 0.5\)), harmful C\(\to\)W transitions explode to 33,804, with the top-5 destination classes absorbing 67.2% of all redirected samples on average.
  • High predictive power of the bias term: The gap-induced bias \(b_c = -\langle g, t_c \rangle\) strongly correlates with the absorption of corrupted predictions (\(\rho_e = 0.778\) mean across 10 datasets), showing that prediction hubs emerge deterministically from the gap alignment geometry.
  • Robustness across scoring and model scale: CSLS scoring roughly halves the over-correction penalty from -15.1 pt to -7.4 pt but preserves the negative concentration-accuracy correlation across all datasets. Sweeping larger backbones (ViT-B/16 and ViT-L/14) shows identical degradation patterns (mean \(\rho_G = -0.920\) on ViT-L/14).

Highlights & Insights

  • Conceptual clarity: Highlights the disconnect between global average alignment (modality centroid distance) and task-level decision geometry (local relative margins), challenging the assumption that gap elimination guarantees better zero-shot classification.
  • Prediction-level hubness definition: Extends the study of hubness from high-dimensional vector search to discrete label predictions, introducing Predicted-Class Gini and transition concentration as practical diagnostics.
  • Rigorous causal closure: Couples algebraic derivation of class-wise bias with score-space subtraction (\(s'_{i,c} = s_{i,c}^{(\alpha)} - \lambda \alpha b_c\)), successfully recovering lost accuracy and confirming the underlying mechanism.

Limitations & Future Work

  • Closed-form mechanics limited to linear shifts: While the bias formula \(b_c\) rigorously characterizes Linear correction, learning-based methods (CLIPRefine, AlignCLIP) involve non-linear manifold adaptations where prediction concentration remains an empirical diagnostic rather than a proven closed-form cause.
  • Evaluation restricted to fixed label sets: Analysis focuses on zero-shot image classification where an argmax is taken over a fixed candidate vocabulary; extending this to open-ended retrieval, captioning, or VQA remains an open challenge.
  • Future directions: Incorporating hubness-penalizing objectives (such as prediction entropy constraints or uniform manifold regularization) directly into multimodal pre-training and alignment losses.
  • vs. Liang et al. (2022) [Mind the Gap]: Liang et al. characterized the modality gap and introduced post-hoc translation, but did not detect the output-space collapse under over-correction; this work provides the missing margin analysis.
  • vs. Yamaguchi et al. (2025) [CLIPRefine]: CLIPRefine demonstrates that learned post-pre-training reduces the gap without inducing severe hubness, showing that data-driven adaptation can preserve decision boundaries.
  • vs. Radovanović et al. (2010) / Dinu et al. (2015): Classical hubness analyzes points appearing excessively in \(k\)-NN candidate lists; this paper elevates the concept to output-space classifier decisions and label margins.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates a pioneering output-space perspective linking modality gap reduction to decision margin distortion.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 10 datasets, 3 correction regimes, multiple model scales, CSLS mitigations, and causal interventions.
  • Writing Quality: ⭐⭐⭐⭐⭐ Cohesive theoretical narrative, clear mathematical derivations, and elegant experimental validation.
  • Value: ⭐⭐⭐⭐⭐ Offers crucial architectural and evaluation guidelines for multimodal representation learning and model alignment.