MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Heterogeneous Distillation, Multimodal Large Language Model, CLIP, Compositional Reasoning, Centered Kernel Alignment
TL;DR¶
Addressing the substantial computational overhead and representation degradation of cascaded synthetic hard-negative pipelines, MLLMCLIP introduces a heterogeneous feature-level distillation framework that directly transfers multimodal representations from a generative MLLM decoder to a discriminative CLIP student via attention-based token selection and CKA loss, establishing state-of-the-art compositional and retrieval performance at 25x lower inference latency.
Background & Motivation¶
Contrastive Language-Image Pretraining (CLIP) has fundamentally revolutionized open-world visual perception, demonstrating unprecedented transferability across zero-shot classification and cross-modal retrieval benchmarks. Despite these broad empirical successes, extensive recent literature has identified a pervasive vulnerability: CLIP fundamentally behaves like a "bag-of-words" model. Because standard InfoNCE contrastive objectives primarily enforce global cosine similarity across matched image-caption pairs, the learned dual-encoder representations largely neglect fine-grained compositional dependencies, including attribute-object binding, subject-verb-object relationships, and precise spatial arrangements. Consequently, when presented with perturbed captions swapping subject and object (e.g., "The horse is eating the grass" versus "The grass is eating the horse"), standard CLIP frequently fails to distinguish the semantically absurd alternative.
To mitigate this compositional deficit, current state-of-the-art methods have predominantly relied on what can be conceptualized as "data-level distillation" (such as NegCLIP, FSC-CLIP, and TripletCLIP). This paradigm assembles an intricate external generation pipeline: an external large language model (e.g., LLaMA or Qwen) is prompted to generate syntactically plausible hard-negative captions, which are subsequently routed into a text-to-image diffusion model (e.g., SDXL-Turbo) to synthesize corresponding negative images. While these synthetic triplets improve contrastive discrimination, data-level distillation incurs two severe structural bottlenecks. First, it relies on disjoint generative components that lack unified multimodal understanding, frequently injecting visual hallucinations and semantic noise into the training budget. Second, the sequential pipeline is computationally inefficient during preprocessing and introduces substantial memory and compute overhead by forcing the student encoder to process extra synthetic samples during pretraining.
Meanwhile, contemporary generative Multimodal Large Language Models (MLLMs, such as Qwen-VL and InternVL) demonstrate exceptional compositional reasoning due to deep cross-attention layers across interleaved visual and textual sequences. However, directly fine-tuning MLLMs into representation encoders (as explored in VLM2Vec and MLLM-as-Embedding frameworks) preserves the full billion-parameter generative decoder during inference, resulting in orders of magnitude higher compute cost and inference latency that remain impractical for scalable real-time retrieval. Core idea: bypass external synthetic hard-negative pipelines entirely, proposing a heterogeneous feature-level distillation framework (MLLMCLIP) that directly transfers rich multimodal hidden states from a frozen generative MLLM teacher into a lightweight discriminative CLIP student via attention-based salient token selection and structural Centered Kernel Alignment (CKA) loss.
Method¶
Overall Architecture¶
MLLMCLIP establishes an end-to-end heterogeneous feature-level distillation pipeline that circumvents synthetic negative sample generation. The framework pairs a completely frozen generative Multimodal Large Language Model (MLLM teacher) with a lightweight dual-encoder student (comprising a ViT-B/32 image encoder and a 12-layer Transformer text encoder). For each training pair, the image and caption are formatted with a structured compositional extraction template and fed into the MLLM decoder in a single forward pass to produce layer-wise multimodal hidden states. Concurrently, the student encodes the image and text independently through standard patch and token projections. An auxiliary projection head on the student matches the teacher feature dimensionality, while an attention-driven token selection operator isolates the most informative multimodal teacher token per layer. Finally, the student is trained from scratch on CC3M using a joint objective combining global InfoNCE contrastive loss and batch-level Centered Kernel Alignment (CKA) distillation loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Image-Caption Pair (x_I, x_T)"] --> B["Compositional Feature Extraction Prompt<br/>Activate fine-grained MLLM reasoning states"]
B --> C["Attention-based Layer-wise Token Selection<br/>Extract salient multimodal tokens by max received attention"]
C --> D["Cross-Architecture Strided Layer Selection<br/>Uniform sampling across full network depth"]
D --> E["Centered Kernel Alignment Relational Loss<br/>Batch Gram matrix alignment preserving geometry"]
E --> F["Joint InfoNCE Contrastive Optimization<br/>Efficient dual-encoder ViT-B/32 inference"]
Key Designs¶
1. Compositional Feature Extraction Prompt: activating latent discrimination without decoding To ensure that the teacher's hidden states exhibit sharp sensitivity to compositional nuances without generating explicit text, MLLMCLIP formulates a specialized extraction template. Rather than feeding the raw image-caption pair passively, the prompt instructs the MLLM to internally reason whether the given caption accurately characterizes the visual scene, synthesize a hypothetical confusing counterpart with subtle compositional perturbations (focusing specifically on object attributes, actions, interactions, and spatial relations), and mentally contrast the two. This internal reasoning objective primes the self-attention layers of the frozen MLLM decoder, enriching intermediate activations with fine-grained discriminatory signals that distinguish valid bindings from subtle permutations.
2. Attention-based Layer-wise Token Selection: identifying salient multimodal anchors
In generative MLLMs, visual patch tokens and textual tokens are concatenated into a unified sequence under causal attention masking, whereas discriminative CLIP models summarize modalities into single [CLS] and [EOS] vectors. Fixed token positionsโsuch as the boundary tokens [Image End] or [Text End]โfail to reliably capture dynamic multimodal interactions across varying network depths. To bridge this gap adaptively, MLLMCLIP computes the multi-head averaged self-attention matrix \(\bar{A}_l \in \mathbb{R}^{N \times N}\) at each decoder layer \(l\):
$\(\bar{A}_l = \frac{1}{K} \sum_{k=1}^K A_l^{(k)}\)$
It then measures the maximum attention weight that each token \(n\) receives from all query positions in the sequence:
$\(a_{l,n} = \max_{i \in \{1,\dots,N\}} \bar{A}_l[i, n]\)$
The token receiving the highest attention mass is selected as the representative anchor for that layer:
$\(n_l^* = \arg\max_{n \in \{1,\dots,N\}} a_{l,n}, \quad h_l^{\mathrm{Teach}} = h_{l, n_l^*}^{\mathrm{MLLM}}\)$
This mechanism adaptively pinpoints the locus of cross-modal synthesis per layer, outperforming rigid position heuristics while preserving the simplicity and storage efficiency of maintaining a single feature vector per layer.
3. Cross-Architecture Strided Layer Selection: spanning fine-grained details and abstract semantics Because the teacher MLLM (typically 28 to 36 Transformer layers) substantially out-depths the 12-layer CLIP student, establishing an effective layer correspondence is essential. Evaluating contiguous layer partitions reveals that confining distillation to lower (1โ12), middle (13โ24), or upper (25โ36) blocks yields suboptimal representations because lower layers capture fine-grained perceptual textures while deeper layers encode high-level semantic abstractions. MLLMCLIP adopts a uniform strided selection strategy (e.g., layers 1, 4, 7, ...), ensuring that the student is supervised evenly across the entire representational hierarchy of the generative teacher.
4. Centered Kernel Alignment Relational Loss: mitigating cross-paradigm representation collapse Direct element-wise feature matching via Mean Squared Error (MSE) enforces rigid coordinate-level alignment between generative and discriminative manifolds, which results in catastrophic optimization collapse. MLLMCLIP instead deploys a structural Centered Kernel Alignment (CKA) objective. Given a batch of size \(B\), the centered student hidden state matrix \(\tilde{X}_l \in \mathbb{R}^{B \times D}\) and teacher hidden state matrix \(\tilde{Y}_{s_l} \in \mathbb{R}^{B \times D}\) are constructed by removing batch column means: $\(\tilde{X}_l = X_l - \frac{1}{B} \mathbf{1}\mathbf{1}^\top X_l, \quad \tilde{Y}_{s_l} = Y_{s_l} - \frac{1}{B} \mathbf{1}\mathbf{1}^\top Y_{s_l}\)$ The batch Gram matrices \(K_l = \tilde{X}_l \tilde{X}_l^\top\) and \(L_{s_l} = \tilde{Y}_{s_l} \tilde{Y}_{s_l}^\top\) capture pairwise relational affinities among samples. The normalized Frobenius inner product defines the CKA metric: $\(\mathrm{CKA}(X_l, Y_{s_l}) = \frac{\langle K_l, L_{s_l} \rangle_F}{\|K_l\|_F \|L_{s_l}\|_F}\)$ A square-root formulation \(\ell_{\mathrm{CKA}}(X_l, Y_{s_l}) = 1 - \sqrt{\mathrm{CKA}(X_l, Y_{s_l})}\) provides favorable gradients when similarity is high. By supervising inter-sample geometric affinities rather than absolute point coordinates, CKA successfully circumvents architectural incompatibility and guarantees stable heterogeneous knowledge transfer.
Loss & Training¶
The overall pretraining objective balances standard symmetric InfoNCE contrastive loss with intermediate CKA distillation: $\(\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{contrast}} + \lambda \mathcal{L}_{\mathrm{distill}}\)$ where \(\mathcal{L}_{\mathrm{distill}}\) averages the CKA losses across image and text modalities over all selected strided layers. All models are trained from scratch on Conceptual Captions 3M (CC3M) across 8 NVIDIA A100 GPUs using the AdamW optimizer with a global batch size of 4096 in bfloat16 precision. The learning rate warms up linearly over the first epoch from \(1 \times 10^{-6}\) to a base rate of \(5 \times 10^{-4}\), decaying via a cosine schedule to \(1 \times 10^{-5}\) over 30 epochs with a weight decay of 0.5. The distillation weighting hyperparameter is set to \(\lambda = 1.0\) by default to balance compositional reasoning, zero-shot classification, and cross-modal retrieval.
Key Experimental Results¶
Main Results¶
MLLMCLIP is evaluated against standard CLIP and four prominent data-level enhancement baselines (LaCLIP, NegCLIP, FSC-CLIP, and TripletCLIP) across 11 compositionality benchmarks (including ARO, SugarCrepe, Winoground, VALSE, and CREPE), 13 zero-shot classification datasets (such as ImageNet-1K, CIFAR-10/100, EuroSAT, and DTD), and 2 image-text retrieval datasets (MSCOCO and Flickr-30K).
Table 1: Performance comparison across 11 compositionality benchmarks (accuracy %)
| Method | External Source | ARO | CREPE | EQBEN | SugarCrepe | SvoProbes | VALSE | Winoground | 11-Benchmark Avg. |
|---|---|---|---|---|---|---|---|---|---|
| CLIP (Base) | โ | 35.6 | 11.6 | 14.8 | 59.7 | 75.8 | 55.1 | 7.00 | 37.1 |
| LaCLIP | LLAMA-7B | 35.2 | 9.98 | 14.4 | 64.0 | 76.9 | 56.6 | 4.75 | 37.5 |
| NegCLIP | Qwen3-4B | 36.2 | 11.7 | 14.8 | 62.4 | 76.3 | 56.1 | 7.00 | 37.6 |
| FSC-CLIP | Qwen3-4B | 35.1 | 9.86 | 16.3 | 63.8 | 78.8 | 57.4 | 5.50 | 38.0 |
| TripletCLIP | Qwen3-4B, SDXL-Turbo | 34.6 | 10.5 | 16.4 | 65.7 | 78.8 | 59.4 | 4.25 | 38.3 |
| MLLMCLIP (LLaVA-1.6) | LLaVA-1.6-Mistral-7B | 32.1 | 10.7 | 15.5 | 69.3 | 79.4 | 55.9 | 7.25 | 38.2 |
| MLLMCLIP (Qwen3-VL) | Qwen3-VL-2B | 34.6 | 12.0 | 17.0 | 71.3 | 82.3 | 59.2 | 8.00 | 40.1 |
| MLLMCLIP (Qwen3.5) | Qwen3.5-2B | 34.1 | 11.9 | 17.9 | 72.1 | 81.4 | 58.8 | 7.75 | 40.0 |
| MLLMCLIP (InternVL3) | InternVL3-2B | 35.0 | 12.3 | 18.1 | 72.0 | 81.8 | 57.9 | 6.00 | 40.0 |
| MLLMCLIP (InternVL3.5) | InternVL3.5-2B | 37.0 | 12.2 | 18.7 | 71.8 | 81.9 | 58.3 | 7.00 | 40.4 |
Table 2: Ablation analysis across loss functions, token selection, and supervision paradigms (Qwen3.5-2B teacher)
| Category | Configuration / Variant | Compositionality Avg. | Zero-shot Cls. Avg. | Retrieval Avg. (R@1) |
|---|---|---|---|---|
| Distillation Loss | MSE (direct element-wise) | 30.2 | 7.00 | 0.05 |
| Cosine Similarity | 38.6 | 27.5 | 17.2 | |
| KL Divergence | 38.2 | 28.1 | 18.4 | |
| JS Divergence | 37.4 | 25.8 | 16.0 | |
| Centered Kernel Alignment (CKA) | 40.0 | 31.3 | 19.1 | |
| Teacher Token Selection | Fixed [Image End] (image branch) |
37.5 | 25.2 | 16.2 |
Fixed [Text End] (image branch) |
38.0 | 26.6 | 17.8 | |
Fixed [Text End] (dual branch) |
38.2 | 28.1 | 17.8 | |
| Attention-weighted Pooling | 40.2 | 31.1 | 18.9 | |
| Top-\(k\) Pooling (\(k=8\)) | 39.7 | 30.8 | 18.2 | |
| Single Token (Max Attention, Ours) | 40.0 | 31.3 | 19.1 | |
| Controlled Supervision | NegCLIP (Data-level, Qwen3.5-2B) | 37.9 | 25.0 | 12.9 |
| TripletCLIP (Data-level, SDXL-Turbo) | 38.5 | 25.6 | 15.7 | |
| MLLMCLIP (Feature-level, Ours) | 40.0 | 31.3 | 19.1 |
Ablation Study & Efficiency Analysis¶
Table 3: Inference cost and preprocessing throughput comparisons
| Model / Pipeline | #Params (B) | Inference Time (ms) | SugarCrepe Acc. (%) | Batch Preprocessing (s, B=32) |
|---|---|---|---|---|
| VLM2Vec-v1 (LLaVA-1.6-7B) | 2.21 | \(115.92 \pm 0.33\) | 66.7 | โ |
| VLM2Vec-v2 (Qwen2-VL-2B) | 2.22 | \(114.60 \pm 0.23\) | 72.7 | โ |
| Qwen3-VL-Embedding-2B | 2.13 | \(52.36 \pm 0.21\) | 83.0 | โ |
| MLLMCLIP (ViT-B/32 Student) | 0.15 | \(4.57 \pm 0.04\) | 71.3 | โ |
| Data-level Pipeline (LLM + SDXL) | โ | โ | โ | \(0.835 \pm 0.012\) |
| Feature-level Pipeline (MLLM Ext.) | โ | โ | โ | \(0.232 \pm 0.005\) (3.6x faster) |
Key Findings¶
- Feature-level distillation avoids negative transfer: In controlled experiments using the exact same Qwen3.5-2B model, MLLMCLIP outperforms NegCLIP and TripletCLIP across all dimensions (e.g., zero-shot classification: 31.3% vs 25.0% and 25.6%; retrieval: 19.1% vs 12.9% and 15.7%). This confirms that data-level synthetic negatives compromise general visual representation, whereas feature-level distillation enhances both compositional precision and general transferability simultaneously.
- Relational CKA loss prevents dimensional collapse: Enforcing strict element-wise Euclidean alignment (MSE) leads to optimization collapse, yielding a disastrous 7.00% classification accuracy and 0.05% retrieval R@1. In contrast, CKA aligns batch Gram matrix structures, decoupling feature-scale mismatches between generative decoders and discriminative encoders.
- Dramatically superior inference economics: Compared to MLLM-as-Embedding approaches (e.g., VLM2Vec-v2), MLLMCLIP operates at approximately 1/15th the parameter count (0.15B vs 2.22B) and 25x faster inference latency (4.57 ms vs 114.60 ms) while achieving competitive SugarCrepe accuracy (71.3% vs 72.7%).
Highlights & Insights¶
- First heterogeneous generative-to-discriminative vision-language distillation: The work pioneers direct feature distillation from an autoregressive MLLM decoder into a dual-encoder CLIP architecture, proving that generative multimodal reasoning can be directly transferred into discriminative encoders without text generation.
- Max-received-attention token selection: Adaptively selecting a single token per layer based on maximum received attention matches or exceeds complex multi-token pooling schemes, drastically reducing offline feature storage overhead to a single vector per layer.
- 3.6x preprocessing acceleration: Eliminating the diffusion model image synthesis bottleneck speeds up offline data preparation by 3.6x, while removing synthetic negative samples from the training loop allows larger effective student batch sizes under the same GPU memory budget.
Limitations & Future Work¶
- Teacher-imposed performance ceiling: The student's compositional capacity remains inherently bounded by the teacher MLLM; any persistent relational blind spots or object hallucination patterns present in the teacher's latent space can be transferred into the student.
- Single-seed evaluation trade-off: To accommodate comprehensive evaluations across five distinct MLLM teachers and 26 downstream datasets, experiments are conducted with a single random seed, although gains remain uniform across all teacher architectures.
- Future directions: Exploring dynamic online feature distillation to eliminate offline disk storage requirements, and extending heterogeneous distillation to video-language temporal reasoning and long-document retrieval.
Related Work & Insights¶
- vs NegCLIP / TripletCLIP (Data-Level Distillation): Prior methods synthesize hard-negative texts and images using cascaded LLMs and diffusion models, which introduces pipeline inefficiency and generative artifacts. MLLMCLIP bypasses data synthesis entirely, aligning latent states via CKA to achieve superior compositional accuracy with 3.6x faster data processing.
- vs VLM2Vec / Qwen3-VL-Embedding (MLLM-as-Embedding): Prior works retain the entire multi-billion parameter MLLM backbone for dense representation tasks, incurring high latency (>50โ115 ms). MLLMCLIP distills multimodal knowledge into a lightweight 0.15B dual-encoder achieving 4.57 ms latency.
- vs CLIP-KD / LLaVA-KD (Homogeneous Distillation): Prior distillation frameworks operate strictly within identical architectural paradigms (encoder-to-encoder or decoder-to-decoder). MLLMCLIP bridges the architectural mismatch between generative decoders and discriminative dual encoders.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering heterogeneous feature-level distillation from generative MLLM decoders to discriminative CLIP dual-encoders.
- Experimental Thoroughness: โญโญโญโญโญ Rigorously evaluated across 5 MLLM teachers, 11 compositionality benchmarks, 13 classification datasets, and 2 retrieval tasks with extensive ablations.
- Writing Quality: โญโญโญโญโญ Clear theoretical justification, cohesive narrative prose, and thorough architectural analysis.
- Value: โญโญโญโญโญ Provides an efficient, practical, and highly scalable blueprint for training compositionally robust dual-encoders for high-throughput industry applications.