MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

Conference: ECCV 2026
arXiv: 2606.22220
Paper: ECCV Official
Area: Multimodal VLM
Keywords: multimodal contrastive learning, memorization, cross-modal consistency, generalization, memorization mitigation

TL;DR

Introduces MultiMem, the first principled leave-one-out metric designed to quantify global memorization across arbitrary modality combinations in contrastive learning, demonstrates that cross-modal semantic misalignment drives harmful memorization, and establishes in-training and post-training mitigation strategies that reduce memorization while boosting downstream retrieval and zero-shot accuracy.

Background & Motivation

Multi-modal contrastive learning frameworks (e.g., CLIP, VideoCLIP, AudioCLIP, and ImageBind) project heterogeneous inputs like text, images, video, and audio into a shared latent space, driving foundational breakthroughs across zero-shot classification, cross-modal retrieval, and question answering. In supervised learning (SL) and self-supervised learning (SSL), prior research has established that model memorization of rare in-distribution training samples is essential for generalization; however, memorizing noisy, mislabeled, or outlier samples severely degrades out-of-distribution robustness. Despite the pervasive deployment of multi-modal encoders, what models actually memorize when reconciling multiple modalities and how this memorization interacts with multi-modal generalization has remained entirely unexplored.

Existing memorization metrics cannot be directly generalized to multi-modal contrastive setups involving three or more modalities. Standard supervised metrics depend on label-prediction flips via the Feldman leave-one-out framework, while self-supervised vision metrics rely on Euclidean distances between augmented single-image views. Bi-modal metrics like CLIPMem and déjà vu are strictly restricted to pairwise image-text alignment. When evaluated on multi-modal architectures (e.g., AudioCLIP or quad-modal AVIT-CLIP), these isolated pairwise metrics fail to capture the global joint dynamics across all modalities, frequently showing contradictory or uncorrelated relationships with downstream generalization. Furthermore, multimodal datasets frequently suffer from severe cross-modal semantic inconsistencies where modalities provide completely unrelated or conflicting information, causing encoders to unnaturally collapse representations to satisfy the contrastive objective.

The paper attacks this challenge from the foundational principle that multi-modal contrastive models optimize for cross-modal consistency across all constituent modalities relative to unrelated reference points. Core idea: construct MultiMem, a leave-one-out global cross-modal consistency metric capable of evaluating any number of modalities, and leverage it to guide in-training dynamic batch augmentation and post-training subset pruning to mitigate harmful memorization.

Method

Overall Architecture

MultiMem establishes a unified paradigm spanning principled measurement and targeted mitigation. During measurement, given an \(n\)-modal contrastive model, MultiMem adopts a leave-one-out formulation comparing a model \(f\) trained on the entire training set \(S\) against a model \(g\) trained on \(S \setminus \{i\}\) where sample \(x_i\) is omitted. The quality of sample representations is captured by a Cross-Modal Consistency (CMC) proxy, which computes the inner products across all pairwise normalized modality representations of \(x_i\) while subtracting cross-modal correlations against a held-out set of unrelated validation samples \(H\). During mitigation, MultiMem guides either dynamic in-training perturbation of the most memorized mini-batches or post-training pruning followed by compact refinement.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input Sample<br/>Audio / Video / Image / Text"] --> B["Cross-Modal Consistency Proxy CMC<br/>Sum of all pairwise normalized inner products"]
    B --> C["MultiMem Metric Computation<br/>Difference in CMC between full model f and leave-one-out model g"]
    C -->|Identify top-memorized samples| D["Memorization Mitigation Strategies"]
    D --> E["In-Training Mitigation<br/>Regroup top 5% memorized samples with Gaussian noise"]
    D --> F["Post-Training Mitigation<br/>Remove top memorized outliers & fine-tune on clean subset"]
    E --> G["Restored Modality Gap & Enhanced Downstream Generalization"]
    F --> G

Key Designs

1. MultiMem Metric: Leave-One-Out Evaluation Across All Modalities

Prior memorization formulations focus exclusively on single modalities or isolated pairs, ignoring high-order interactions in tri-modal (e.g., AudioCLIP) and quad-modal (e.g., AVIT-CLIP) settings. MultiMem formalizes memorization across \(n\) modalities. For an input sample \(x_i\), let \(\Phi_{x_i} = [\hat{\phi}_1, \hat{\phi}_2, \dots, \hat{\phi}_n]^\top \in \mathbb{R}^{n \times d}\) denote its stacked, \(\ell_2\)-normalized modality representation matrix. To quantify how tightly \(x_i\)'s modalities are aligned compared to unrelated samples, the Cross-Modal Consistency \(\text{CMC}(i, H)\) with respect to a held-out set \(H\) is formulated as:

\[\text{CMC}(i, H) = \underset{(\cdot)' \sim \text{Aug}}{\mathbb{E}} \mathbf{1}_{n}^\top (\Phi_{x_i'} \Phi_{x_i'}^\top) \mathbf{1}_n - \frac{1}{2|H|} \sum_{h \in H} \underset{(\cdot)' \sim \text{Aug}}{\mathbb{E}} \mathbf{1}_{n}^\top (\Phi_{x_i'} \Phi_{h'}^\top) \mathbf{1}_n\]

where \(\mathbf{1}_n\) denotes the all-ones vector, the first term computes the sum of all intra-sample cross-modal inner products under random data augmentations, and the second term discounts false cross-modal correlations against unrelated held-out samples. The global MultiMem score is then defined as the CMC shift between the full model \(f\) and the leave-one-out model \(g\):

\[\text{MultiMem}(i, H, f) = \underset{f \sim S}{\text{CMC}}(i, H) - \underset{g \sim S \setminus \{i\}}{\text{CMC}}(i, H)\]

This metric captures joint semantic alignment across the entire modality distribution rather than decoupled pairwise projections.

2. In-Training Mitigation: Dynamic Regrouping and Targeted Perturbation

Applying uniform noise or heavy data augmentations across the entire dataset impairs global representation learning and incurs prohibitive training overhead. MultiMem resolves this through targeted in-training intervention. At periodic training intervals (e.g., every 10 epochs), MultiMem evaluates all training samples to identify the top 5% most memorized instances (an optimal ratio validated empirically). These high-memorization samples are dynamically regrouped into dedicated mini-batches, where zero-mean Gaussian noise (\(\sigma = 0.1\)) is injected specifically into their latent representations during subsequent forward passes. The remaining 95% regular samples continue standard training undisturbed. This targeted disruption prevents the network from over-optimizing on cross-modal semantic mismatches while preserving generalizable multi-modal concepts.

3. Post-Training Mitigation: Outlier Pruning and Subset Refinement

For pre-trained models where retraining from scratch is computationally infeasible, MultiMem offers a lightweight post-training mitigation pipeline. MultiMem scores the entire pre-training dataset to rank samples by memorization severity, pruning the top \(K\) most memorized samples (typically \(K \in [50, 500]\)). Empirical inspection reveals that these top memorized instances are almost universally characterized by complete cross-modal semantic discordance (e.g., urban traffic audio paired with cooking video and an unrelated caption). After purging these destabilizing outliers, the model undergoes brief fine-tuning (e.g., 25 epochs) on the remaining dataset. This process restores the compressed modality gap and eliminates distorted representation boundaries without requiring architectural changes.

Loss & Training

Multi-modal contrastive backbones are trained with a symmetric pairwise InfoNCE objective summing contrastive losses across all distinct modality combinations: $\(\mathcal{L} = \sum_{1 \le j < k \le n} w_{jk} \left( \mathcal{L}_{\text{InfoNCE}}(\hat{\phi}_j, \hat{\phi}_k) + \mathcal{L}_{\text{InfoNCE}}(\hat{\phi}_k, \hat{\phi}_j) \right)\)$ In standard training, weights \(w_{jk}\) are uniform (\(w_{jk}=1\)). In balanced training experiments, MultiMem guides an inverse weighting scheme that assigns lower weights to modality pairs contributing excessively to memorization, thereby establishing a harmonized memorization profile across all modalities.

Key Experimental Results

Main Results

The mitigation capabilities guided by MultiMem were evaluated across AudioCLIP, VideoCLIP, AVT-CLIP, and AVIT-CLIP on cross-modal retrieval (TOP@5, T@5), linear probing accuracy, and zero-shot classification. The table below presents the comparative results on AudioCLIP (trained on UrbanSound8K):

Mitigation Stage	Strategy	Retrieval T@5 (%)	Linear Probing Acc (%)	Zero-Shot Acc (%)	MultiMem Score ↓
Baseline	None	36.9%	76.7%	25.4%	0.332
Post-training	Random Removal	37.1%	76.8%	25.5%	0.330
Post-training	Gradient-based	38.1%	77.8%	29.4%	0.307
Post-training	Loss-based	38.4%	77.9%	29.6%	0.308
Post-training	MultiMem (most)	39.3%	78.6%	30.5%	0.282
In-training	Random Noise	37.2%	76.9%	25.5%	0.329
In-training	Gradient-based	40.4%	78.8%	31.1%	0.281
In-training	Modality Augmentation	42.5%	80.1%	34.0%	0.269
In-training	MultiMem (most)	43.3%	80.8%	35.1%	0.262

(Data source: Table 3 in the original paper, "Regulation verification by comparing MultiMem with other mitigation strategy")

Ablation Study

To investigate how perturbing individual or multiple modalities affects pairwise and global memorization along with cross-modal retrieval, an extensive ablation was conducted on AudioCLIP (Audio A, Image I, Text T) using noise injection (\(\sigma = 0.1\)):

Augmented Modality	IT MultiMem	AT MultiMem	AI MultiMem	AIT MultiMem ↓	T@5 I-T (%)	T@5 A-T (%)	T@5 A-I (%)	T@5 AI-T (%) ↑
None	0.222	0.227	0.214	0.332	33.1	30.8	26.5	36.9
Audio only	0.235	0.201	0.204	0.320	32.2	31.3	27.1	38.2
Image only	0.201	0.235	0.188	0.312	33.8	30.0	28.3	39.0
Text only	0.181	0.191	0.216	0.294	34.4	32.1	26.3	39.8
Audio + Image	0.200	0.199	0.199	0.299	33.9	31.8	27.5	39.4
Audio + Text	0.187	0.187	0.192	0.283	34.9	32.6	28.0	40.7
Image + Text	0.180	0.189	0.174	0.271	35.4	32.3	29.1	42.1
Audio + Image + Text	0.177	0.184	0.169	0.264	35.7	33.0	29.5	43.1

(Data source: Table 2 in the original paper, "Adding augmentations to different modalities in AudioCLIP")

Key Findings

• Partial-modal metrics decouple from generalizability: Bimodal metrics (such as CLIPMem or pairwise MultiMem) display erratic, non-monotonic correlations with model generalization; only the joint MultiMem metric consistently exhibits a strong negative correlation with downstream retrieval and classification accuracy.
• Cross-modal semantic misalignment is the primary memorization driver: While bi-modal models primarily memorize mis-captioned pairs, multimodal architectures exhibit extreme memorization on samples where all modalities are mutually disjoint and unaligned semantically.
• Emergence of pattern-driven memorization behavior: Layer-wise UnitMem analysis reveals that incorporating additional modalities (such as audio) dilutes text-label dominance, causing the vision encoder to transition from supervised label memorization toward SSL-like pattern memorization.

Highlights & Insights

• First unified metric for arbitrary modal cardinality: Formulates a closed-form Cross-Modal Consistency proxy combined with a leave-one-out framework, generalizing memorization quantification from simple image-text pairs to complex \(N\)-modal learning systems.
• Restoration of the geometric modality gap: Demonstrates that severe memorization unnaturally collapses cross-modal representation distance on unaligned samples; MultiMem mitigation systematically restores healthy modality gaps and enhances feature transferability.
• Efficient 5% targeted intervention: Intervening on merely the top 5% most memorized instances in-training boosts multi-to-uni retrieval by +6.4% (36.9% to 43.3%) and zero-shot accuracy by +9.7% (25.4% to 35.1%) with negligible training overhead.

Limitations & Future Work

• Computational overhead of leave-one-out training: Computing exact MultiMem scores necessitates training two parallel model instances (\(f\) and \(g\)), which becomes computationally prohibitive on massive web-scale multimodal foundation datasets. Developing gradient-based or influence function approximations is an essential next step.
• Uniform weighting across heterogeneous modalities: The current CMC proxy applies equal weighting across all modal pairs, overlooking disparities in information entropy between modalities (e.g., dense video frames vs. sparse text tags).

Related Work & Insights

• vs CLIPMem & déjà vu: Prior works only evaluate pairwise image-text alignment; MultiMem captures joint global dependencies across arbitrary numbers of modalities with reference-set normalization.
• vs Supervised Learning Memorization: Classical supervised metrics measure discrete class label flipping, whereas MultiMem captures continuous multi-modal subspace alignment in contrastive representation learning.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Establishes the first formal theoretical and empirical foundation for measuring and mitigating memorization in multi-modal contrastive learning beyond two modalities.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across bi-modal, tri-modal, and quad-modal architectures with orthogonal ablations, noise controls, and layer-wise UnitMem analysis.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulations, clear pedagogical progression, and tightly supported empirical claims.
• Value: ⭐⭐⭐⭐⭐ Provides invaluable guidance and practical diagnostic tooling for multimodal dataset filtering, pre-training stability, and privacy leakage prevention.