Skip to content

MCA: Modality Composition Awareness for Robust Composed Multimodal Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/sony/mca
Area: Multimodal VLM
Keywords: composed multimodal retrieval, modality shortcut learning, multimodal large language models, preference optimization, compositional regularization

TL;DR

To tackle modality shortcut learning where unified MLLM encoders over-rely on a dominant modality, MCA introduces Modality Composition Preference (MCP) to enforce composed query discrimination over unimodal counterparts and Modality Composition Regularization (MCR) to anchor joint embeddings to a lightweight unimodal prototype, yielding +3.2 and +3.1 point gains on out-of-domain retrieval and zero-shot grounding.

Background & Motivation

Multimodal retrieval forms the backbone of open-world AI search, visual commerce, and multimodal retrieval-augmented generation (RAG). Conventional systems have predominantly embraced the dual-encoder paradigm (such as CLIP and CLAP), where text and image modalities are processed through independent backbone encoders and projected into a shared semantic embedding space using symmetric contrastive learning. However, in composed retrieval settingsโ€”where a user provides an image alongside a natural language modification instruction to search for target mediaโ€”dual encoders struggle due to a fundamental lack of deep early cross-modal reasoning, requiring ad-hoc external fusion layers or synthetic data pipelines.

Recent advancements in multimodal large language models (MLLMs) offer a natural alternative: employing a unified autoregressive architecture to jointly encode composed inputs across modalities. While this unified encoder paradigm enables expressive cross-modal attention, it inadvertently exacerbates the vulnerability to modality shortcut learning. Because the shared Transformer weights process all tokens jointly under a standard contrastive loss, the objective can be effortlessly minimized by over-relying on a single dominant modality (e.g., matching coarse background visual similarity while discarding textual instruction modifiers, or latching onto prominent lexical keywords while ignoring the base image). Consequently, under distribution shifts or out-of-domain (OOD) scenarios, this shortcut reliance leads to severe performance degradation.

The central tension lies in the fact that conventional contrastive learning solely encourages proximity between the joint query embedding and the positive target, lacking any structural constraint to ensure that the multimodal representation is both strictly more discriminative than its unimodal components and geometrically anchored to them. Core idea: explicitly model the structural relationship between composed multimodal representations and their unimodal constituents by enforcing a preference margin over unimodal counterparts and regularizing the composed embedding against a lightweight unimodal prototype.

Method

Overall Architecture

The overall MCA pipeline centers on a unified MLLM encoder \(f_\theta(\cdot)\) (e.g., Qwen2-VL). For any multimodal composed input \(x\) (such as an image paired with an editing text instruction), the framework simultaneously extracts the joint composed embedding \(f_\theta(x)\) as well as unimodal representations for each constituent part \(x_m \in \mathcal{U}(x) = \{x_{\text{text}}, x_{\text{image}}\}\). During training, two complementary objectives are integrated alongside the primary contrastive loss: Modality Composition Preference (MCP), which penalizes instances where unimodal components yield higher target retrieval similarity than the composed input, and Modality Composition Regularization (MCR), which computes a channel-gated prototype from unimodal embeddings and contrasts it against the joint embedding to prevent semantic drift.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Composed Input x<br/>(Image + Text Instruction)"] --> B["Unified MLLM Encoder<br/>Joint Forward Extraction"]
    A --> C["Unimodal Part Decomposition<br/>U(x) = {x_text, x_image}"]
    C --> B
    B --> D["Modality Composition Preference (MCP)<br/>Enforce sim(x, y+) > sim(x_m, y+)"]
    B --> E["Lightweight Mixer Module mix_phi<br/>Compute Dimension-Gated Prototype"]
    E --> F["Modality Composition Regularization (MCR)<br/>Contrast Joint Embedding with Prototype"]
    D --> G["Joint Multi-Task Objective<br/>L_CL + alpha*L_MCP + beta*L_MCR"]
    F --> G
    G --> H["Robust Composed Retrieval Space"]

Key Designs

1. Modality Composition Preference (MCP): penalizing unimodal shortcut dominance

Conventional contrastive learning optimizes only the similarity between the joint embedding \(f_\theta(x)\) and target \(y^+\), permitting a degenerate solution where \(f_\theta(x) \approx f_\theta(x_m)\) that ignores complementary modality cues. MCP introduces a preference-style constraint directly on the target matching scores: for any positive target \(y^+\), the similarity produced by the composed representation must strictly surpass that of any constituent unimodal part \(x_m \in \mathcal{U}(x)\):

\[\mathcal{L}_{\text{MCP}} = \mathbb{E}_{x, y^+ \sim \mathcal{D}} \left[ - \sum_{x_m \in \mathcal{U}(x)} \log \frac{e^{\text{sim}_\theta(x, y^+)}}{e^{\text{sim}_\theta(x, y^+)} + e^{\text{sim}_\theta(x_m, y^+)}} \right]\]

This objective acts as a relative ranking penalty. If the encoder attempts to shortcut the task by relying solely on the vision signal or text prompt, \(\text{sim}_\theta(x_m, y^+)\) matches or exceeds \(\text{sim}_\theta(x, y^+)\), incurring a sharp loss penalty. To maintain embedding symmetry, MCP is applied bidirectionally whenever either the query or target document is multimodal.

2. Modality Composition Regularization (MCR): anchoring representation space to unimodal prototypes

While MCP enforces relative discriminative dominance, the high-dimensional capacity of an MLLM could still cause the composed embedding to drift into spurious, ungrounded latent manifolds. MCR establishes topological consistency by requiring the composed embedding \(f_\theta(x)\) to remain anchored within the semantic subspace spanned by its constituent unimodal parts. A lightweight aggregation mixer \(\text{mix}_\phi\) synthesizes a prototype from unimodal embeddings \(\{h_m\}_{m=1}^M\):

\[[g_1; \dots; g_M] = \text{softmax}_m(W[h_1; \dots; h_M] + b), \quad \text{mix}_{\text{gated},\phi}(\{h_m\}_{m=1}^M) = \sum_{m=1}^M g_m \odot h_m\]

where gating coefficients \(g_m\) are normalized across modalities per embedding dimension. The MCR objective then pulls the composed embedding closer to its own prototype relative to in-batch negative prototypes:

\[\mathcal{L}_{\text{MCR}} = \mathbb{E}_{x, \mathcal{X} \sim \mathcal{D}} \left[ - \log \frac{e^{\text{sim}_{\text{mix}}(x, \mathcal{U}(x))}}{e^{\text{sim}_{\text{mix}}(x, \mathcal{U}(x))} + \sum_{x' \in \mathcal{X}} e^{\text{sim}_{\text{mix}}(x, \mathcal{U}(x'))}} \right]\]

Crucially, \(\text{mix}_\phi\) is kept minimally parameterized or non-parametric. Because the mixer lacks sufficient capacity to overfit, the expressive MLLM encoder is forced to structure its joint latent space in harmony with the constituent parts without collapsing into trivial identity mapping.

3. Composition Margin Diagnostic and Quality-Adaptive Weighting

To quantify the shortcut phenomenon, the paper introduces the composition margin diagnostic metric \(\Delta_{\text{comp}}(x, y^+) = \text{sim}[f_\theta(x), f_\theta(y^+)] - \max_{x_m} \text{sim}[f_\theta(x_m), f_\theta(y^+)]\). A shrinking margin signals escalating shortcut vulnerability. When visual input is degraded (e.g., low image resolution of \(128 \times 128\)), the model experiences a strong optimization pull toward text-only shortcuts. Under such modality imbalance, tiny regularization weights (\(\alpha=\beta=0.01\)) fail to escape the shortcut attractor, necessitating stronger regularization (\(\alpha=\beta=1.0\)). Conversely, when input modalities are rich and balanced (e.g., \(1344 \times 1344\)), lighter regularization (\(\alpha=\beta=0.01\)) suffices to prevent over-constraining the model while ensuring optimal OOD generalization.

Loss & Training

The complete multi-task optimization objective incorporates the standard InfoNCE contrastive loss alongside the two proposed regularizers:

\[\mathcal{L} = \mathcal{L}_{\text{CL}} + \alpha \mathcal{L}_{\text{MCP}} + \beta \mathcal{L}_{\text{MCR}}\]

The primary backbone is Qwen2-VL-2B-Instruct fine-tuned with LoRA (\(r=8\), \(\alpha_{\text{lora}}=16\)) on 8\(\times\) Nvidia A100 GPUs using AdamW (learning rate \(2\times 10^{-5}\), linear warmup for 200 steps, 2000 total steps) with a global batch size of 1024 and contrastive temperature \(\tau=0.02\). Training batches mix unimodal and composed pairs across 6 datasets (MSCOCO, VisualNews, VisDial, CIRR, NIGHTS, WebQA), with auxiliary losses activated dynamically only when composed inputs appear. At inference time, the mixer module \(\text{mix}_\phi\) is completely discarded, incurring zero computational or memory overhead.

Key Experimental Results

Main Results

Evaluation covers in-domain (IND) benchmarks, out-of-domain (OOD) composed retrieval benchmarks, and zero-shot multimodal grounding tasks. All figures report Accuracy@1 (%).

Model Class Representative Method IND Retrieval Avg. OOD Retrieval Avg. Zero-shot Grounding Avg.
Dual-Encoder CLIP (ViT-B/32) 55.8 44.5 51.7
Dual-Encoder SigLIP 46.7 33.2 59.5
Unified MLLM E5-V 11.6 11.8 18.9
Unified MLLM VLM2Vec (Supervised Grounding) 73.5 53.5 -
Controlled Baseline Qwen2-VL-2B (Vanilla CL) 73.2 54.2 57.9
Proposed Method MCA (Ours, 2B) 73.2 (\(\Delta\) 0.0) 57.4 (\(\Delta\) +3.2) 61.0 (\(\Delta\) +3.1)

On specific OOD benchmarks, MCA demonstrates substantial gains: OVEN rises from 63.0 to 66.7, FashionIQ from 15.6 to 19.3, and EDIS from 84.0 to 86.1. In zero-shot grounding, RefCOCO improves from 60.3 to 65.0 and MSCOCO from 41.0 to 42.7.

Ablation Study

Detailed loss breakdown and prototype mixer architecture ablation on Qwen2-VL-2B:

Configuration IND Avg. OOD Retrieval Avg. Zero-shot Grounding Avg. OOD Overall Avg.
Vanilla CL Baseline 73.2 54.2 57.9 56.3
+ MCP only (CL + MCP) 73.2 55.3 (+1.1) 58.2 (+0.3) 56.9 (+0.6)
+ MCR only (CL + MCR) 73.2 55.4 (+1.2) 58.8 (+0.9) 57.3 (+1.0)
Full MCA (CL + MCP + MCR) 73.2 57.4 (+3.2) 61.0 (+3.1) 59.4 (+3.1)
Mixer: Mean Pooling 73.3 54.8 (+0.6) 57.8 (-0.1) 56.5 (+0.2)
Mixer: Gated Fusion (Default) 73.2 57.4 (+3.2) 61.0 (+3.1) 59.4 (+3.1)
Mixer: Bilinear Pooling (MFB) 73.0 54.6 (+0.4) 60.0 (+2.1) 57.7 (+1.4)

Key Findings

  • Synergistic Complementarity of Objectives: Isolated application of either MCP or MCR delivers modest OOD gains (+0.6% and +1.0% respectively), whereas combining preference ranking with prototype anchoring unlocks a +3.1% overall OOD improvement, confirming their orthogonal roles in suppressing shortcut collapse.
  • Inverse Law of Input Quality and Regularization: Under degraded \(128 \times 128\) resolution, visual representations are noisy, creating a strong text shortcut attractor. Weak regularization (\(\alpha=\beta=0.01\)) destabilizes optimization (-11.6% drop on OOD retrieval), while robust weighting (\(\alpha=\beta=1.0\)) yields a massive +13.1% surge. Conversely, high-resolution inputs (\(1344 \times 1344\)) favor smaller weights (0.01) to preserve fine-grained semantic plasticity.
  • Composition Margin Diagnostic Validation: On held-out probes from CIRR and NIGHTS, standard CL training causes the composition margin \(\Delta_{\text{comp}}\) to contract from 0.1744 down to 0.1248 (-28.4%), proving an increased reliance on unimodal shortcuts. In contrast, MCA expands the margin to 0.2125 (+21.8% over base) and reduces preference violations by 85.0%.

Highlights & Insights

  • Root-Cause Solution for MLLM Modality Bias: The paper establishes that unified autoregressive encoders are inherently prone to modality shortcuts under symmetric contrastive learning, formalizing preference and prototype constraints as general solutions.
  • Training-Time Regularizer with Zero Inference Overhead: All unimodal forward passes and prototype mixers exist strictly during backpropagation. Inference preserves the standard single-pass architecture without additional latency or memory footprints.
  • Omni-Modal Scalability: Validated on Qwen2.5-Omni-3B across 67 tasks spanning image, audio, and video compositions, yielding a consistent +1.18 point gain on composed tasks and demonstrating general applicability across diverse modality mixtures.

Limitations & Future Work

  • Training Computational Overhead: Computing separate unimodal embeddings alongside the composed input introduces a 2.26\(\times\) wall-clock training overhead on 8\(\times\) A100 GPUs. Future implementations could exploit autoregressive causal caching to reuse intermediate hidden states of unimodal prefixes.
  • Hyperparameter Sensitivity under Severe Noise: When one modality suffers extreme signal degradation, optimal regularization coefficients require careful tuning to prevent optimization instability.
  • Restricted Scope for Pure Unimodal Queries: MCA triggers only when at least one side of the query-document pair is multimodal, leaving unimodal-to-unimodal retrieval under standard contrastive learning.
  • vs Dual-Encoder Fusion (UniIR / CoLLM): UniIR relies on frozen unimodal encoders with late fusion layers, lacking deep unified cross-attention; CoLLM depends on LLM-synthesized prompt augmentations. MCA operates directly on the unified MLLM space, correcting intrinsic representation bias via geometric constraints.
  • vs Standard MLLM Retrievers (E5-V / VLM2Vec): E5-V and VLM2Vec directly adapt MLLMs with conventional InfoNCE or supervised multi-task fine-tuning. MCA proves that without adding supervised task annotations, structural composition awareness alone unlocks superior out-of-domain robustness.

Rating

  • Novelty: โญโญโญโญ [Pinpoints modality shortcuts in unified MLLM retrievers and proposes a principled dual-objective formulation combining preference optimization and prototype regularization.]
  • Experimental Thoroughness: โญโญโญโญโญ [Exhaustive evaluation across in-domain and OOD retrieval, zero-shot grounding, model scales (2B/7B), omni-modal tasks (67 datasets), diagnostic probes, and t-SNE analysis.]
  • Writing Quality: โญโญโญโญโญ [Crisp formulation, precise motivation, transparent reporting of failure cases, and clear diagrams.]
  • Value: โญโญโญโญ [Provides an actionable, zero-inference-overhead training recipe for next-generation unified multimodal retrievers.]