Skip to content

MARS: What Retrieval Signals Are Hidden in Multimodal Large Language Models for Text-Video Retrieval?

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/sejong-rcv/MARS
Area: LLM Reasoning
Keywords: Text-Video Retrieval, Multimodal Large Language Models, Multi-Layer Fusion, Adaptive Representation Slots, Hard-Negative Specialization

TL;DR

Addressing the severe information bottleneck caused by single-token single-layer embedding extraction in MLLM-based text-video retrieval, MARS introduces multi-layer adaptive representation slots, slot-aligned similarity aggregation, and hard-negative-aware slot specialization, achieving new state-of-the-art performance across four benchmarks within an efficient dual-encoder framework.

Background & Motivation

Recent advances in multimodal large language models (MLLMs) have spurred growing interest in adapting them as universal embedding models for discriminative representation learning. Existing MLLM-based embedders predominantly rely on prompt-based single-token extraction schemes—such as appending summary prompts like "summarize the input in one word" and extracting the hidden state of a single specific token (e.g., [EOS] or the final token) from the last decoder layer. While this single-vector paradigm functions adequately for coarse-grained image classification or broad semantic matching, it encounters an acute representational bottleneck in text-video retrieval.

Video content is inherently characterized by dense temporal dynamics, multi-granular compositional entities, and evolving relational interactions. Forcing these multifaceted visual-linguistic signals into a single global vector inevitably entangles heterogeneous semantic cues—such as actor identities, subtle action transitions, and background context—obscuring critical discriminative details needed to differentiate visually similar candidate videos. Furthermore, a deeper structural tension exists between model capacity and extraction mechanisms: language models distribute valuable syntactic, compositional, and relational evidence across multiple intermediate and upper decoder layers rather than concentrating everything exclusively at the terminal layer. Relying solely on the final layer discards substantial complementary evidence encoded throughout the network depth.

To overcome both the loss of layer-distributed cues and the semantic collapse of single-vector representations, text-video retrieval requires an embedding framework capable of mining multi-layer signals and preserving multi-granular semantic dimensions without full vector collapse. Core idea: Introduce learnable adaptive representation tokens to construct multiple feature slots via weighted multi-layer decoder fusion, conduct slot-aligned similarity aggregation, and apply hard-negative-aware slot specialization to encourage distinct slots to capture complementary discriminative cues.

Method

Overall Architecture

MARS operates within an efficient dual-encoder retrieval paradigm consisting of four core components: representation token prompting, layer-fused slot construction, slot-wise matching, and hard-negative-aware slot specialization. Given paired text and video inputs, \(M\) learnable representation tokens are appended to the response field of a shared chat prompt, and their hidden states are gathered across all \(N\) decoder layers. Next, slot-specific learnable layer weights fuse these hidden states into normalized slot embeddings. The model computes slot-aligned cosine similarities and aggregates them uniformly. During training, it optimizes a symmetric contrastive loss alongside slot diversity regularization and a hard-negative-aware margin objective.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Text prompt / 16-frame video sequence"] --> B["Adaptive Representation Token Prompting<br/>Append M shared prompt tokens: slot_1...slot_M"]
    B --> C["MLLM Backbone Forward Pass<br/>Extract 28-layer hidden states for each slot"]
    C --> D["Multi-Layer Adaptive Slot Construction<br/>Softmax-normalized learnable layer fusion"]
    D --> E["Slot-Aligned Matching & Score Aggregation<br/>Compute slot-wise cosine similarity and uniform average"]
    E --> F["Hard-Negative-Aware Slot Specialization<br/>Identify hardest negative and widen maximal slot margin"]
    F --> G["Dual-Encoder Retrieval & Ranking Scores"]

Key Designs

1. Adaptive Representation Token Prompting: Expanding single-token bottlenecks into multi-slot channels

Standard MLLM embedding extraction collapses entire multimodal inputs onto a single terminal token, creating an information bottleneck for complex video clips. MARS addresses this by appending \(M\) learnable adaptive representation tokens \(\{e_{\text{slot}}^m\}_{m=1}^M \subset \mathbb{R}^D\) into the assistant response field of a structured chat prompt: user: {caption or video} Represent this {text|video} in detail. assistant: <slot_1> ... <slot_M>. Because the MLLM enforces causal self-attention, the hidden states corresponding to these prompt tokens naturally condition on the preceding multimodal context. Sharing these tokens across text and video inputs establishes structural slot correspondence across modalities from the outset.

2. Multi-Layer Adaptive Slot Construction: Unlocking distributed intermediate evidence beyond the final layer

While prior works assume the final decoder layer is universally optimal, empirical analysis demonstrates that retrieval-relevant cues are dispersed across intermediate and upper decoder layers. MARS assigns a slot-specific learnable weight vector \(\mathbf{w}_m \in \mathbb{R}^N\) across all \(N\) decoder layers for the \(m\)-th slot, normalized via Softmax:

\[\alpha_{m,n} = \frac{\exp(w_{m,n})}{\sum_{n'=1}^N \exp(w_{m,n'})}\]

The text and video slot representations are formed by convex combinations of layer hidden states: \(\mathbf{z}_{t_i, m} = \sum_{n=1}^N \alpha_{m,n} \mathbf{h}_{i,m,t}^{(n)}\) and \(\mathbf{z}_{v_j, m} = \sum_{n=1}^N \alpha_{m,n} \mathbf{h}_{j,m,v}^{(n)}\), followed by \(L_2\)-normalization to produce \(\hat{\mathbf{z}}\). This parameterization allows each slot to dynamically focus on different abstraction depths within the MLLM hierarchy.

3. Slot-Aligned Matching & Score Aggregation: Preserving fine-grained cues without pre-mature vector pooling

To prevent disentangled sub-representations from collapsing into an entangled vector prior to comparison, MARS performs slot-aligned matching. For the \(m\)-th slot, the slot-wise cosine similarity between text sample \(i\) and video sample \(j\) is computed directly as \(S_m[i,j] = \langle \hat{\mathbf{z}}_{t_i, m}, \hat{\mathbf{z}}_{v_j, m} \rangle\). The final holistic retrieval score aggregates similarities uniformly across all \(M\) slots:

\[S[i,j] = \frac{1}{M} \sum_{m=1}^M S_m[i,j]\]

Ablations confirm that uniform averaging achieves more robust empirical performance than parameterized learnable slot weighting or MaxSim extreme-value selection, ensuring every specialized slot consistently contributes to the final ranking decision.

4. Hard-Negative-Aware Slot Specialization: Inducing functional specialization on confusing distractors

Without explicit incentives, multiple slots risk collapsing into redundant copies during standard contrastive training. MARS introduces a slot specialization objective grounded in hardest negatives. For each query side \(a \in \{t, v\}\), the framework identifies the most confusing mismatched candidate \(j^*(i) = \arg\max_{j \ne i} S_a[i,j]\) and evaluates the slot-wise discrimination gap:

\[\Delta_{m}^a(i) = S_m^a[i,i] - S_m^a[i, j^*(i)]\]

By identifying the maximum gap across all slots, \(\text{gap}_a(i) = \max_{1 \le m \le M} \Delta_m^a(i)\), the objective penalizes cases where the strongest separating slot fails to exceed margin \(\delta\):

\[\mathcal{L}_{\text{hn}}^a = \frac{1}{B} \sum_{i=1}^B \max(0, \delta - \text{gap}_a(i))\]

Because different hard negatives confuse the model along distinct semantic axes, the \(\max\) operator dynamically delegates separation duty to the most competent slot, driving functional specialization across slots.

Loss & Training

The overall training objective combines three terms: symmetric InfoNCE contrastive alignment \(\mathcal{L}_{\text{NCE}}\), within-modality slot diversity regularization \(\mathcal{L}_{\text{div}}\), and hard-negative-aware slot specialization \(\mathcal{L}_{\text{hn}}\):

\[\mathcal{L} = \mathcal{L}_{\text{NCE}} + \lambda_{\text{div}} \mathcal{L}_{\text{div}} + \lambda_{\text{hn}} \mathcal{L}_{\text{hn}}\]

The diversity regularizer penalizes high squared cosine similarity between different slots of the same input: \(\mathcal{L}_{\text{div}}^a = \frac{1}{B M (M-1)} \sum_{i=1}^B \sum_{m \ne m'} (\hat{\mathbf{z}}_{a_i, m}^\top \hat{\mathbf{z}}_{a_i, m'})^2\). Training is conducted on two NVIDIA A100 40GB GPUs using VideoChat-Flash 7B as the base MLLM with frozen pretrained weights, optimizing only the prompt tokens, layer-fusion weights, and LoRA parameters applied to the multimodal projection layer and Qwen2 decoder (\(r=128, \alpha=256, \text{dropout}=0.05\)). Hyper-parameters are set to \(\lambda_{\text{div}} = 0.1\), \(\lambda_{\text{hn}} = 0.05\), and margin \(\delta = 0.04\).

Key Experimental Results

Main Results

Evaluations across four established text-video retrieval benchmarks (DiDeMo, ActivityNet, LSMDC, and MSR-VTT) demonstrate that MARS sets new state-of-the-art benchmarks in both direct similarity-based retrieval and reranking paradigms.

Protocol & Method DiDeMo (T2V R@1) ActivityNet (T2V R@1) LSMDC (T2V R@1) MSR-VTT (T2V R@1) Avg. T2V mR@1 Avg. V2T mR@1
Direct Similarity Retrieval
CLIP4Clip (Luo et al.) 42.8 40.5 21.6 44.5 37.4 37.3
CLIP-ViP (Xue et al.) 48.6 51.1 25.6 50.1 43.9 –
ViCLIP (Wang et al.) 49.4 49.8 33.0 52.5 46.2 45.6
InternVideo (Wang et al.) 57.9 62.2 34.0 55.2 52.3 53.7
MARS (Ours) 79.7 75.7 51.0 61.6 67.0 64.5
MARS* (Ours + DSL) 84.6 82.1 54.4 65.5 71.6 72.0
Cross-Modal Reranking
UMT (Li et al.) 70.4 66.8 43.0 58.8 59.8 58.1
InternVideo2-1B* (Wang et al.) 72.8 66.2 42.7 58.7 60.1 57.8
InternVideo2-6B* (Wang et al.) 74.2 74.1 46.4 62.8 64.4 61.9
BLiM (Ko et al.) 86.4 81.0 55.7 64.7 72.0 67.1
MARS-R (Ours + BLiM) 87.6 83.5 56.1 65.8 73.2 67.9

Ablation Study

Component-wise ablations isolate the contribution of multi-layer fusion (LF), adaptive slots (Slots), diversity loss (\(\mathcal{L}_{\text{div}}\)), and hard-negative specialization (\(\mathcal{L}_{\text{hn}}\)) across benchmarks (R@1 metric):

Configuration Variant DiDeMo (T2V/V2T) ActivityNet (T2V/V2T) LSMDC (T2V/V2T) MSR-VTT (T2V/V2T) Global Avg. R@1 Incremental Gain
1. Final-Layer Single-Slot Baseline 72.9 / 71.5 69.5 / 66.0 47.5 / 47.1 58.3 / 56.7 61.2 Base baseline
2. + Multi-Layer Fusion (LF) 76.6 / 74.9 72.7 / 70.0 47.9 / 46.6 59.8 / 57.9 63.3 (+2.1) Largest individual gain
3. + Adaptive Representation Slots 77.4 / 75.0 75.6 / 73.6 49.9 / 50.2 60.8 / 57.9 65.1 (+1.8) Multi-slot composition
4. + Slot Diversity Loss (\(\mathcal{L}_{\text{div}}\)) 77.5 / 75.0 75.7 / 73.2 50.9 / 49.7 61.1 / 59.0 65.3 (+0.2) Prevents slot collapse
5. Full Model (+ \(\mathcal{L}_{\text{hn}}\)) 79.7 / 75.7 75.7 / 73.2 51.0 / 50.0 61.6 / 59.0 65.7 (+0.4) Optimal fine-grained margins

Key Findings

  • Multi-layer evidence aggregation: While single-layer baselines peak sharply at the terminal layer, multi-layer training enables multiple intermediate-to-high layers to develop comparable retrieval accuracy. Notably, learned layer weights do not mirror single-layer R@1 curves; intermediate layers with moderate standalone performance receive non-trivial weights, proving their complementary utility.
  • Slot complementarity in action: Uniform aggregation substantially outperforms any standalone individual slot (e.g., +12.3 R@1 gain on DiDeMo and +10.2 on ActivityNet). Furthermore, counting "slot-exclusive rank-1 cases" reveals hundreds of queries (785 on ActivityNet, 209 on MSR-VTT) correctly retrieved at rank 1 exclusively by one specific slot while failing under others.
  • Optimal slot capacity: Varying slot count \(M\) demonstrates an optimal operating point at \(M=4\) (average R@1 reaches 77.7 on DiDeMo vs. 75.6 at \(M=2\)), whereas further increasing to \(M=6\) degrades accuracy (75.9), indicating that excessive slots inject noise or redundant representations.
  • Drastic online computational savings: As a dual-encoder model with precomputed offline video embeddings, MARS achieves exceptionally low online FLOPs per text query (e.g., 322.3 GFLOPs on DiDeMo vs. 4,576.1 GFLOPs for InternVideo2-1B) and evaluates entire test splits on an A100 GPU in 3,626 seconds compared to 10,424 seconds for InternVideo2-6B.

Highlights & Insights

  • Rethinking MLLM embedding extraction: This paper provides the first systematic evidence that compressing an MLLM into a single-token final-layer vector creates an unnecessary bottleneck for video retrieval, presenting multi-layer multi-slot extraction as a principled paradigm.
  • Decoupled hard-negative specialization: Rather than penalizing all representation channels uniformly, the max-margin formulation \(\max_m \Delta_m\) assigns separation responsibility to the single most discriminative slot for each specific hard negative, fostering true functional division of labor.
  • Broad cross-backbone and cross-modal generality: MARS achieves consistent +2.3 to +3.1 average R@1 gains across diverse MLLM backbones (VideoLLaMA3-7B, Qwen2-VL-7B, Qwen2-VL-2B) and generalizes out-of-the-box to text-image retrieval on COCO and Flickr30K, outperforming BEiT-3 without architectural modifications.

Limitations & Future Work

  • Evaluation on uncurated long-form open-domain videos: Benchmarks tested primarily feature trimmed clips or pre-segmented event videos; performance on unconstrained multi-hour videos with noisy automatic speech recognition (ASR) transcripts remains unverified.
  • Lack of temporal grounding capabilities: MARS focuses on clip-level retrieval and does not output explicit timestamp intervals or localized bounding boxes for moment retrieval.
  • Extension to non-visual modalities: While validated across video and image modalities, extending adaptive representation slots to 3D point clouds, spatial-audio, or sensor-linguistic retrieval remains open for exploration.
  • vs CLIP4Clip / InternVideo / ViCLIP: Conventional video foundation models rely on lightweight text encoders paired with visual temporal pooling, missing the rich world knowledge and linguistic reasoning of MLLMs; MARS leverages MLLM reasoning without falling into single-vector compression traps.
  • vs InternVideo2 / BLiM: Heavy reranking architectures depend on costly cross-attention decoders for quadratic candidate scoring; MARS outperforms them in direct dual-encoder retrieval while cutting online query latency by an order of magnitude.
  • vs E5-V / VLM2Vec: General-purpose MLLM embedders extract single-token global vectors; MARS demonstrates that multi-layer weighted fusion combined with multi-slot alignment is critical for fine-grained multimodal matching.

Rating

  • Novelty: ⭐⭐⭐⭐ Elegant identification of MLLM layer-wise potential and intuitive multi-slot specialization design.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across four video benchmarks, two image benchmarks, three MLLM backbones, plus CKA/effective rank geometric diagnostics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear structure, crisp prose, and comprehensive empirical dissections.
  • Value: ⭐⭐⭐⭐⭐ Delivers state-of-the-art retrieval accuracy with exceptional inference efficiency, offering high practical value for large-scale multimodal search systems.