Adapting MLLMs for Nuanced Video Retrieval¶

Conference: ECCV2026
Paper: Official paper page · PDF
Project: TARA
Authors: Piyush Bagad, Andrew Zisserman
Area: Multimodal VLM
Keywords: nuanced video retrieval, temporally opposite actions, negation understanding, composed video retrieval, modality gap

TL;DR¶

TARA turns a generative MLLM into a unified retrieval encoder by fine-tuning its shared language model on just 20,000 carefully constructed text triplets, raising Tarsier 2's CiA-SSv2 Chiral mAP from 77.7 to 88.9 while also improving negated queries and composed video retrieval.

Background & Motivation¶

Searching for someone closing a door is not the same as finding someone standing next to a door. When opening and closing appear together in the gallery, recognizing people, objects, and scenes is insufficient: the model must preserve action direction. Similarly, a query about a dog that is not on grass requires negation understanding, while a reference video accompanied by an instruction to turn red flowers yellow requires retaining the reference content and applying a semantic change. These demands test temporal, negation, and multimodal composition abilities, yet static shortcuts in conventional retrieval benchmarks can conceal failures on all three.

Pretrained MLLMs already possess video and language knowledge, but their objective usually involves generating answers rather than arranging input representations for cosine retrieval. Reading hidden states directly does not guarantee sensitivity to subtle meanings. Standard NLI contrastive adaptation can also miss the problem: a negative sentence may change the action, object, and background simultaneously, allowing the model to solve training examples through easy static differences. Adding large quantities of paired video data is possible, but does not itself ensure that the learning signal targets the distinctions that matter.

The paper instead controls the semantic structure of text negatives. Most of a description is preserved while action direction, negation, or whether an edit was applied changes. The shared language backbone then offers a route for transferring those distinctions to video inputs. Here, text-only refers to this adaptation stage, not to a backbone that has never learned from visual data. Core idea: reshape a shared embedding space with text hard negatives targeting subtle meanings, improving real video retrieval without using videos during adaptation.

Method¶

Overall Architecture¶

TARA stands for Text Adapted Retrieval Alignment. It constructs the NLI-Nuance text-triplet dataset, extracts final-layer hidden representations through a one-word summarization prompt, and updates the shared LLM using text contrastive learning. At inference, the adapted encoder accepts text queries, videos, or a video paired with an edit instruction. Queries and candidates are embedded separately and ranked by similarity, without generating a textual answer for each candidate.

The training and inference paths are distinct: existing descriptions stand in for videos during training, whereas inference uses actual video inputs. The vision encoder and projection network are frozen, but video tokens still pass through the updated LLM. Optimizing a text objective can therefore change the final video representation. The default implementation uses Tarsier 2 and uniformly samples 16 frames per candidate video.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["NLI and training captions"] --> B["Nuance-targeted<br/>triplet construction"]
    B --> C["One-word<br/>summary embeddings"]
    C --> D["Text contrastive adaptation<br/>of the shared backbone"]
    D --> E["Adapted unified encoder"]
    F["Text or video-plus-edit query<br/>Actual candidate videos"] --> E
    E --> G["Similarity ranking<br/>Retrieved videos"]

Key Designs¶

1. Nuance-targeted triplet construction: make negatives violate the semantic condition that matters

NLI-Nuance contains 8,000 ordinary NLI triplets plus 1,000 temporal, 1,000 negation, and 10,000 composition triplets, totaling 20,000. Ordinary NLI examples help preserve general semantic discrimination; the added examples prevent the model from succeeding through topic similarity alone. The temporal branch extracts verb-object pairs from Ego4D training captions and selects actions using a pregenerated list of temporally opposite verbs. Positives share the action verb, while Qwen 3 generates a negative with the opposite direction and replaces first-person markers with natural subject descriptions. For example, picking up fruit and picking an apple may form a positive pair, while putting down fruit is the negative. Positives need not be literal paraphrases: they agree in the target action semantics. Verification details are deferred to another version; this cache only states that LLM judging and manual review are used.

The negation branch selects NLI negatives containing negation operators and, as illustrated in the construction diagram, checks their negation relationship with the anchor rather than treating the presence of “not” as sufficient. The composition branch replaces source and target videos in WebVid-CoVR training examples with their captions. Its anchor combines the source description and edit instruction, its positive is the edited target description, and its negative is the unchanged source description. That negative forces the model to recognize whether the edit has been applied instead of merely reproducing the source content. All three branches use the same triplet interface while targeting different errors. The paper explicitly excludes evaluation-set captions from training.

2. One-word summary embeddings: give different inputs a shared semantic readout interface

The method follows Explicit One-word Limitation, or EOL: the model is prompted to summarize an input in one word, and the final-layer hidden vector at the readout position serves as its embedding. Retrieval uses the high-dimensional hidden state, not the emitted word, so the representation is not restricted to a single discrete category. Compared with choosing an arbitrary token position, the prompt gives the model an explicit semantic aggregation task and provides similar output interfaces for text and video.

Composed queries require a more specific prompt. After receiving a source video and an edit instruction, the model is asked to imagine that the instruction has been applied and summarize the resulting content. The query embedding is therefore intended to represent target semantics rather than an uninterpreted combination of a reference video and a sentence. This is not a video generator and does not produce an edited video; actual candidate videos remain independently encoded. Embedding extraction builds on prior work, while the paper emphasizes its use with video and composed inputs together with targeted text adaptation.

3. Text contrastive adaptation of the shared backbone: let language-side distinctions affect video representations

Training increases the anchor's cosine similarity to its positive relative to other positives and hard negatives in the batch, with a temperature controlling the contrastive scoring scale. Only the shared LLM weights are updated; the vision encoder and visual projection remain frozen. Because both text and video tokens eventually traverse that backbone, adaptation changes final hidden representations for videos as well as text. This relies on cross-modal correspondences already present in the pretrained model: it does not create visual action perception from text alone.

The authors explain part of the gain through the modality gap. Even with a shared backbone, separate input pathways can leave video and text representations systematically offset. The paper measures this offset as the Euclidean norm of the mean difference between paired embeddings. The following is a mathematical rendering of that intact prose definition, not a reconstruction of the damaged training equation:

\[ \|\Delta_{\mathrm{gap}}\|_2 =\left\|\frac{1}{N}\sum_{i=1}^{N} \left(e_{v_i}-e_{t_i}\right)\right\|_2. \]

Here, \(N\) is the number of paired samples, and \(e_{v_i}\) and \(e_{t_i}\) are the corresponding video and text embeddings. This measures an overall centroid offset, not the average error of individual paired representations. The authors hypothesize that contrastive uniformity pressure spreads embeddings out and brings modality centroids closer, leaving more representational capacity for fine-grained semantics. Their evidence supports this explanation but does not establish it as the sole cause of every improvement.

A Worked Example¶

Consider the paper's example of blooming red flowers with an instruction to make them yellow. During training, the anchor contains only the red-flower caption and edit request, the positive describes yellow flowers, and the negative still describes red flowers. Although the negative almost exactly matches the source caption, the model must push it away because it fails to satisfy the requested edit. This is precisely the distinction that general topic matching can overlook.

At inference, the red-flower video's 16 frames and the same edit request enter the composed EOL prompt, producing a query embedding intended to describe the yellow-flower target. Candidate videos are independently embedded with the same encoder and ranked by similarity. This example explains the training-to-inference connection rather than reporting an actual observed ranking, and it does not require candidate captions during inference.

Loss & Training¶

The default setup trains for 2 epochs with a batch size of 768 and a base learning rate of \(2\times10^{-5}\). The authors report adaptation in less than 1 hour on 8 RTX A6000 GPUs. Inference uniformly samples 16 frames. Equation (2), the contrastive loss, is severely corrupted in the cached text extraction, so this note retains its verifiable training semantics without inventing summation boundaries, normalization details, or an undisclosed temperature value.

The theoretical analysis further assumes that a paired video-text embedding difference consists of a constant modality offset orthogonal to both representation spaces, plus Gaussian alignment noise. The constant offset cancels when subtracting positive and negative video embeddings; a convexity argument then places the cross-modal loss above the corresponding text loss due to alignment noise. This provides a conditional explanation for transferring improved text discrimination to cross-modal discrimination. It depends on assumptions such as fixed noise and is not a guarantee that arbitrary data or training trajectories always improve retrieval.

Key Experimental Results¶

Main Results¶

CiA's Chiral setting retrieves among the correct action and its temporal opposite; All also includes other actions. Both use mAP. RTime retrieves between an original and reversed video or their corresponding captions and reports R@1. NegBench's R-Neg@5 is top-5 recall for negated queries, while WebVid-CoVR retrieves target videos from video-plus-edit queries. All gains below are metric-point differences, not relative percentages.

Evaluation and metric	Comparator	Comparator result	Tarsier 2 + TARA	Gain
CiA-SSv2 Chiral mAP	Tarsier 2	77.7	88.9	+11.2
CiA-SSv2 All mAP	Tarsier 2	24.0	58.6	+34.6
CiA-EPIC All mAP	Tarsier 2	15.3	38.9	+23.6
CiA-Charades All mAP	Tarsier 2	9.2	29.0	+19.8
RTime T2V R@1	Tarsier 2	58.8	67.2	+8.4
RTime V2T R@1	Tarsier 2	59.5	77.9	+18.4
NegBench MSRVTT R-Neg@5	Tarsier 2	18.9	65.0	+46.1
WebVid-CoVR R@1	CoVR-2, Ventura et al.	59.8	66.3	+6.5

Sources are Tables 2 through 5. The last row compares different methods rather than ablating the same backbone. The CoVR test set has 2,556 query-video examples. The paper labels TARA's CoVR evaluation zero-shot, but adaptation uses captions and edit information from the WebVid-CoVR training set. This should mean adaptation without CoVR videos, not complete absence of exposure to task-specific training information.

Ablation Study¶

The cache explicitly defers training-data composition and scale ablations to an unavailable arXiv version, so it cannot support numerical claims about removing temporal examples. The following preserves the fixed-Qwen2VL-7B comparison from Table 7 to reduce backbone confounding. It is still not a strict single-factor ablation.

Method	Adaptation modalities	CiA-SSv2 Chiral	CiA-SSv2 All	RTime T2V	CoVR R@1	Original Avg.
Base	No extra adaptation	60.2	17.3	59.9	15.5	40.0
ArrowRL	Video, text	67.5	22.5	57.1	41.8	51.3
CaRe Stage 2	Video, text	66.4	23.7	59.8	35.6	51.9
Base + TARA	Text	72.7	28.3	65.9	44.8	56.8

Avg. averages all seven metrics in the original table, not the four displayed here. The tabulated improvement over runner-up CaRe is 4.9 points, whereas the nearby prose says “+10 points,” which does not match the table. There is another inconsistency: Table 2 reports Qwen2VL + TARA Chiral/All as 70.1/27.4, while Table 7 gives 72.7/28.3. This note preserves each table's values without combining them or guessing the reason for the discrepancy.

Figure 4 measures the modality gap on 1,000 MSRVTT video-text pairs. Lower values indicate closer overall modality centroids:

Extraction or adaptation configuration	Modality-gap norm	Interpretation
Qwen2VL without EOL	0.39	The generative backbone has a clear offset
Qwen2VL with EOL	0.35	Prompting alone does not remove the offset
Qwen3VL-Embedding	0.23	Multimodal adaptation reference
CaRe	0.29	Multimodal adaptation reference
TARA	0.20	Text-only adaptation also reduces the offset

Key Findings¶

Improvements extend beyond temporally opposite actions. Negated queries and composed edits also benefit, showing that unified embedding adaptation can address different semantic demands. Missing data-composition ablations prevent attribution of an independent contribution to each triplet category.
EOL lowers the gap from 0.39 to 0.35 but leaves a substantial offset; TARA's 0.20 supports the need for training beyond the prompt interface. However, Figure 4 is a representation analysis and does not by itself establish a causal link between gap reduction and all retrieval gains.
Standard MMEB-V2 video tasks improve too: Tarsier 2's classification/retrieval averages rise from 27.4/16.4 to 66.1/45.9, still below Qwen3VL-Embedding's 77.5/58.9. Concatenating the two embeddings yields 78.5/59.5, indicating complementary strengths rather than universal superiority of standalone TARA.

Highlights & Insights¶

Spend the data budget on negative quality. The main contribution is not a new large architecture but control over which query condition a negative violates. For retrieval where the scene remains similar but semantic direction changes, this targets the problem more directly than adding easily distinguished pairs.
Use descriptions to supervise transferable edit semantics. Turning composed video retrieval into composed text retrieval makes failure to apply an edit an explicit error. The construction may transfer to other reference-plus-modification retrieval tasks, but its effectiveness still depends on the backbone's cross-modal capabilities.
Separate shared parameters from actual alignment. Passing video and text through the same LLM does not ensure a common semantic output distribution. The EOL and modality-gap comparisons make this distinction measurable.

Limitations & Future Work¶

Important ablations cannot be verified here. The cache lacks full studies of data composition, scale, and hard-negative quality. It does not establish individual triplet-category contributions or show that 20,000 is an optimal budget.
Text adaptation depends on the visual backbone. Freezing the vision network cannot directly recover action information lost during visual encoding. Uniform 16-frame sampling is also insufficiently tested here for brief events and long videos. These are scope concerns inferred from the design.
The theoretical account is conditional. A constant orthogonal modality offset, the noise distribution, and cross-example consistency are strong assumptions. Closer centroids do not guarantee accurate alignment for every video-caption pair. A stronger follow-up would independently control negative semantics and modality offset.
Comparison and reproduction require precise qualifications. CoVR's zero-shot label needs the task-text exposure caveat, and fixed-backbone results differ from the main table. Adaptation in less than 1 hour does not imply inexpensive online encoding; large-scale retrieval latency and memory comparisons are not reported here.
Author-proposed extensions. Future work could incorporate videos and other modalities during training and use a retrieval-and-reranking system. These are proposed directions, not validated results in this paper.

E5-V / EOL shares the one-word summarization prompt and hidden-state readout approach. TARA emphasizes hard negatives for video, negation, and composed queries, and shows that EOL alone does not remove the video-text modality gap.
CaRe and ArrowRL represent routes using video-text supervision for fine-grained or temporal understanding. TARA performs better in the fixed-Qwen2VL comparison, but objectives and data both differ; the result does not imply that text supervision generally surpasses video supervision.
Qwen3VL-Embedding and VLM2Vec emphasize large-scale universal multimodal embeddings. TARA offers targeted semantic adaptation, while concatenated embeddings on standard tasks suggest complementarity rather than mandatory replacement.
CoVR / CoVR-2 provides the reference-video-plus-edit retrieval setting. This paper constructs a text proxy from training captions and transfers it back to actual videos. The project entry is linked above; the supplied material does not establish a separate code repository URL.

Rating¶

Novelty: 4/5. Embedding extraction follows prior work, but unified adaptation with three types of hard negatives and text-only cross-modal transfer is a useful combination.
Experimental Thoroughness: 4/5. Multiple backbones, nuanced tasks, and standard video benchmarks are covered, but key ablations are absent from the cache and some claims conflict with table values.
Writing Quality: 4/5. The task, data construction, and representation analysis form a clear account, while the zero-shot terminology and fixed-backbone comparisons need more precise qualification.
Value: 4/5. A practical retrieval adaptation approach with low additional data requirements, provided pretrained visual capability and large-model inference costs are not overlooked.