Intrinsic Temporal Adaptation of CLIP for Partially Relevant Video Retrieval¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/hynnsk/ITA
Area: Multimodal VLM
Keywords: partially relevant video retrieval, CLIP temporal adaptation, grouped temporal attention, affinity-weighted gradient propagation, parameter-efficient fine-tuning
TL;DR¶
Addressing the dual dilemmas of isolated frame features and noisy top-1 weak supervision in partially relevant video retrieval (PRVR), this paper proposes Intrinsic Temporal Adaptation (ITA), which integrates grouped temporal self-attention into deep visual transformer layers and distributes contrastive learning gradients across candidate frames via affinity-weighted gradient propagation, achieving state-of-the-art PRVR performance and precise frame-level temporal grounding at low computational cost.
Background & Motivation¶
Partially Relevant Video Retrieval (PRVR) has become an essential capability as real-world untrimmed video databases continue to expand at scale. Unlike standard text-to-video retrieval, where a text query is presumed to summarize the overarching video theme, PRVR reflects realistic query scenarios where the user's textual description pertains only to a short temporal segment or an isolated moment within a lengthy, untrimmed video. Consequently, models cannot rely solely on coarse video-level holistic embeddings; instead, they must possess fine-grained visual-temporal understanding to identify whether and where the supporting evidence resides. State-of-the-art PRVR methods commonly exploit pre-trained vision-language models such as CLIP as visual backbones, extracting frame-level embeddings and subsequently deploying external temporal modulesโsuch as multi-scale similarity networks, Gaussian mixture temporal intervals, or prototype clusteringโto build moment-level representations.
However, existing paradigms suffer from two profound bottlenecks. First, conventional external temporal modules are forced to infer dynamic event semantics from static, isolated frame features extracted independently by frozen CLIP encoders. Even recent efforts that inject learnable temporal prompt pyramids into the backbone (e.g., ProPy) frequently suffer from temporal collapse, exhibiting cross-video static stripe artifacts and position-dependent transition biases that fail to capture video-specific dynamics. Second, PRVR benchmarks provide supervision strictly at the video-text level without segment-level temporal annotations. Prevailing methods routinely measure video similarity by taking the maximum text-frame cosine score (a hard top-1 selection), which channels contrastive gradients predominantly to a single frame during training. This makes the model fragile to background distractors and noisy frames, leading to superficial video-level matches supported by incoherent or incorrect frame-level evidence.
To resolve the disjoint between isolated frame extraction and fragile single-frame weak supervision, the authors explore how to internalize temporal awareness directly inside the feature encoding backbone while softly spreading learning signals across the event moment. The core idea is to introduce Backbone-Internal Temporal Adaptation by allowing the final layers of a frozen CLIP visual encoder to compute grouped self-attention across neighboring frames with relative temporal bias, coupled with Affinity-Weighted Gradient Propagation (AWGP) that softly aggregates top-\(k\) candidate frames according to text affinities to propagate contrastive gradients over the full query-relevant moment.
Method¶
Overall Architecture¶
The ITA framework equips the pre-trained CLIP visual backbone with temporal reasoning capabilities while preserving foundational multi-modal representations and high parameter efficiency. Given an untrimmed video uniformly sampled into \(T\) frames, early visual transformer layers independently encode patch and class tokens to extract basic spatial patterns. In the final visual transformer layers, Backbone-Internal Temporal Adaptation partitions the frames into consecutive temporal groups, allowing all visual tokens within a group to jointly compute self-attention augmented with learnable temporal position embeddings and relative temporal biases. Next, a lightweight global frame attention module processes the resulting frame class tokens to aggregate video-wide macro context. During matching, frame-text cosine affinities are computed, and Affinity-Weighted Gradient Propagation softly selects and weights the top-\(k\) candidate frames to produce a unified video representation for bidirectional InfoNCE contrastive training and online retrieval.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Video Frames & Text Query Input<br/>V sampled to T frames, Q tokenized"] --> B["Backbone-Internal Temporal Adaptation<br/>Grouped Temporal Attention + Relative Temporal Bias"]
B --> C["Global Frame Attention Refinement<br/>Class Token Sequence Context Fusion"]
C --> D["Affinity-Weighted Gradient Propagation<br/>Top-k Soft Aggregation + Bidirectional InfoNCE"]
D --> E["Video Retrieval & Temporal Grounding Output<br/>Efficient Video Ranking + Frame-Level Grounding"]
Key Designs¶
1. Backbone-Internal Temporal Adaptation: Bridging the Temporal Gap Inside the Visual Encoder
Rather than extracting isolated frame features and deferring temporal modeling entirely to external post-processing layers, Backbone-Internal Temporal Adaptation enables visual tokens to interact before the final representations are formed. Recognizing that shallow transformer layers primarily capture local visual textures while deeper layers encode abstract, high-level semantics, the adaptation is applied exclusively to the final visual layers (e.g., group sizes \(g_\ell \in \{2, 4\}\) in the final two layers). For layer \(\ell\), tokens from \(g_\ell\) consecutive frames are concatenated into a joint sequence \(G_{j,m}^\ell = \text{Cat}(F_{j,(m-1)g_\ell+1}^\ell, \dots, F_{j,mg_\ell}^\ell)\) and processed by multi-head self-attention and feed-forward networks before being partitioned back into frame sequences. To inject temporal order awareness, a learnable frame positional embedding \(e_t\) is added to each frame's tokens, and a relative temporal bias is added directly to the attention logits:
where \(\gamma(u)\) and \(\gamma(v)\) denote the frame indices of tokens \(u\) and \(v\) within the group, and \(r_\Delta\) provides a learnable scalar bias for relative temporal offset \(\Delta \in \{-(g_\ell-1), \dots, g_\ell-1\}\). To maintain strict parameter efficiency, the original backbone weights remain frozen, and LoRA rank-8 adapters are inserted solely into the query-key-value projection matrices across attention blocks in both visual and text encoders.
2. Global Frame Attention Refinement: Capturing Long-Range Video Semantics
While grouped temporal attention models dense local motion dynamics with high fidelity, individual groups possess a bounded temporal receptive field. Performing dense patch-level attention across all \(T\) frames would incur prohibitive quadratic computational complexity. To provide overarching temporal context efficiently, ITA appends a single lightweight global frame attention layer on top of the final visual encoder outputs. This layer acts strictly on the sequence of frame-level class-token embeddings (\(T \times d\)), omitting spatial patch tokens. With an input sequence length of only \(T = 32\), this refinement captures long-range video dependencies across the entire sequence at negligible computational overhead, producing final temporally enriched frame representations \(F_j = [f_{j,1}, \dots, f_{j,T}]\).
3. Affinity-Weighted Gradient Propagation: Overcoming Single-Frame Weak Supervision Biases
Under the weak supervision setting of PRVR, standard methods optimize models by selecting only the single frame with the maximum similarity to the text query (\(s_{i,j} = \max_t t_i^\top f_{j,t}\)). However, query-relevant events naturally extend across multiple frames, and the top-1 frame is frequently noisy, unrepresentative, or vulnerable to superficial background correlation. Affinity-Weighted Gradient Propagation (AWGP) resolves this by dynamically aggregating top-\(k\) relevant frames. For query embedding \(t_i\) and frame representations \(f_{j,t}\), frame affinities \(a_{i,j,t} = t_i^\top f_{j,t}\) are calculated to select the top-\(k\) frame indices \(\mathcal{K}_{i,j} = \text{TopK}_t(a_{i,j,t})\). Softmax affinity weights are then computed with temperature \(\tau\):
The final video embedding is obtained via normalized weighted summation \(h_{i,j} = \text{Normalize}\left(\sum_{t \in \mathcal{K}_{i,j}} w_{i,j,t} f_{j,t}\right)\), yielding text-video matching score \(s_{i,j} = t_i^\top h_{i,j}\). Even though \(k\) is fixed, the effective supervision remains query-adaptive: candidates with weak relative affinity receive near-zero weights, while multiple genuinely relevant frames simultaneously receive contrastive gradient signals, propagating temporal evidence across the event duration.
Loss & Training¶
The entire framework is trained end-to-end via a symmetric bidirectional InfoNCE contrastive loss over mini-batches of size \(B\):
For implementation details, all videos are uniformly sampled at \(T = 32\) frames. Visual backbone grouped temporal attention employs group sizes \(\{2, 4\}\) in the final two layers. LoRA rank is set to \(r = 8\) for all QKV projection layers. AWGP parameters are set to \(k = 4\) and temperature \(\tau = 0.05\) across all datasets. Models are optimized for 10 epochs using AdamW on a single NVIDIA RTX A6000 GPU (batch size 48). Learning rates are \(2 \times 10^{-4}\) for ActivityNet Captions and Charades-STA, \(1 \times 10^{-3}\) for TVR, and \(1 \times 10^{-4}\) for QVHighlights, with training completing in approximately 0.5 to 2.0 hours per dataset.
Key Experimental Results¶
Main Results¶
Evaluations conducted across four standard PRVR benchmarks using the standard CLIP ViT-B/32 backbone (32 frames per video) demonstrate that ITA consistently establishes new state-of-the-art retrieval performance across all metrics:
| Dataset | Metric | Ours (ITA) | Prev. SOTA (ProPy) | Gain |
|---|---|---|---|---|
| TVR | R@1 / R@5 / SumR | 25.3 / 49.2 / 224.7 | 22.4 / 45.0 / 212.8 | +2.9 / +4.2 / +11.9 |
| ActivityNet Captions | R@1 / R@5 / SumR | 19.2 / 41.7 / 201.4 | 14.9 / 34.9 / 180.0 | +4.3 / +6.8 / +21.4 |
| Charades-STA | R@1 / R@5 / SumR | 3.5 / 10.6 / 87.4 | 2.9 / 9.8 / 82.5 | +0.6 / +0.8 / +4.9 |
| QVHighlights | R@1 / R@5 / SumR | 39.8 / 67.2 / 281.9 | 37.4 / 65.6 / 275.5 | +2.4 / +1.6 / +6.4 |
On the diverse and challenging ActivityNet Captions benchmark, ITA achieves a remarkable performance leap, outperforming ProPy by +21.4 SumR points (+4.3% R@1, +6.8% R@5). Consistent gains are likewise observed on TVR (+11.9 SumR), Charades-STA (+4.9 SumR), and QVHighlights (+6.4 SumR), proving that internal temporal adaptation significantly elevates partial moment retrieval across varied video domains.
Ablation Study¶
A component-wise ablation study systematically evaluates the individual contribution of Parameter-Efficient Adaptation (PEA), Grouped Temporal Attention (GTA), Temporal Bias (TB), and Affinity-Weighted Gradient Propagation (AWGP):
| Config | TVR | ActivityNet Captions | Charades-STA | QVHighlights | Average SumR | Note |
|---|---|---|---|---|---|---|
| PEA Only | 217.7 | 195.9 | 78.5 | 271.2 | 190.8 | Baseline frozen CLIP + LoRA |
| PEA + GTA | 221.1 | 199.0 | 82.0 | 277.6 | 194.9 | Adds internal frame grouping (+4.1) |
| PEA + GTA + TB | 223.2 | 199.6 | 83.9 | 277.9 | 196.2 | Adds temporal position/relative bias (+1.3) |
| PEA + AWGP | 219.6 | 199.5 | 81.7 | 280.8 | 195.4 | Adds soft top-k training without GTA (+4.6) |
| Full Model (ITA) | 224.7 | 201.4 | 87.4 | 281.9 | 198.9 | Complete synergistic framework (+8.1) |
Key quantitative diagnostic analyses further highlight the robustness and efficiency of ITA: - Ground-Truth Moment Grounding Accuracy (Frame-level R@1): When evaluating whether retrieved frames fall within ground-truth query moments, ITA outperforms ProPy across TVR (11.90% vs. 6.95%), ActivityNet (9.86% vs. 5.61%), and QVHighlights (32.52% vs. 23.23%), demonstrating that video retrieval is supported by authentic temporal evidence rather than spurious background correlation. - Cross-Dataset Zero-Shot Transfer: When trained exclusively on QVHighlights and evaluated on unseen datasets without fine-tuning, ITA achieves SumR scores of 120.3 on TVR and 173.5 on ActivityNet (surpassing ProPy's 102.5 and 150.3), confirming that internal temporal modeling avoids overfitting to specific dataset distributions. - Efficiency and Resource Profile: ITA requires only 1.57M trainable parameters (compared to ProPy's 7.98M and GMMFormer-V2's 30.11M). Offline video encoding cost is reduced by 23.3% (283.7 vs. 370.0 GFLOPs), while online retrieval over a 10,000-video gallery requires 10.1% fewer operations (5.490 vs. 6.107 GFLOPs).
Key Findings¶
- Synergy Between Representation and Supervision: Injecting internal temporal interactions (GTA) and regularizing weak supervision gradients (AWGP) independently yield over +4.0 points in average SumR. Combining them produces compound improvements (+8.1 points), demonstrating that temporally grounded representations require multi-frame supervision to realize their full potential.
- Optimal Temporal Receptive Field: Expanding group sizes to full-sequence attention or adapting early layers degrades retrieval accuracy and inflates GFLOPs, confirming that localized temporal grouping in deep semantic layers preserves frame-specific visual discriminability while capturing essential dynamic transitions.
- Robustness of Top-\(k\) over Absolute Thresholds: Experiments contrasting fixed top-\(k\) against hard cosine similarity thresholds show that global similarity thresholds fail to transfer across queries and datasets due to uncalibrated margin variations. Top-\(k\) selection combined with temperature scaling dynamically adapts to arbitrary affinity distributions.
Highlights & Insights¶
- Internalizing Temporal Interactions inside Frozen Foundation Models: Instead of treating vision backbones as static feature extractors and stacking heavyweight temporal post-processors, ITA demonstrates that localized self-attention in deep transformer layers provides a much stronger, video-dependent temporal representation.
- Soft Multi-Frame Contrastive Backpropagation: AWGP replaces rigid top-1 frame selection with a differentiable, affinity-weighted top-\(k\) pool, allowing learning signals to diffuse across the true span of an event while suppressing irrelevant frames via temperature-controlled exponential decay.
- Exceptional Parameter and Computational Efficiency: With only 1.57M tunable parameters and a 23.3% reduction in offline GFLOPs compared to state-of-the-art prompt tuning, ITA proves that effective temporal adaptation does not require bulky prompt pyramids or complex parametric density estimators.
Limitations & Future Work¶
- Inherent Foundation Model Biases: Built upon pre-trained CLIP, the framework may inherit representation biases towards static visual attributes. Under weak supervision, the model might occasionally favor frames that align strongly with CLIP's pre-training distribution over temporally nuanced evidence.
- Uniform Frame Sampling on Extremely Long Videos: Operating with a fixed budget of 32 uniformly sampled frames may miss very short or subtle action events in videos spanning dozens of minutes or hours.
- Future Directions: Integrating adaptive or hierarchical frame sampling strategies, and exploring the transferability of internal grouped temporal adaptation to modern Video-LLM backbones for unified grounding and dense captioning.
Related Work & Insights¶
- vs ProPy (Pan et al., 2025): ProPy injects interactive prompt pyramids into CLIP but induces video-agnostic static striping artifacts and incurs high encoding complexity (370.0 GFLOPs); ITA enables direct grouped token self-attention and relative temporal bias inside the backbone, saving 23.3% offline FLOPs while generating genuine content-dependent temporal transitions.
- vs GMMFormer / GMMFormer-V2 (Wang et al., 2024a,b): GMMFormer uses parametric Gaussian mixture networks to approximate event boundaries, requiring over 30M trainable parameters and complex anchor tuning; ITA replaces heavy parametric temporal modules with clean internal grouped attention and AWGP, cutting trainable parameters by 95% while achieving substantial retrieval accuracy gains.
Rating¶
- Novelty: โญโญโญโญโ Elegantly internalizes grouped temporal interaction into deep CLIP layers and formulates affinity-weighted multi-frame gradient propagation under weak supervision.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive benchmarking across four PRVR datasets, extensive component ablations, strict frame-level ground-truth moment evaluations, cross-dataset transfer tests, and computational efficiency profiling.
- Writing Quality: โญโญโญโญโญ Clear, logical narrative supported by compelling diagnostic visualizations (frame-transition analysis and top-\(k\) retrieval degradation curves).
- Value: โญโญโญโญโญ Delivers an efficient, robust, and easily adoptable paradigm for parameter-efficient video temporal adaptation that provides strong utility for large-scale video retrieval systems.