MoVA: Modular Long Video-Text Alignment via Asymmetric Dual Projection

Conference: ECCV 2026
arXiv: 2607.00858
Code: None
Area: Multimodal VLM / Video Understanding
Keywords: Video-Text Alignment, Contrastive Learning, Temporal Misalignment, Semantic Asymmetry, Modular Representation

TL;DR

MoVA models the video-text alignment problem as a latent variable identification problem, proposing an asymmetric dual projection architecture—a temporal mask network on the text side to filter relevant textual concepts frame-by-frame, and a concept mask network on the video side to isolate text-related visual factors—paired with a modular contrastive learning objective to achieve a structured decoupled alignment of long videos and long texts.

Background & Motivation

Contrastive pre-training (especially the CLIP paradigm) has significantly advanced vision-language representation learning, achieving remarkable success in tasks such as video retrieval, video captioning, and action recognition. However, when generalizing from image-text to video-text, CLIP faces two fundamental new challenges. The first is temporal misalignment: textual descriptions of videos often relate to specific temporal windows, and different frames in a long video correspond to completely different clauses in the text—for instance, one frame describes "a person talking" while the next may describe "a table tennis match", while a large number of transitional frames in between are virtually irrelevant to the text. The second is semantic asymmetry: the association between texts and frames is bidirectional but unequal—the text only activates a small portion of visual concepts in a frame (e.g., "archery" focuses on the bow and arrow, ignoring the background trees), while each frame contains far richer visual information than described in the text, forming a sparse, bidirectional asymmetric relationship. Existing methods either use short captions, which leads to ambiguity, or use long captions, which наоборот exacerbates the entanglement between static objects and temporal evolution.

The core limitation of this dilemma lies in: CLIP-style models inherently learn global, image-level alignment—one image paired with one text segment, where concepts are one-off. When extended to videos, multi-frame sequences introduce dynamic concept evolution along the temporal dimension, whereas CLIP-style global contrastive losses fail to distinguish "which text words correspond to which frames" and "which visual concepts are frame-specific versus globally shared." Simply applying SmartCLIP (image-level decoupled masking) to videos is ineffective because it lacks both the active selection of frames by text and the backward constraint on text subsets by frames. MoVA's key insight is: since video-text alignment is naturally a dual-perspective latent variable identification problem—where text descriptions encode abstract semantics across frames, and each frame provides a local view of these semantics—introducing a pair of asymmetric projection masks to control the sparse correspondence from text-to-frame and frame-to-text can achieve frame-level concept decoupling while maintaining global semantics. Core Idea: Formalize video-text alignment as a latent variable identification problem, prove that the frame-text correspondence can be recovered under a sparse dual-mask constraint (block-level identifiability), and accordingly design a dual-path asymmetric projection network—TMN to screen text-related subspaces frame-by-frame and CMN to isolate visual concepts frame-by-frame—cooperating with modular contrastive learning to achieve both global semantic preservation and frame-level concept decoupling.

Method

The mechanism of MoVA is to insert a pair of asymmetric masking projection modules on top of the dual-tower CLIP encoders: the Temporal Mask Network (TMN) manages the "text-to-frame" subspace projection, and the Concept Mask Network (CMN) handles the "frame-to-text" visual concept filtering. These two projections share a dual-mask constraint (dual-mask constraint); that is, the text-side subspace mask activated for each frame should align in latent semantics with the visual-side subspace mask activated for that frame, ensuring both temporal consistency of global semantics and concept decoupling.

Taking long video-long caption scenarios like VideoUFO or UltraVideo as examples: a 155-word video description contains multiple sequential events (e.g., "a black labrador sits on a wooden boat; it picks up a metal cup and drinks; it turns its head to scan the surroundings"). MoVA decides which event clause in the text each frame should focus on via frame-by-frame analysis, isolates the visual factors related to that event (e.g., the dog's head pose, the cup) from the frame, ignores irrelevant visual information (e.g., water ripples), and finally contrasts only the aligned "text subset to visual subset" at each time step. This avoids the issue of "irrelevant frames polluting relevant concepts" in global contrast.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video Frame Sequence X₁…X_T"] --> B["Frame Encoder\nCLIP ViT"]
    C["Text Description"] --> D["Text Encoder\nCLIP Text"]
    B --> E["Frame-level Visual Embedding ẑ^V_t"]
    D --> F["Global Text Embedding ẑ^T"]

    F --> G["Temporal Mask Network\nTMN: 2-layer Transformer"]
    E --> G
    G --> H["Frame-level Text Mask m̂^T_t"]
    H --> I["Frame-aligned Text Embedding\nẑ^T ⊙ m̂^T_t"]

    I --> J["Concept Mask Network\nCMN: 1-layer Transformer+Pooling"]
    E --> J
    J --> K["Visual Concept Mask m̂^V_t"]
    K --> L["Text-aligned Visual Embedding\nẑ^V_t ⊙ m̂^V_t"]

    I --> M["Frame-level Text→Video Contrast\nℒ_align^(t→v)"]
    E --> M

    I --> N["Same-Frame Different-Mask Contrast ℒ_sfdm"]
    L --> N
    N --> O["Modular Video→Text Contrast\nℒ_align^(v→t)"]
    I --> P["Different-Frame Same-Mask Contrast ℒ_dfsm"]
    L --> P
    P --> O

    E --> Q["Global Retrieval Loss ℒ_g\n(CLIP4Clip-style)"]
    F --> Q

Key Designs

1. Temporal Mask Network: Text-to-Frame Temporal Subspace Projection

The core problem TMN aims to solve is "which words in a global text description should the current frame focus on." It consists of a two-layer Transformer Block, which uses the current frame's visual embedding as the query, allowing all tokens in the text to fully interact with the frame embedding and with each other. Finally, it outputs a binarized mask vector \(\hat{\mathbf{m}}_t^{\mathrm{T}} \in \{0,1\}^{d(\mathbf{z})}\) via a Straight-Through Estimator, marking the text dimensions activated by the current frame. The key constraint is that \(\hat{\mathbf{m}}_t^{\mathrm{T}}\) must be sufficiently sparse (only the parts of the text semantically relevant to the frame are activated), while the cosine similarity between \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) and \(\hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) must be greater than that between the frame and the masked text of other frames. To this end, MoVA proposes a frame-level contrastive loss \(\ell_{\text{ctrf}}(t)\): the similarity of positive pairs is pushed towards 1, while negative pairs (masked text of other frames in the same video, masked text of frames in other videos) are pushed away by a margin \(\Delta\). The sequence modeling capability of TMN ensures smooth transitions along the temporal axis—when the video switches from "archery" to "target collapsing", the text subset activated by the mask shifts naturally. From an identifiability theory perspective, TMN's implementation is equivalent to estimating the text-side mask \(\mathbf{m}_t^{\mathrm{T}}\) in Theorem 4.2—under the optimal solution, it can recover the true support set of the frame-level text subset (as guaranteed by the "no redundant, no missing" property in Lem. 2).

2. Concept Mask Network: Vision-to-Text Concept Isolation

CMN is the symmetric, reverse projection of TMN, solving: "given the current frame and the text subset selected by TMN, which visual factors in the frame actually correspond to this text." It consists of a single-layer Transformer followed by an Attention Pooling layer. It takes the masked text embedding \(\hat{\mathbf{s}}^{\mathrm{T}} = \hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) output by TMN and the frame embedding \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) as inputs, and outputs the mask \(\hat{\mathbf{m}}_t^{\mathrm{V}}\). Its core constraint comes from the sparse dual-mask consistency shared with TMN: \(\hat{\mathbf{z}}_t^{\mathrm{V}} \odot \hat{\mathbf{m}}_t^{\mathrm{V}} \approx \hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\). The ingenuity of the CMN design lies in its introduction of Attention Pooling to adaptively downsample and match the CLIP representation dimensions; concepts of different granularities (object-level vs. motion-level) naturally fall into different dimensional subspaces. Ablation experiments (Table 4) show that removing CMN (using only the unidirectional visual-to-text mask of SmartCLIP) is the worst-performing variant—ActivityNet R@1 plunges from 47.8 to 35.6. This directly validates the core claim that "semantic asymmetry must be modeled bidirectionally": text selection over frames alone is insufficient; the reverse selection of which visual factors in a frame are activated by the text is equally critical.

3. Dual-Stream Modular Contrastive Loss: Same-Frame Different-Mask and Different-Frame Same-Mask

With the dual-path projected embeddings \(\mathbf{z}_t^{\mathrm{V}} \odot \hat{\mathbf{m}}_t^{\mathrm{V}}\) and \(\mathbf{z}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) generated by TMN and CMN, MoVA designs two complementary contrastive learning objectives. The first is Same-Frame Different-Mask (sfdm): fixing a frame \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) and contrasting masks \(\hat{\mathbf{m}}_{i,t}^{\mathrm{V}}\) generated from different descriptions—this forces the model to understand that "the same frame should emphasize different visual content under different textual contexts" (for example, in a forest scene, the descriptions "bear" and "river" should activate different sub-regions in the frame). The second is Different-Frame Same-Mask (dfsm): fixing a text subset \(\hat{\mathbf{s}}^{\mathrm{T}}\) and contrasting embeddings of different frames under the same mask—this forces the model to identify invariant concepts shared across frames from the temporal sequence (such as recurring characters or actions), thereby achieving the goal of recovering atomic concepts through mask intersection operations as described in theory. Removing dfsm causes a drop of 3.6 points in ActivityNet R@1, indicating that cross-frame constancy modeling is indispensable for overall alignment.

Loss & Training

The complete training objective of MoVA is a weighted sum of four terms:

\[\mathcal{L} = \lambda_{\mathrm{g}}\mathcal{L}_{\mathrm{g}} + \lambda_{\mathrm{tv}}\mathcal{L}_{\mathrm{align}}^{\mathrm{t}\to\mathrm{v}} + \lambda_{\mathrm{vt}}\mathcal{L}_{\mathrm{align}}^{\mathrm{v}\to\mathrm{t}} + \lambda_{\mathrm{s}}\mathcal{L}_{\mathrm{s}}\]

where \(\mathcal{L}_{\mathrm{g}}\) is the CLIP4Clip-style global video-text retrieval loss (symmetric cross-entropy on the global similarity matrix); \(\mathcal{L}_{\mathrm{align}}^{\mathrm{t}\to\mathrm{v}}\) is the frame-level contrastive loss on the TMN side; \(\mathcal{L}_{\mathrm{align}}^{\mathrm{v}\to\mathrm{t}}\) is the modular contrastive loss on the CMN side (sfdm+dfsm); and \(\mathcal{L}_{\mathrm{s}}\) is the \(L_0\) sparsity constraint on \(\hat{\mathbf{m}}^{\mathrm{T}}\) and \(\hat{\mathbf{m}}^{\mathrm{V}}\).

Key training details: The model is first warm-up initialized on ShareGPT4v image-caption data (transferring CLIP's image-level sparse mapping knowledge to video), and then the encoders are fine-tuned with a learning rate of \(10^{-7}\) while the new modules are trained with a learning rate of \(10^{-4}\) with a batch size of 256 using a ViT-B/16 backbone, supporting up to 248 tokens (breaking CLIP's original 77-token limit). Ablation experiments (Figure 7d) reveal that the optimal ratio for \(\lambda_{\mathrm{g}}/\lambda_{\mathrm{tv}}\) is around 1.0—a ratio too small (<0.2) loses global information, whereas a ratio too large (>1.2) causes the frame-level mask to collapse.

Key Experimental Results

Main Results

Dataset	Direction	Ours R@1	Prev. SOTA R@1	Gain
ActivityNet	T→V	47.8	46.9 (VideoCLIP-XL)	+0.9
ActivityNet	V→T	46.7	42.9 (CLIP4Clip)	+3.8
MSVD	T→V	52.6	50.4 (X-CLIP)	+2.2
DiDeMo	T→V	57.5	50.8 (InternVideo)	+6.7
VideoUFO	T→V	62.4	57.4 (VideoCLIP-XL)	+5.0
UltraVideo	T→V	58.5	51.8 (ProST)	+6.7

Ablation Study

Configuration	T→V R@1 (ActivityNet)	V→T R@1 (ActivityNet)	Description
Full MoVA (iv)	47.8	46.7	Full dual asymmetric projection
Global loss only (i)	43.7	42.5	Removes all modular objectives, degenerating to fine-tuned CLIP
CMN unidirectional only (ii)	35.6	33.7	Removes TMN, SmartCLIP image-level unidirectional mask
TMN + original token mask (iii)	42.0	41.9	TMN retained but temporally masking directly on original tokens
w/o different-frame same-mask	~44.2	—	Significant performance drop after removing ℒ_dfsm

Key Findings

• Both Paths Are Indispensable: Ablation (ii) shows that having only the frame-to-text mask (removing the text-to-frame direction of TMN) is the worst configuration, with R@1 plunging by over 12 points, directly proving that semantic asymmetry must be modeled bidirectionally.
• Respective Contributions of Modular Objectives: Both ℒ_sfdm and ℒ_dfsm are indispensable. Removing ℒ_dfsm alone leads to a 3.6 drop in R@1, indicating that cross-frame constancy (different-frame same-mask) is a key link in decoupling.
• TMN Depth Insensitivity: A 1-2 layer Transformer is sufficient (Table 8); deeper layers (3-8 layers) yield no extra gain, as frame-level text subspace selection itself is not a computationally intensive task.
• Robustness to Frame Count: Performance remains stable within the range of 32 to 160 frames (Table 7), peaking around 96 frames, showing that the model does not rely on brute-force frame count increases to cover concepts.
• Parameter Efficiency: MoVA has only 174.7M parameters (+7.6% compared to CLIP4Clip's 162.3M), which is much smaller than VideoCLIP-XL (427.9M), and its training time per epoch is actually faster than CLIP4Clip (76.7 min vs 92.9 min on 8×MI210).

Highlights & Insights

• Counter-Ablation Validation of Asymmetric Dual-Projection: Most multimodal methods only construct a single-directional attention or mask. MoVA proves through rigorous ablation that both directions (text-to-frame subspace selection and frame-to-text visual factor selection) contribute independently and are indispensable. This "asymmetric-symmetric" design intuition is rare in CLIP-style works.
• Closed Loop from Latent Variable Identifiability Theory to Algorithm Design: The paper first establishes a data generation model (Section 3), proves block-level identifiability (Theorem 4.2), and then directly translates the identification conditions into the TMN/CMN architecture and sparse contrastive objectives. This rigorous dialog of "theoretical derivation to algorithmic implementation" serves as a model for contrastive learning research.
• Transfer Benefits in Long Text-to-Video Generation: When MoVA's text encoder zero-shot replaces the CLIP text encoder in VideoCrafter2, it not only retains basic objects but also generates extremely fine textural descriptions like "splashed rust-orange paint stains" and complex temporal motions like "rhythmic swaying," vividly demonstrating the transfer value of decoupled multimodal representations beyond retrieval.

Limitations & Future Work

• Dependency on Image-Level Warm-Up: MoVA needs a warm-up initialization on ShareGPT4v (starting from SmartCLIP weights). If the target domain differs significantly from natural images (e.g., medical videos, remote sensing videos), the warm-up transfer effect may be suboptimal.
• Scalability of Frame Sampling in Ultra-Long Videos: The current method is stable for around 64 frames, but for videos spanning thousands of frames (e.g., movies, long surveillance footage), linear sampling of a fixed frame count may miss key events. The paper does not discuss adaptive frame selection or hierarchical sampling strategies.
• Information Loss from Hard Masking: The straight-through estimator (STE) hard mask aids interpretability (Figure 4 in the paper demonstrates clear text subset activations), but a soft mask version is close in performance. For scenarios requiring continuous semantic transitions (e.g., blurry event boundaries), hard binarization might lose intermediate semantics.
• Gap Between Theoretical Assumptions and Reality: The identifiability theorem (Theorem 4.2) relies on assumptions like smooth invertibility of the generator and view diversity. In real-world data, these assumptions are only approximately met, and the strictness of the theoretical guarantee requires further ablation (e.g., validating failure modes on synthetic data where assumptions do not hold).

Related Work & Insights

• vs SmartCLIP [Xie2025, ICCV 2025]: SmartCLIP proposes adaptive masking and modular contrast at the image level. MoVA extends this logic to video, with the core difference being the introduction of TMN to achieve text-to-frame temporal masking and dual-mask constraints. On a theoretical level, MoVA unifies image and video alignment under a latent variable identification framework, proving identifiability in the temporal dimension.
• vs CLIP4Clip [Luo2022, CVPR 2022]: CLIP4Clip is a milestone in adapting CLIP to videos, focusing on similarity computation and late fusion. MoVA inherits its global retrieval loss as a backbone, but replaces simple global contrast with modular alignment objectives, achieving frame-level decoupled alignment without sacrificing global semantics.
• vs DGL [Yang2024]: DGL generates local dynamic prompts using shared latent spaces coupled with global-local attention. MoVA differs by being guided by identifiability theory, featuring a unique asymmetric dual-mask design—whereas DGL's local prompts are implicit, MoVA's masks are explicitly sparse and interpretable (Figures 4 and 6 in the paper).
• vs VideoCLIP-XL [Wang2024]: VideoCLIP-XL uses a larger ViT-L/14 and additional post-video pre-training data (YT-VidLA-800M). MoVA achieves superior results across multiple datasets using a ViT-B/16 with significantly fewer parameters (174.7M vs 427.9M), demonstrating that systematic modular alignment designs can replace brute-force model scaling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces latent variable identifiability theory to video-text contrastive learning, proposing an asymmetric dual-projection architecture with a complete theory-algorithm loop.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 6 datasets (short and long videos), two applications (retrieval and generation), and ablation studies spanning all modules, frame counts, TMN depth, loss weights, and soft/hard masking.
• Writing Quality: ⭐⭐⭐⭐ Rigorously derived theory and clearly described method, though readability in symbol-heavy sections (Sections 3-4) might be slightly challenging for non-theoretical readers.
• Value: ⭐⭐⭐⭐⭐ Addresses the core bottleneck of extending CLIP from images to videos, offering a new paradigm for long video-text alignment, with direct plug-and-play capability to replace existing CLIP text encoders.

Related Papers

• [ECCV 2026] Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval
• [ECCV 2026] AdaBoosting Text Prompts for Vision-Language Models
• [ECCV 2026] UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation
• [ECCV 2026] Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models
• [ECCV 2026] Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models