MoVA: Asymmetric Dual Projection for Modular Long Video-Text Alignment

Conference: ECCV 2026
arXiv: 2607.00858
Code: None
Area: Multimodal VLM / Video Understanding
Keywords: Video-Text Alignment, Contrastive Learning, Temporal Misalignment, Semantic Asymmetries, Modular Representation

TL;DR

MoVA models the video-text alignment problem as a latent variable identification problem. It proposes an asymmetric dual projection architecture, where a Temporal Mask Network on the text side dynamically filters relevant textual concepts frame-by-frame, and a Concept Mask Network on the video side isolates text-related visual factors. Combined with modular contrastive learning objectives, MoVA achieves structured, decoupled alignment of long videos and long texts.

Background & Motivation

Contrastive pre-training, particularly the CLIP paradigm, has significantly advanced vision-language representation learning, achieving remarkable success in tasks such as video retrieval, captioning, and action recognition. However, when extending from image-text to video-text, CLIP faces two fundamental new challenges. First is temporal misalignment: the textual description of a video is often only relevant to specific time windows, with different frames in a long video corresponding to entirely different clauses in the text—for example, one frame describes "a person speaking" while a subsequent one depicts a "ping-pong match," with a massive amount of transitional frames in between being completely unrelated. Second is semantic asymmetry: the association between texts and frames is bidirectional yet non-equivalent—text only activates a small portion of visual concepts within a frame (e.g., "archery" focuses on the bow and arrow, ignoring background trees), while each frame contains far richer visual information than described by the text, resulting in a sparse, bidirectional asymmetric relationship. Existing methods either use short captions that lead to ambiguity or adopt long captions that instead exacerbate the entanglement between static objects and temporal evolution.

The Key Challenge of this dilemma lies in: CLIP-style models inherently learn image-level global alignment—one image paired with one text segment, where concepts are one-offs. Translating this to videos introduces dynamic conceptual evolution along the temporal dimension via multi-frame sequences, whereas CLIP-style global contrastive loss cannot distinguish "which text words correspond to which frames" and "which visual concepts are frame-specific versus globally shared." Simply applying SmartCLIP (image-level decoupled masking) to videos fails due to the lack of active selection from text to frames and feedback constraints from frames to text subsets. The Key Insight of MoVA is that since video-text alignment is naturally a dual-perspective latent variable identification problem—where the textual description encodes abstract semantics across frames, and each frame provides a local view of these semantics—introducing a pair of asymmetric projection masks to control the sparse correspondence of text-to-frame and frame-to-text can achieve frame-level concept decoupling while maintaining global semantics. Core Idea: Formulate video-text alignment as a latent variable identification problem, prove that frame-to-text correspondence can be recovered under sparse dual-mask constraints (block-level identifiability), and accordingly design a dual-path asymmetric projection network—Temporal Mask Network (TMN) for frame-by-frame text subspace filtering and Concept Mask Network (CMN) for frame-by-frame visual concept isolation—cooperating with modular contrastive learning to achieve global semantic preservation and frame-level concept decoupling.

Method

The Mechanism of MoVA is to insert a pair of asymmetric mask projection modules atop the dual-tower CLIP encoders: the Temporal Mask Network (TMN) is responsible for the "text-to-frame" subspace projection, while the Concept Mask Network (CMN) is responsible for "frame-to-text" visual concept filtering. These two projections share a dual-mask constraint, where the text subspace activated by the text-side mask of each frame should align in latent semantics with the visual subspace activated by the vision-side mask, thereby ensuring global semantic consistency and concept decoupling across temporal evolution.

Taking long-video long-description scenarios in VideoUFO or UltraVideo as examples: a video description of up to 155 words contains multiple temporal events (e.g., "a black Labrador sits on a wooden boat; it picks up a metal cup to drink water; it turns its head to look around"). MoVA analyzes frame by frame to decide which event clause in the text the current frame should prioritize, isolates the visual factors related to that event (e.g., the dog's head pose, the cup) while ignoring irrelevant visual information (e.g., water ripples), and ultimately contrasts only the aligned "text subset to visual subset" at each timestep, avoiding the issue of "irrelevant frames polluting relevant concepts" in global contrast.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Video Frame Sequence X₁…X_T"] --> B["Frame Encoder\nCLIP ViT"]
    C["Text Description"] --> D["Text Encoder\nCLIP Text"]
    B --> E["Frame-wise Visual Embedding ẑ^V_t"]
    D --> F["Global Text Embedding ẑ^T"]

    F --> G["Temporal Mask Network\nTMN: Two-layer Transformer"]
    E --> G
    G --> H["Frame-level Text Mask m̂^T_t"]
    H --> I["Frame-aligned Text Embedding\nẑ^T ⊙ m̂^T_t"]

    I --> J["Concept Mask Network\nCMN: One-layer Transformer+Pooling"]
    E --> J
    J --> K["Visual Concept Mask m̂^V_t"]
    K --> L["Text-aligned Visual Embedding\nẑ^V_t ⊙ m̂^V_t"]

    I --> M["Frame-level Text-to-Video Contrast\nℒ_align^(t→v)"]
    E --> M

    I --> N["Same-Frame Different-Mask Contrast ℒ_sfdm"]
    L --> N
    N --> O["Modular Video-to-Text Contrast\nℒ_align^(v→t)"]
    I --> P["Different-Frame Same-Mask Contrast ℒ_dfsm"]
    L --> P
    P --> O

    E --> Q["Global Retrieval Loss ℒ_g\n(CLIP4Clip-style)"]
    F --> Q

Key Designs

1. Temporal Mask Network: Text-to-Frame Temporal Subspace Projection

The core problem TMN addresses is "which words in a global text description the current frame should focus on." Consisting of a two-layer Transformer Block, it uses the current frame's visual embedding as the query, allowing all tokens in the text to fully interact with the frame embedding and with one another. It eventually outputs a binarized mask vector \(\hat{\mathbf{m}}_t^{\mathrm{T}} \in \{0,1\}^{d(\mathbf{z})}\) via a Straight-Through Estimator, marking the text dimensions activated by the current frame. The critical constraint is that \(\hat{\mathbf{m}}_t^{\mathrm{T}}\) must be sufficiently sparse (only textual components semantically relevant to the frame are activated), and the cosine similarity between \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) and \(\hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) must be greater than the similarity between this frame and the masked text of other frames. To this end, MoVA proposes a frame-level contrastive loss \(\ell_{\text{ctrf}}(t)\): pushing the similarity of positive pairs towards 1 while keeping negative pairs (masked texts of other frames in the same video or other video frames) apart by a margin \(\Delta\). The sequence modeling nature of TMN ensures smooth transitions over the timeline—when a video switches from "archery" to "the target collapsing," the text subset activated by the mask shifts naturally. From the perspective of identification theory, the implementation of TMN is equivalent to estimating the text-side mask \(\mathbf{m}_t^{\mathrm{T}}\) in Theorem 4.2—under optimal conditions, it recovers the true support set of the frame-level text subset (with the "no redundancy, no omission" guarantee of Lem. 2).

2. Concept Mask Network: Vision-to-Text Concept Isolation

CMN is a reverse projection symmetric to TMN, solving the problem: "Given the current frame and the text subset selected by TMN, which visual factors within the frame truly correspond to this text." It comprises a single-layer Transformer plus an Attention Pooling layer, taking the masked text embeddings \(\hat{\mathbf{s}}^{\mathrm{T}} = \hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) output by TMN and the frame embedding \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) as inputs, and outputting the mask \(\hat{\mathbf{m}}_t^{\mathrm{V}}\). Its core constraint stems from the sparse dual-mask consistency shared with TMN: \(\hat{\mathbf{z}}_t^{\mathrm{V}} \odot \hat{\mathbf{m}}_t^{\mathrm{V}} \approx \hat{\mathbf{z}}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\). The elegance of CMN lies in its introduction of Attention Pooling to adaptively downsample the embeddings to match the CLIP representation dimensions, allowing concepts of varying granularities (object-level vs. motion-level) to naturally fall into different dimensional subspaces. Ablation studies (Table 4) show that removing CMN (using only SmartCLIP's unidirectional vision-to-text mask) is the poorest variant—ActivityNet R@1 plummets from 47.8 to 35.6, directly verifying the core assertion that "semantic asymmetry must be modeled bidirectionally": selecting text on the frame side is insufficient, and checking which visual factors are activated by the text in reverse is equally critical.

3. Dual-Stream Modular Contrastive Loss: Same-Frame Different-Mask and Different-Frame Same-Mask

Utilizing the dual-path projection embeddings \(\mathbf{z}_t^{\mathrm{V}} \odot \hat{\mathbf{m}}_t^{\mathrm{V}}\) and \(\mathbf{z}^{\mathrm{T}} \odot \hat{\mathbf{m}}_t^{\mathrm{T}}\) generated by TMN and CMN, MoVA designs two complementary contrastive learning objectives. The first is Same-Frame Different-Mask (sfdm): keeping a frame \(\hat{\mathbf{z}}_t^{\mathrm{V}}\) fixed and contrasting visual masks \(\hat{\mathbf{m}}_{i,t}^{\mathrm{V}}\) generated from different descriptions. This forces the model to learn that "the same frame should emphasize different visual content under different textual contexts" (e.g., in a forest scene, descriptions of "a bear" vs. "a river" should activate different sub-regions in the frame). The second is Different-Frame Same-Mask (dfsm): keeping a text subset \(\hat{\mathbf{s}}^{\mathrm{T}}\) fixed and contrasting embeddings of different frames under the same mask. This forces the model to identify persistent concepts shared across frames in a temporal sequence (e.g., recurring characters or actions), thereby achieving the theoretical goal of recovering atomic concepts via shift operations. Removing dfsm causes ActivityNet R@1 to drop by 3.6 points, showing that modeling cross-frame constancy is indispensable for overall alignment.

Loss & Training

The overall training objective of MoVA is formulated as a weighted sum of four terms:

\[\mathcal{L} = \lambda_{\mathrm{g}}\mathcal{L}_{\mathrm{g}} + \lambda_{\mathrm{tv}}\mathcal{L}_{\mathrm{align}}^{\mathrm{t}\to\mathrm{v}} + \lambda_{\mathrm{vt}}\mathcal{L}_{\mathrm{align}}^{\mathrm{v}\to\mathrm{t}} + \lambda_{\mathrm{s}}\mathcal{L}_{\mathrm{s}}\]

where \(\mathcal{L}_{\mathrm{g}}\) is the CLIP4Clip-style global video-text retrieval loss (symmetric cross-entropy on the global similarity matrix); \(\mathcal{L}_{\mathrm{align}}^{\mathrm{t}\to\mathrm{v}}\) is the frame-level contrastive loss on the TMN side; \(\mathcal{L}_{\mathrm{align}}^{\mathrm{v}\to\mathrm{t}}\) is the modular contrastive loss on the CMN side (sfdm+dfsm); and \(\mathcal{L}_{\mathrm{s}}\) is the L0 sparsity constraint on \(\hat{\mathbf{m}}^{\mathrm{T}}\) and \(\hat{\mathbf{m}}^{\mathrm{V}}\).

Crucial training details: The model is first warm-up initialized on ShareGPT4v image-caption data (transferring CLIP's image-level sparse mapping knowledge to videos), followed by fine-tuning the encoders at a learning rate of \(10^{-7}\) and training the new modules at \(10^{-4}\), with a batch size of 256 using a ViT-B/16 backbone, supporting up to 248 tokens (surpassing CLIP's original 77-token constraint). Ablation studies (Fig. 7d) reveal that the optimal ratio for \(\lambda_{\mathrm{g}}/\lambda_{\mathrm{tv}}\) is around 1.0; a ratio too small (<0.2) loses global information, whereas a ratio too high (>1.2) leads to frame-level mask collapse.

Key Experimental Results

Main Results

Dataset	Direction	Ours R@1	Prev. SOTA R@1	Gain
ActivityNet	T→V	47.8	46.9 (VideoCLIP-XL)	+0.9
ActivityNet	V→T	46.7	42.9 (CLIP4Clip)	+3.8
MSVD	T→V	52.6	50.4 (X-CLIP)	+2.2
DiDeMo	T→V	57.5	50.8 (InternVideo)	+6.7
VideoUFO	T→V	62.4	57.4 (VideoCLIP-XL)	+5.0
UltraVideo	T→V	58.5	51.8 (ProST)	+6.7

Ablation Study

Configuration	T→V R@1 (ActivityNet)	V→T R@1 (ActivityNet)	Description
Full MoVA (iv)	47.8	46.7	Full dual asymmetric projection
Global loss only (i)	43.7	42.5	Remove all modular objectives, degrading to fine-tuned CLIP
CMN unidirectional only (ii)	35.6	33.7	Remove TMN, SmartCLIP image-level unidirectional mask
TMN + raw token mask (iii)	42.0	41.9	TMN retained, but directly performing temporal masking on raw tokens
w/o Different-Frame Same-Mask	~44.2	—	Significant drop after removing \(\mathcal{L}_{\mathrm{dfsm}}\)

Key Findings

• Both streams are indispensable: Ablation (ii) demonstrates that relying solely on frame-to-text masking (removing the text-to-frame direction of TMN) results in the poorest configuration, with R@1 dropping by over 12 points, directly showing that semantic asymmetry must be modeled bidirectionally.
• Modular objectives each contribute uniquely: Both \(\mathcal{L}_{\mathrm{sfdm}}\) and \(\mathcal{L}_{\mathrm{dfsm}}\) are indispensable. Removing only \(\mathcal{L}_{\mathrm{dfsm}}\) leads to a 3.6 drop in R@1, illustrating that cross-frame constancy (different-frame same-mask) is a key link for decoupling.
• TMN depth insensitivity: 1-2 Transformer layers are already sufficient (Table 8); deeper alternatives (3-8 layers) yield no additional benefits, suggesting that frame-level text subspace selection itself is not a computationally intensive task.
• Robustness to frame count: Performance remains stable within the range of 32 to 160 frames (Table 7), peaking around 96 frames, showing that the model does not rely on brute-force frame increment to capture concepts.
• Parameter efficiency: MoVA has only 174.7M parameters (+7.6% compared to CLIP4Clip's 162.3M), which is far smaller than VideoCLIP-XL (427.9M). Moreover, its training time per epoch is actually faster than CLIP4Clip (76.7min vs 92.9min on 8×MI210).

Highlights & Insights

• Reverse ablation verification of the asymmetric dual-projection design: While most multimodal methods perform single-directional attention or masking, MoVA proves through rigorous ablation that both directions (text-to-frame subspace selection and frame-to-text visual factor selection) contribute independently and are indispensable. This "asymmetric symmetry" design intuition is highly uncommon in CLIP-style literature.
• Closed-loop from latent variable identification theory to algorithm design: The paper first establishes a data generation model (Section 3), proves block-level identifiability (Theorem 4.2), and then directly translates the identification conditions into TMN/CMN architectures coupled with sparse contrastive objectives. This rigorous "theory-to-implementation" dialog is exemplary for contrastive learning studies.
• Transfer benefits from long-text to video generation: Zero-shot replacing the CLIP text encoder in VideoCrafter2 with MoVA's text encoder not only preserves baseline objects but also generates fine texture descriptions like "splatters of rusty orange paint stains" and complex temporal actions like "with rhythmic sway." This vividly demonstrates the transfer value of decoupled multimodal representations beyond retrieval.

Limitations & Future Work

• Reliance on image-level warm-up: MoVA requires warm-up initialization on ShareGPT4v (starting from SmartCLIP weights). If the target domain differs significantly from natural image distributions (such as medical or remote-sensing videos), the warm-up transfer effect may degrade.
• Scalability of frame sampling for long videos: The current method is stable for around 64 frames. However, when videos span thousands of frames (e.g., movies or long surveillance clips), linear sampling of fixed frame counts might miss crucial events. The paper does not address adaptive frame selection or hierarchical sampling strategies.
• Information loss from hard masking: Straight-Through Estimator (STE) hard masking benefits interpretability (with clear text subset activations shown in Figure 4), but the soft-mask variant yields similar performance—for scenes requiring continuous semantic transitions (like blurred event boundaries), hard binarization may drop intermediate semantics.
• Gap between theoretical assumptions and reality: The identification theorem (Theorem 4.2) relies on assumptions such as the smoothness and invertibility of the generator and view diversity. In real-world data, these assumptions can only be approximately satisfied. The rigor of these theoretical guarantees needs further ablation (e.g., validating failure modes on synthetic data where assumptions are violated).

Related Work & Insights

• vs SmartCLIP [Xie2025, ICCV 2025]: SmartCLIP proposes adaptive masking and modular contrastive learning at the image level. MoVA extends this paradigm to video, with the core difference being the introduction of TMN to achieve text-to-frame temporal masking and dual-mask constraints. On the theoretical side, MoVA unifies image and video alignment under a latent variable identification framework, proving identifiability along the temporal dimension.
• vs CLIP4Clip [Luo2022, CVPR 2022]: CLIP4Clip is a milestone in video-adapted CLIP, focusing on similarity computation and late fusion. MoVA inherits its global retrieval loss as a baseline but replaces simple global contrast with modular alignment objectives, achieving frame-level decoupled alignment without sacrificing global semantics.
• vs DGL [Yang2024]: DGL generates local dynamic prompts using shared latent spaces coupled with global-local attention. MoVA differs by basing its design on identifiability theory, and its asymmetric dual masks are also uniquely engineered—DGL's local prompts are implicit, whereas MoVA's masks are explicitly sparse and interpretable (as shown in Figures 4 and 6 of the paper).
• vs VideoCLIP-XL [Wang2024]: VideoCLIP-XL utilizes a larger ViT-L/14 backbone and additional post-pretraining video data (YT-VidLA-800M). MoVA achieves superior results across multiple datasets using a ViT-B/16 backbone with significantly fewer parameters (174.7M vs 427.9M), proving that systematic modular alignment design can substitute for brute-force model scaling.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces latent variable identification theory to video-text contrastive learning, proposes an asymmetric dual projection architecture, and presents a complete theory-algorithm loop.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 6 datasets (long and short videos), dual applications in retrieval and generation, and extensive ablations covering each module, frame count, TMN depth, loss weight, and hard vs. soft masking.
• Writing Quality: ⭐⭐⭐⭐ High theoretical rigor and clear method description, though sections with heavy mathematical symbols (Sections 3-4) might be less accessible to non-theoretical readers.
• Value: ⭐⭐⭐⭐⭐ Resolves the main bottleneck of scaling CLIP from images to videos, offering a new paradigm direction for long-video-text alignment, and can directly replace existing CLIP text encoders in practice.

Related Papers

• [ECCV 2026] Generating a Paracosm for Training-Free Zero-Shot Composed Image Retrieval
• [ECCV 2026] AdaBoosting Text Prompts for Vision-Language Models
• [ECCV 2026] UniTac: A Unified Multimodal Model for Cross-Sensor Tactile Understanding and Generation
• [ECCV 2026] Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models
• [ECCV 2026] Skin-R1: Clinical Knowledge-Guided Dermatological Diagnosis Using Vision-Language Models