MomentSeg: Moment-Centric Sampling for Enhanced Referring Video Object Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/Dmmm1997/MomentSeg
Area: Segmentation
Keywords: Referring Video Object Segmentation, Moment-Centric Sampling, Temporal Sentence Grounding, Bidirectional Mask Propagation, Video Large Multimodal Models

TL;DR¶

MomentSeg presents a unified framework that co-optimizes Temporal Sentence Grounding (TSG) and Referring Video Object Segmentation (RefVOS) through a native [FIND] token matching mechanism, a density-modulated Moment-Centric Sampling (MCS) strategy, and Bidirectional Anchor-updated Propagation (BAP), outperforming existing models by 5% on MeViS and 6% on ReVOS without relying on external keyframe models.

Background & Motivation¶

Referring Video Object Segmentation (RefVOS) requires segmenting target objects across video sequences guided by natural language expressions, necessitating both temporal action reasoning and fine-grained spatial tracking. The advent of Video Large Multimodal Models (Video LMMs) integrated with foundation segmentation decoders like SAM2 has popularized the "One Token Seg All" paradigm, wherein a single [SEG] token encodes the target entity across the full video. However, feeding entire video sequences into large multimodal backbones is computationally prohibitive and carries massive visual redundancy, making the selection of query-relevant keyframes a decisive bottleneck for accuracy and efficiency.

Existing frame sampling schemes in LMM-based RefVOS fall into two problematic extremes. On one hand, heuristic strategies such as uniform sampling or taking the initial segment of frames (FirstK) completely ignore text-guided semantic timings, frequently failing when targets appear midway through a clip or when disambiguation requires distinguishing specific actions. On the other hand, recent architectures turn to external models—such as LLaMA-VID, Grounded-VideoLLM, or CLIP—to extract temporal timestamps prior to segmentation. Although external models alleviate frame omission, they introduce severe pipeline fragmentation, heavier memory footprints, and multi-stage latency.

This fundamental tension motivates exploring whether a multimodal model can natively learn to identify query-relevant moments to steer its own sampling and tracking. Core Idea: Co-optimize Temporal Sentence Grounding (TSG) and RefVOS in a unified architecture using a dedicated [FIND] token for timestamp-free temporal matching, which directly steers Moment-Centric Sampling (MCS) for stratified dense-local and sparse-global frame selection and Bidirectional Anchor-updated Propagation (BAP) for drift-resilient tracking.

Method¶

Overall Architecture¶

MomentSeg builds upon the Qwen2.5-VL foundation model and a SAM2 mask decoder, establishing a unified pipeline that eliminates the need for auxiliary keyframe modules. During training, the network jointly learns temporal grounding and referring segmentation: low-resolution uniform frames are injected into the backbone alongside high-resolution sampled frames, where a [FIND] token computes cosine similarities with pooled temporal tokens to learn frame-level alignment without explicit timestamp generation, while the [SEG] token guides mask decoding through the SAM2 decoder. During inference, the similarity distribution generated by the [FIND] token guides Moment-Centric Sampling (MCS) to allocate sampling budgets adaptively, after which Bidirectional Anchor-updated Propagation (BAP) initializes tracking at the most confident key moment and selectively updates memory to prevent cumulative error drift.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Video Frames and Referring Expression"] --> B["Stage 1: [FIND] Token Temporal Matching<br/>Compute frame-query similarity on low-res tokens"]
    B --> C["Stage 2: Moment-Centric Sampling (MCS)<br/>Locate moment center and apply inverse CDF sampling"]
    C --> D["Stage 3: Bidirectional Anchor-updated Propagation (BAP)<br/>Bidirectional SAM2 tracking with adaptive memory cleanup"]
    D --> E["Output: Frame-by-Frame Object Segmentation Masks"]

Key Designs¶

1. [FIND] Token and Frame-Level Similarity Matching: Native Key Moment Identification Without Explicit Timestamps

Prior video LMMs addressing temporal sentence grounding rely heavily on explicit timestamp token generation or specialized positional encoding vectors, which often suffer from autoregressive hallucinations and complex decoding steps. MomentSeg introduces a dedicated [FIND] token that natively maps textual queries to frame features via contrastive metric learning. Low-resolution uniformly sampled frames $I^l_{1:L}$ (injecting no more than 16 visual tokens per frame) and language description $R$ are processed by the multimodal backbone. Frame tokens are projected via an MLP and temporally average-pooled into representations $T^v \in \mathbb{R}^{L_t \times C}$, while [FIND] tokens are mapped via the same MLP to query vectors $T^f \in \mathbb{R}^{N_f \times C}$. The pairwise similarity matrix $\ell_{ij}$ is computed directly using normalized cosine similarity:

\[\ell_{ij} = \frac{(T^f_i)^\top T^v_j}{\|T^f_i\| \|T^v_j\| \tau}\]

\[\mathcal{L}_{\text{find}} = -\frac{1}{|\Omega|} \sum_{(i,j) \in \Omega} \left[ \lambda_p y_{ij} \log \sigma(\ell_{ij}) + (1 - y_{ij}) \log (1 - \sigma(\ell_{ij})) \right]\]

where $\tau = 0.07$, $\sigma$ denotes the sigmoid activation, $y_{ij} \in \{0, 1\}$ is the ground-truth temporal alignment label, and positive weight $\lambda_p = 2.0$. Because the visual backbone spatially and temporally pools video features ($L_t = 0.5 \times L$), bilinear interpolation is applied to map similarity distributions back to the full sequence length. This mechanism produces an exact, continuous frame-query relevance response without invoking external networks.

2. Moment-Centric Sampling (MCS): Stratified Density Allocation Combining Local Detail and Global Scope

Directly selecting the highest-scoring frames via TopK or NearbyK leads to over-clustered frame sets, omitting contextual transitions and temporal boundaries. MCS resolves this trade-off through distribution-guided stratified sampling. The raw similarity scores are smoothed via a 1D Gaussian kernel to produce distribution $S \in \mathbb{R}^T$. A sliding window of length $w$ then locates the optimal moment center $c^*$ maximizing cumulative density:

\[i^* = \arg\max_{i=0 \to T-w} \sum_{j=i}^{i+w-1} S_j, \quad c^* = i^* + \lfloor w/2 \rfloor\]

Using $c^*$ as the split point, the sequence is partitioned into left and right intervals $[0, c^*-1]$ and $[c^*+1, T-1]$. The remaining budget of $K-1$ frames is divided proportionally based on cumulative mass $w_L$ and $w_R$: $k_L = \lfloor (K-1) \frac{w_L}{w_L + w_R} \rfloor$ and $k_R = K - 1 - k_L$. Within each partition, the normalized distribution is converted into a cumulative distribution function (CDF) $F(i)$, and frame indices are chosen at uniformly spaced quantiles $\hat{u}_m = (m - 0.5) / k$ via inverse CDF mapping:

\[i_m = \min \left\{ i \in [a, b] \mid F(i) \ge \hat{u}_m \right\}, \quad m = 1, \dots, k\]

This ensures dense sampling coverage around focal action intervals while maintaining uniform representative coverage across low-relevance background segments, adding less than 0.1% overhead to overall latency.

3. Bidirectional Anchor-updated Propagation (BAP): Dual-Directional Tracking with Adaptive Memory Cleanup

Standard mask propagation mechanisms run forward from the first frame of a video. If the target entity appears late in the sequence, the tracker risks drifting onto false foregrounds before seeing the actual target. BAP addresses this by using the most confident moment center $c^*$ as an anchor to launch bidirectional propagation both forward and backward through time. To further eliminate error accumulation over long temporal horizons, BAP monitors tracking stability at each sampled anchor node $i$:

\[U_i = \begin{cases} 1, & \text{if } \prod_{j=1}^t S_j^t < \lambda \cdot S_i^p \\ 0, & \text{otherwise} \end{cases}\]

where $S_j^t$ is the tracking confidence at step $j$, $\prod_{j=1}^t S_j^t$ reflects accumulated tracking reliability, $S_i^p$ is the direct prediction confidence from the LMM mask output at node $i$, and $\lambda = 0.9$ controls sensitivity. When cumulative tracking confidence drops below the quality threshold established by the current direct prediction, memory reset is triggered ($U_i = 1$). This purges outdated, corrupted cache embeddings and reinitializes propagation state using the fresh LMM prediction, effectively arresting visual drift during occlusions.

Loss & Training¶

The composite optimization objective integrates segmentation and grounding losses: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{seg}} + \alpha \mathcal{L}_{\text{find}}$$ where $\mathcal{L}_{\text{seg}}$ incorporates binary cross-entropy and Dice losses decoded from the [SEG] token state through the SAM2 mask decoder, and $\alpha = 1.0$. Training utilizes 8 NVIDIA H20 GPUs (96GB memory) using LoRA fine-tuning on the Qwen2.5-VL backbone, with a maximum sequence context of 8,192 tokens. Training MomentSeg-3B takes approximately 24 hours.

Key Experimental Results¶

Main Results¶

1. Performance on Referring Video Object Segmentation (RefVOS) Benchmarks

Method	Size	MeViS (valu) $\mathcal{J}\&\mathcal{F}$	MeViS (val) $\mathcal{J}\&\mathcal{F}$	Ref-Youtube-VOS $\mathcal{J}\&\mathcal{F}$	Ref-DAVIS17 $\mathcal{J}\&\mathcal{F}$
ReferFormer	-	-	31.0	62.9	61.1
DsHmp	-	55.3	46.4	67.1	64.9
DeRVOS	-	60.6	51.8	70.0	70.9
LISA-7B	7B	43.2	37.2	53.9	64.8
VideoLISA-3.8B	3.8B	54.5	44.4	63.7	68.8
VISA-7B	7B	-	43.5	61.5	69.4
ViLLa-6B	6B	-	49.4	67.5	74.3
GLUS-7B	7B	60.9	51.3	67.3	-
Sa2VA-Qwen2.5-VL-3B	3B	57.5	50.0	71.0	74.4
MomentSeg-3B (Ours)	3B	62.0	54.8	72.0	76.4
MomentSeg-7B (Ours)	7B	62.6	57.1	72.3	77.4

2. Performance on Reasoning Video Object Segmentation Benchmarks

Method	Size	ReVOS Overall $\mathcal{J}\&\mathcal{F}$	ReVOS Referring $\mathcal{J}\&\mathcal{F}$	ReVOS Reasoning $\mathcal{J}\&\mathcal{F}$	ReasonVOS $\mathcal{J}\&\mathcal{F}$
TrackGPT-13B	13B	45.0	49.5	40.5	-
VISA-13B	13B	50.9	57.4	44.3	-
VideoLISA-3.8B	3.8B	-	-	-	47.5
HyperSeg-3B	3B	55.7	58.5	53.0	-
ViLLa-6B	6B	57.0	-	-	55.4
GLUS-7B	7B	54.9	58.3	51.4	49.9
VRS-HQ-7B	7B	59.1	62.1	56.1	-
Sa2VA-Qwen2.5-VL-3B	3B	58.7	61.5	55.9	49.6
MomentSeg-3B (Ours)	3B	62.6	65.4	59.9	61.7
MomentSeg-7B (Ours)	7B	65.1	67.4	62.8	62.7

Ablation Study¶

1. Progressive Contribution of Core Architectural Components ($\mathcal{J}\&\mathcal{F}$ metric)

ID	Temporal Token Injecting (TTI)	Moment-Centric Sampling (MCS)	Bidirectional Anchor Propagation (BAP)	MeViS (valu)	MeViS (val)	Ref-DAVIS17	Ref-Youtube-VOS	ReasonVOS	ReVOS
1 (Baseline)	-	-	-	57.0	51.3	75.5	70.1	54.1	58.3
2	✓	-	-	58.7 (+1.7)	52.3 (+1.0)	76.2 (+0.7)	70.3 (+0.2)	58.1 (+4.0)	59.2 (+0.9)
3	✓	✓	-	61.3 (+2.6)	53.9 (+1.6)	76.8 (+0.6)	70.8 (+0.5)	60.8 (+2.7)	60.4 (+1.2)
4 (Full)	✓	✓	✓	62.0 (+0.7)	54.1 (+0.2)	76.4 (-0.4)	72.0 (+1.2)	61.7 (+0.9)	62.6 (+2.2)

2. Comparison Across Frame Sampling Strategies

Strategy	MeViS (valu) $\mathcal{J}\&\mathcal{F}$	Ref-DAVIS17 $\mathcal{J}\&\mathcal{F}$	ReasonVOS $\mathcal{J}\&\mathcal{F}$	Behavioral Characteristics
FirstK (Baseline)	58.3	76.3	55.9	Blinds the model to target actions occurring in later segments
Uniform	59.6 (+1.3)	76.2 (-0.1)	57.1 (+1.2)	Spans the entire video but dilutes dense motion cues
KeyFrame (TopK)	60.1	76.9	60.2	Concentrates heavily on peak frames, losing broader context
KeyFrame (NearbyK)	60.9 (+0.8)	77.1 (+0.2)	59.9 (-0.3)	Clusters strictly around local extrema, weak on long videos
MCS (Ours)	61.3 (+3.0)	77.3 (+1.0)	60.8 (+4.9)	Balances dense focus on key moments with sparse global context

Key Findings¶

Crucial Role of Adaptive Sampling: On MeViS, which features dense action expressions and visually indistinguishable candidate objects, MCS surpasses FirstK by 3.0 percentage points. On ReasonVOS, MCS brings an even larger gain of 4.9 percentage points, highlighting that uniform or prefix sampling fails in complex relational reasoning where critical moments occur midway through videos.
Tracking Robustness via BAP: On MeViS, BAP increases the initial target hit rate (Initial Shot Ratio) from 88.7% to 96.6% (+7.9%) and improves initial frame mIoU by 9.3 percentage points. Furthermore, adaptive memory cleaning corrects 7.1% of severe tracking drift cases (frames with IoU < 0.3 improved to IoU > 0.7), validating its stability over long sequences.
Negligible Latency Overhead: Detailed timing breakdowns demonstrate that native TSG adds only 7% MLLM computation overhead, while the MCS algorithm accounts for less than 0.1% of runtime. Overall inference latency increases by merely 3% compared to Sa2VA-8B while delivering more than 10 points improvement on MeViS.

Highlights & Insights¶

Native Grounding via Single Query Token: By training a dedicated [FIND] token through direct frame-feature contrastive matching, MomentSeg internalizes temporal sentence grounding without relying on clumsy text-based timestamp generation or external pre-trained temporal localization networks.
Statistical Inverse CDF for Continuous Temporal Allocation: Treating smoothed similarity distributions as continuous densities and partitioning sampling budgets via inverse CDF quantiles offers an elegant, closed-form solution to the trade-off between local fine-grained motion capture and global contextual representation.
Broad Transferability Beyond RefVOS: The combination of [FIND]-guided moment discovery, density-modulated sampling, and bidirectional anchor propagation can directly enhance long-video question answering, spatio-temporal action localization, and embodied trajectory reasoning under constrained visual budgets.

Limitations & Future Work¶

Marginal Utility on Static Sequences: On datasets like Ref-DAVIS17 where targets persist statically throughout the clip, the advantage of moment-centric sampling diminishes, and BAP yields a slight variation (-0.4 points), indicating that heuristic uniform sampling is already near-optimal for static videos.
Fixed Window Prior: Identifying the moment center $c^*$ currently relies on a fixed sliding window hyperparameter $w$, which may not dynamically scale to both transient micro-actions and prolonged interactions lasting hundreds of frames.
Future Directions: Exploring multi-scale learned kernel windows for variable-duration events and extending the [FIND] token mechanism to handle multi-query, multi-instance concurrent temporal retrieval.

vs Sa2VA / VideoLISA: Sa2VA relies on fixed prefix FirstK frames while VideoLISA uses uniform downsampling; both ignore temporal content relevance. MomentSeg uses native [FIND]-driven MCS to dynamically allocate frames where action occurs, outperforming them significantly on motion-centric benchmarks.
vs VISA / ViLLa / GLUS: These approaches insert standalone external models like LLaMA-VID or Grounded-VideoLLM into the pipeline for keyframe localization. MomentSeg consolidates TSG and RefVOS into a unified model, cutting system complexity and external dependencies.
vs Standard Forward SAM2: Default SAM2 propagates sequentially from the opening frame forward. BAP initiates propagation from high-confidence moment centers bidirectionally and cleans accumulated tracking noise, substantially improving long-range tracking stability.

Rating¶

Novelty: ⭐⭐⭐⭐☆ [Novel native [FIND] token temporal matching unified with RefVOS, accompanied by an elegant moment-centric stratified sampling design]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across RefVOS, Reasoning VOS, TSG, and Referring Image Segmentation, backed by thorough ablations on latency, robustness, and sampling strategies]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural formulation, rigorous methodology, and crisp experimental narrative addressing exact failure modes of prior pipelines]
Value: ⭐⭐⭐⭐☆ [Provides an effective blueprint for processing long video sequences within multimodal foundation models under tight computational constraints]