SAMWise: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation

Conference: CVPR 2025
arXiv: 2411.17646
Code: https://github.com/ClaudiaCuttano/SAMWISE
Area: Video Segmentation / Multimodal VLM
Keywords: Referring Video Object Segmentation, SAM2, Cross-Modal Temporal Adapter, Tracking Bias, Streaming Processing

TL;DR

By introducing a Cross-Modal Temporal Adapter (CMT) and a Conditional Memory Encoder (CME), SAMWISE infuses natural language understanding and explicit temporal modeling into SAM2 without fine-tuning its original weights. Operating in a streaming fashion, it achieves state-of-the-art (SOTA) performance on Referring Video Object Segmentation (RVOS) with less than 5M additional parameters.

Background & Motivation

Background: Referring Video Object Segmentation (RVOS) aims to segment target objects in a video based on natural language expressions. Existing methods mainly follow two paradigms: first, processing the video in independent short clips (e.g., ReferFormer, MTTR), which loses global temporal context; second, processing the entire video offline (e.g., DsHmp) by first modeling the trajectories of all instances and then selecting the best match, which is not applicable to streaming scenarios.

Limitations of Prior Work: Short-clip methods perform poorly in scenarios requiring long-term motion reasoning (such as the MeViS dataset) because actions can span multiple frames. Although offline methods perform well, they require access to the entire video at once, making them unusable for real-time streaming scenarios. OnlineRefer attempts online context propagation but relies only on past information from a single frame, failing to capture long-term dependencies.

Key Challenge: The fundamental challenge in RVOS is how to maintain global contextual information while operating in a streaming manner (without requiring the full video). Although SAM2 is naturally suited for streaming and has a memory bank mechanism, it lacks three critical capabilities: (i) text understanding (it only accepts prompts like spatial points); (ii) temporal modeling (it extracts features frame-by-frame independently, lacking motion reasoning); and (iii) tracking bias (once it starts tracking an incorrect object, it persists in the error).

Goal: To endow SAM2 with natural language understanding, temporal modeling, and autonomous error-correcting tracking capabilities without fine-tuning SAM2 weights or relying on external large models.

Key Insight: Leveraging lightweight adapter modules injected into a frozen SAM2, preserving its superior segmentation and tracking capabilities while introducing new functionalities. This paradigm draws inspiration from the adapter fine-tuning strategies of pre-trained models like CLIP.

Core Idea: To design a Cross-Modal Temporal (CMT) adapter that models vision-language interactions and temporal evolution simultaneously during the feature extraction stage, and a Conditional Memory Encoder (CME) to detect tracking bias and softly shift the tracking focus.

Method

Overall Architecture

SAMWISE is built upon a frozen SAM2 and a frozen text encoder. The inputs are a video frame sequence and a textual description. In each layer of feature extraction, the CMT adapter is embedded into both the vision encoder and the text encoder, realizing cross-modal fusion and temporal modeling. The [CLS] and verb embeddings from the extracted textual features are extracted as the Contextual Prompt and Motion Prompt respectively, which are then projected via MLPs and passed into the SAM2 Mask Decoder to generate segmentation masks. Finally, the CME module detects whether a new object that better matches the text appears in the current frame, dynamically adjusting the tracking information in the memory bank.

Key Designs

1. Cross-Modal Temporal Adapter (CMT Adapter):

   • Function: Simultaneously inject temporal information and cross-modal cues into the feature extraction stage.
   • Mechanism: The CMT consists of three sub-modules: Hierarchical Selective Attention (HSA) for temporal modeling, Vision-to-Text Attention (VTA) and Text-to-Vision Attention (TVA) for cross-modal fusion. For the temporal part, the feature volume is decomposed into spatio-temporal blocks of size \(T \times P \times P\), and self-attention is performed within each block to avoid the high computational cost of global attention. The block size \(P\) is progressively scaled according to the feature resolution hierarchy to realize multi-scale temporal modeling. In the cross-modal part, VTA allows visual features to attend to textual expressions to identify candidate regions matching the description, and TVA allows text tokens to absorb visual information to adjust semantic understanding based on frame content.
   • Design Motivation: SAM2 extracts features frame-by-frame independently and lacks temporal reasoning. Object motion in videos is usually localized in spatial regions. HSA leverages this prior to perform attention only within local spatio-temporal neighborhoods, which is much more efficient than full token self-attention. Cross-modal interactions align features at early stages rather than only fusing them at the final stage.

2. Dual Prompt Strategy (Contextual + Motion Prompt):

   • Function: Provides dual guidance of semantics and actions for the SAM2 Mask Decoder.
   • Mechanism: The [CLS] embedding is extracted from the adapted text features as the Contextual Prompt (encoding the main subject semantics), and the verb embedding is extracted as the Motion Prompt (encoding action cues). The two are concatenated and projected through a three-layer MLP into the prompt vector for SAM2: \(\rho = W_{\text{prompt}}(\text{CAT}[\mathcal{E}_C, \mathcal{E}_M])\). This prompt is injected in every frame, ensuring that the model focuses on the current frame content while tracking.
   • Design Motivation: In datasets like MeViS, textual descriptions contain action information (e.g., "a cat that is climbing"). Relying solely on global semantics is insufficient, requiring explicit encoding of motion-related verb cues.

3. Conditional Memory Encoder (CME):

   • Function: Detects tracking biases and dynamically shifts the tracking focus.
   • Mechanism: A memory-less token \(\tau_l\) is extracted via cross-attention from memory-free features (unaffected by historical prediction bias) and compared with the mask token \(\tau_m\) output by the Mask Decoder. These two tokens and a learnable [DEC] token are concatenated to perform self-attention, followed by a linear classifier to determine whether a new, better-matching target is detected (\(p_{detect} > 0.5\)). If detected, an unbiased mask \(\mathcal{P}_l\) is computed and spatially soft-fused with the tracking mask \(\mathcal{P}_m\) before being sent to the memory encoder, enabling SAM2 to "see" the new target.
   • Design Motivation: SAM2 suffers from tracking bias: when the correct target has not yet appeared, it may track an incorrect target, and even when the correct target later appears, it does not automatically switch. CME leverages the alignment of unbiased features with text prompts to detect switching opportunities, utilizing soft assignment instead of a hard switch to avoid false positives.

Loss & Training

• Trains only the adapters and the CME module (approx. 4.2-4.9M parameters), while SAM2 and the text encoder are fully frozen.
• Pre-trained on RefCOCO/+/g for 6 epochs (learning rate 1e-4), then fine-tuned on Ref-Youtube-VOS for 4 epochs (learning rate 1e-5), using the Adam optimizer.
• Fine-tuned on MeViS for only 1 epoch, with clip length T=8.
• CME is trained in a self-supervised manner using cross-entropy loss, with samples where memory-free features highlight different objects.

Key Experimental Results

Main Results

Method	MeViS J&F	Ref-YT-VOS J&F	Ref-DAVIS J&F	Total Params
DsHmp (Prev. SOTA)	46.4	67.1	64.9	339M
OnlineRefer	32.3	63.5	64.8	232M
VISA (VLM-7B)	43.5	61.5	69.4	7B
SAMWISE (RoBERTa)	49.5	69.2	70.6	202M
SAMWISE (CLIP)	48.3	67.2	68.5	150M

Ablation Study

Configuration	MeViS J&F
MLP-only (No CMT)	45.2
+ Text-to-Visual	47.5
+ Visual-to-Text	48.3
+ HSA Temporal Modeling	50.3
+ Full CMT	54.2
+ CME	55.5

HSA Block Size	Fixed P=1	Fixed P=4	Fixed P=8	Hierarchical 8/4/2	Hierarchical 16/8/4
J&F	49.7	52.3	53.1	54.2	53.8

Key Findings

• Under streaming processing, SAMWISE outperforms the offline method DsHmp (which requires the entire video clip) by +3.1% on MeViS, proving that a streaming mode combined with a memory bank is more effective than global offline methods.
• It achieves SOTA by training only 4.9M parameters (2.4% of the total), demonstrating extremely high parameter efficiency.
• Within the CMT adapter, cross-modal interaction contributes +5.1%, temporal modeling contributes +3.9%, and CME contributes +1.3%.
• Hierarchical block sizes (8/4/2) in HSA outperform fixed sizes (+1.1%), validating the value of multi-scale temporal modeling.
• Adaptive detection in CME outperforms fixed-frequency detection: "Always" detection drops J&F by 3.5% (50.7 vs 54.2), as overly frequent switching introduces noise.

Highlights & Insights

1. The discovery of the "tracking bias" phenomenon is a significant contribution—once SAM2 tracks an incorrect object, it fails to self-correct, which is particularly fatal in RVOS.
2. Adapting the adapter paradigm from image-based CLIP to video-based SAM2 is a clever strategy, introducing new features while preserving the original model capabilities.
3. HSA leverages the prior of motion locality in videos to perform attention within spatio-temporal neighborhoods, which is far more computationally efficient than full-token self-attention.
4. The 150M-parameter SAMWISE-CLIP version outperforms the 7B-parameter VISA, indicating that task-specialized lightweight adapters can be more effective than general large vision-language models.

Limitations & Future Work

• Streaming processing still requires a certain video clip length (e.g., T=8 frames), which may introduce latency in extreme real-time scenarios.
• The detection threshold of CME is fixed at 0.5, whereas different scenarios might require different thresholds.
• The frozen text encoder may limit the depth of understanding complex semantics.
• Future work could explore longer temporal windows and adaptation to larger variants of SAM2 (e.g., Hiera-L).

Related Work & Insights

• Compared to OnlineRefer's single-frame propagation, SAMWISE utilizes the SAM2 memory bank to achieve true long-range context propagation.
• The cross-modal fusion concept of CMT can be transferred to other vision foundation models requiring textual understanding (e.g., DINO, DINOv2).
• Comparison with large VLM methods indicates that task-specialized lightweight designs outperform general-purpose large models in specific scenarios.

Rating

• Novelty: 8/10 — The discovery of tracking bias and the proposed CME solution are highly creative, and CMT integrates multiple innovations.
• Experimental Thoroughness: 8/10 — Evaluation on three mainstream datasets with detailed ablations, though speed comparison is missing.
• Writing Quality: 8/10 — Well-structured, with intuitive visualizations demonstrating tracking bias.
• Value: 8/10 — Provides an efficient and powerful solution for extending SAM2 to RVOS.

Related Papers

• [CVPR 2025] Style-Editor: Text-driven Object-Centric Style Editing
• [ICCV 2025] Correspondence as Video: Test-Time Adaption on SAM2 for Reference Segmentation in the Wild
• [CVPR 2025] A Distractor-Aware Memory for Visual Object Tracking with SAM2
• [CVPR 2025] SAM2-LOVE: Segment Anything Model 2 in Language-Aided Audio-Visual Scenes
• [CVPR 2025] The Devil is in Temporal Token: High Quality Video Reasoning Segmentation