title: >- [Paper Note] QST-SAM: Leveraging Cross-modal Instructions for Few-shot Referring Video Object Segmentation description: >- [ECCV 2026][Segmentation][Few-Shot VOS] Decoupling query text intent and support visual exemplars with SAM 2 via tripartite cross-modal attention and multi-priors prompting. tags: - ECCV 2026 - Segmentation - Video Object Segmentation - Few-Shot Learning - SAM 2 date: 2026-09-19 content_hash: 436dc917ea41dcdc
QST-SAM: Leveraging Cross-modal Instructions for Few-shot Referring Video Object Segmentation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Segmentation
Keywords: Video Object Segmentation, Few-Shot Learning, Cross-modal Fusion, SAM 2, Referring Segmentation
TL;DR¶
Addressing cross-video textual semantic mismatch and multi-modal alignment challenges in Few-Shot Referring Video Object Segmentation (FSRVOS), this paper decouples semantic intent from visual exemplars (retaining language supervision exclusively at the query level) and introduces QST-SAM on top of SAM 2, integrating a Support Instruction Compressor, a tripartite Q-S-T Transformer, and a Multi-Priors Prompting module to achieve state-of-the-art segmentation.
Background & Motivation¶
Video Object Segmentation (VOS) faces substantial practical bottlenecks due to its prohibitive reliance on labor-intensive, frame-by-frame dense annotations. To alleviate this supervision burden, three alternative paradigms have evolved: unsupervised VOS removes manual inputs entirely but yields unstable or drifting masks; referring VOS (RVOS) allows users to specify targets via natural language descriptions but struggles with cross-modal ambiguity when descriptions are under-specified or multiple similar instances co-exist; and few-shot VOS (FSVOS) relies on a small set of annotated support exemplars of the same category, but falters when distractors appear or when support and query frames exhibit dramatic variations in pose, viewpoint, and appearance. Consequently, natural language (encoding what to segment as semantic intent) and visual exemplars (encoding what it looks like as empirical evidence) offer fundamentally complementary supervision cues.
A recent pioneer in Few-Shot Referring Video Object Segmentation (FSRVOS), CMA-ISM, attempted to unify both paradigms by providing textual descriptions for both query videos and individual support episodes. However, support-side descriptions are written independently for specific support instances and only share coarse category-level semantics across videos; their instance-specific attributes do not transfer directly to the query, inevitably introducing cross-video semantic misalignment and supervisory redundancy. In reality, the query-level textual description is already sufficient to provide context-aware, unambiguous intent, making extra support-side textual descriptions an unnecessary source of noise and annotation overhead.
The angle of attack in this paper is to revisit FSRVOS under a streamlined, decoupled formulation: language supervision is retained exclusively at the query level to specify semantic intent, while masked support images serve purely as instance-level visual evidence. Under this cleaner formulation, the core challenge is how to stably integrate query video spatiotemporal features, support visual appearance, and textual intent without noise contamination. Core idea: decouple query-level linguistic intent from support-level visual evidence, and build a SAM 2-based framework named QST-SAM that condenses support features into compact instructions, aligns cross-modal cues via a Q-S-T Transformer, and conditions the mask decoder via adaptive spatial mask priors and semantic prompt priors.
Method¶
Overall Architecture¶
Built upon the Segment Anything Model 2 (SAM 2), QST-SAM operates across three streams: the query video stream (Query), the support exemplar stream (Support), and the query text stream (Text). The inputs consist of an \(F\)-frame query video \(V=\{I_f\}_{f=1}^F\), a natural language referring expression, and \(N\) external masked support images sharing the same target category. Using frozen SAM 2 image and text backbones, the model extracts high-dimensional representations. The Support Instruction Compressor first distills dense support tokens into compact instruction vectors. Next, the Q-S-T Transformer performs bidirectional, multi-stage cross-modal refinement among query, support, and text tokens. Finally, the Multi-Priors Prompting module establishes adaptive sparse query-support correspondences to construct continuous spatial mask priors alongside text semantic priors, jointly conditioning SAM 2's Memory Attention and Mask Decoder for robust per-frame mask generation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-modal Inputs<br/>Query video frames + Query text + Masked support images"] --> B["Feature Extraction & Temporal Alignment<br/>Frozen SAM 2 backbones + Cross-modal temporal adapter"]
B --> C["Support Instruction Compressor<br/>Attention-weighted compression to filter background clutter"]
B --> D["Text & Query Spatiotemporal Tokens"]
C & D --> E["Q-S-T Transformer<br/>T2S/S2T semantic refinement + S2Q/Q2S contextual enhancement"]
E --> F["Multi-Priors Prompting<br/>Adaptive percentile sparse Q-S mask prior + Text semantic prior"]
F --> G["Spatiotemporal Mask Decoding<br/>SAM 2 memory attention & mask decoder per frame"]
Key Designs¶
1. Support Instruction Compressor: Distilling Compact Visual Evidence
Under the \(N\)-shot setting, support images produce a dense feature tensor \(T_s \in \mathbb{R}^{N \times H_s \times W_s \times C}\). Direct cross-attention between this raw tensor and query frames leads to massive computational overhead and injects irrelevant background clutter that corrupts the segmentation guidance. The Support Instruction Compressor resolves this by distilling \(T_s\) into a compact set of \(K\) highly informative instruction tokens \(S=\{S_k\}_{k=1}^K\).
The module first projects \(T_s\) down to dimension \(C'\) through a linear layer and splits it into two parallel MLP branches: \(\text{MLP}_v\) generates spatial value features \(V_s \in \mathbb{R}^{N \times H \times W \times C'}\), while \(\text{MLP}_{\text{attn}}\) predicts \(K\) spatial attention maps \(\alpha_s \in \mathbb{R}^{N \times H \times W \times K}\). To eliminate background noise, the attention maps are explicitly gated by element-wise multiplication with the interpolated support ground-truth masks \(\tilde{M}_s\) and normalized: \(\alpha'_s = \sigma(\alpha_s) \odot \tilde{M}_s\). The instruction set \(S\) is then derived by weighted aggregation across all support instances: $\(S_k = \frac{1}{N} \sum_{n=1}^N \sum_{i=1}^H \sum_{j=1}^W \alpha'_s{}^{(n, i, j, k)} V_s^{(n, i, j, :)}\)$ The resulting \(S \in \mathbb{R}^{K \times C'}\) discards spatial redundancies and isolates the most discriminative visual patterns of the novel class.
2. Q-S-T Transformer: Bidirectional Multi-modal Synergy
Although support exemplars and query targets share the same semantic class, they often exhibit pronounced intra-class variations in posture, viewpoint, lighting, and texture. Naively matching raw support cues with query frames can easily misguide the model. The Q-S-T Transformer addresses this through a two-stage bidirectional interaction: - Text-Guided Semantic Refinement: Refines the support instructions \(S\) under the supervision of text tokens \(T\). A Text-to-Support (T2S) cross-attention step allows text tokens to extract consistent visual attributes to produce \(T'\), followed by a Support-to-Text (S2T) cross-attention step that propagates the refined semantics back to the support branch to obtain \(S'\). Irrelevant visual attributes divergent from the referring expression are thereby suppressed. - Spatio-Contextual Enhancement: Couples the refined support instructions \(S'\) with query tokens \(Q\). Support-to-Query (S2Q) enables the support stream to absorb dynamic temporal motions and spatial contexts from the query to yield \(S''\), while Query-to-Support (Q2S) injects category-level structural knowledge back into query tokens \(Q\), yielding enriched query representations \(Q'\) optimized for boundary localization.
3. Multi-Priors Prompting: Adaptive Percentile Sparse Matching
Conventional RVOS models pass only textual embeddings into the prompt encoder, leaving the mask decoder without precise spatial initialization. The Multi-Priors Prompting module constructs complementary spatial mask priors alongside semantic textual priors to provide well-grounded prompts for SAM 2.
To derive the spatial mask prior, a Q-S Matching module evaluates cosine similarities between foreground-masked support tokens and query tokens, producing affinity matrices \(A_n\). To suppress spurious long-tail correlations without relying on a brittle fixed threshold, the module applies an adaptive percentile-based sparsification: only the top \(\tau\%\) affinity values within each matrix are retained, and all others are zeroed out: $\(\hat{\mathbf{A}}_n(i, j) = \begin{cases} \mathbf{A}_n(i, j), & \text{if } \mathbf{A}_n(i, j) \ge \mathcal{T}_\tau(\mathbf{A}_n) \\ 0, & \text{otherwise} \end{cases}\)$ The structural priors from support masks \(\tilde{M}_s^n\) are then projected into the query domain through \(\hat{\mathbf{A}}_n\), yielding the visual mask prior \(P_m = \frac{1}{N}\sum_{n=1}^N \hat{\mathbf{A}}_n \tilde{\mathbf{M}}_s^n\). Simultaneously, text tokens are mapped via an MLP to form semantic prior \(P_t\). Conditioning SAM 2's memory-enhanced query feature \(Q_{\text{out}}\) on both \(P_m\) and \(P_t\) empowers the Mask Decoder to delineate sharp, contextually grounded object boundaries.
Loss & Training¶
The framework is initialized from SAMWISE weights pre-trained on RefCOCO/+/g and fine-tuned for 4 epochs on the new Few-Shot Ref-YouTube-VOS dataset split. During training, the entire SAM 2 vision backbone (Hiera-B) and text encoder (RoBERTa) remain frozen; only the newly added compressor, Q-S-T Transformer, prompting layers, and cross-modal adapter are optimized. The training uses the Adam optimizer with a learning rate of \(1 \times 10^{-5}\) and clip lengths of \(T = 8\) frames.
Key Experimental Results¶
Main Results¶
The model is evaluated on the newly constructed decoupled FSRVOS benchmark (Ref-DAVIS17 and Ref-YouTube-VOS) as well as the legacy FSRVOS benchmark (Mini-Ref-YouTube-VOS). All few-shot tests use \(N=5\) support shots by default. The table below details performance comparisons on Ref-DAVIS17 and Ref-YouTube-VOS (\(\uparrow\) indicates gain over corresponding zero-shot baselines):
| Method | Venue | Setting | Ref-DAVIS17 \(\mathcal{J}\&\mathcal{F}\) | Ref-DAVIS17 \(\mathcal{J}\) | Ref-DAVIS17 \(\mathcal{F}\) | Ref-YouTube-VOS \(\mathcal{J}\&\mathcal{F}\) | Ref-YouTube-VOS \(\mathcal{J}\) | Ref-YouTube-VOS \(\mathcal{F}\) |
|---|---|---|---|---|---|---|---|---|
| DsHmp | CVPR'24 | Zero-shot | 61.3 | 58.1 | 64.4 | 54.1 | 51.7 | 56.4 |
| DsHmp | CVPR'24 | Few-shot | 64.5 (+3.2) | 62.7 | 66.2 | 57.7 (+3.6) | 56.4 | 58.9 |
| LoSh | CVPR'24 | Zero-shot | 60.4 | 57.4 | 63.4 | 55.9 | 53.2 | 58.5 |
| LoSh | CVPR'24 | Few-shot | 64.9 (+4.5) | 62.2 | 67.5 | 58.1 (+2.2) | 56.7 | 59.4 |
| ReferDINO | ICCV'25 | Zero-shot | 65.9 | 63.7 | 68.1 | 59.8 | 57.5 | 62.1 |
| ReferDINO | ICCV'25 | Few-shot | 67.8 (+1.9) | 65.5 | 70.1 | 60.6 (+0.8) | 58.4 | 62.8 |
| SAMWISE | CVPR'25 | Zero-shot | 65.5 | 63.5 | 67.4 | 59.3 | 57.2 | 61.3 |
| SAMWISE | CVPR'25 | Few-shot | 67.5 (+2.0) | 65.3 | 69.6 | 60.4 (+1.1) | 58.3 | 62.4 |
| QST-SAM (Ours) | ECCV'26 | Few-shot | 70.3 (+4.8) | 67.5 | 73.1 | 62.4 (+3.1) | 60.9 | 63.9 |
Comparison on the legacy Mini-Ref-YouTube-VOS benchmark (\(\text{Text}_{\text{sup}}\) denotes reliance on support-side text):
| Method | Support Text \(\text{Text}_{\text{sup}}\) | \(\mathcal{J}\&\mathcal{F}\) | \(\mathcal{J}\) | \(\mathcal{F}\) |
|---|---|---|---|---|
| DANet (CVPR'21) | ✓ | 42.3 | 40.8 | 43.7 |
| HPAN (TCSVT'23) | ✓ | 50.9 | 51.1 | 50.7 |
| CMA-ISM (IJCV'25) | ✓ | 54.0 | 53.1 | 54.8 |
| QST-SAM (Ours) | ✗ (Support-text-free) | 59.2 | 58.2 | 60.1 |
Ablation Study¶
Ablations on the Ref-YouTube-VOS dataset isolate the contributions of key architectural components and analyze instruction budget \(K\):
Table 1: Ablation of Architectural Components
| Config | \(\mathcal{J}\&\mathcal{F}\) | \(\Delta \mathcal{J}\&\mathcal{F}\) | \(\mathcal{J}\) | \(\mathcal{F}\) | Note |
|---|---|---|---|---|---|
| QST-SAM (Full Model) | 62.4 | — | 60.9 | 63.9 | Complete proposed model |
| w/o Q-S-T Transformer | 60.8 | -1.6 | 59.4 | 62.1 | Largest drop; confirms importance of multi-modal fusion core |
| w/o Multi-Priors Prompting | 61.7 | -0.7 | 60.3 | 63.1 | Lacks visual mask prior conditioning for prompt encoder |
| w/o both | 60.4 | -2.0 | 58.3 | 62.4 | Degrades to naive few-shot baseline |
Table 2: Instruction Count \(K\) and Memory Footprint
| Num. of Instructions \(K\) | \(\mathcal{J}\&\mathcal{F}\) | \(\mathcal{J}\) | \(\mathcal{F}\) | Memory Usage (GB) |
|---|---|---|---|---|
| \(K = 4\) | 59.8 | 58.1 | 61.4 | 38.1 |
| \(K = 8\) | 61.3 | 59.8 | 62.7 | 38.9 |
| \(K = 16\) | 61.8 | 60.4 | 63.2 | 39.7 |
| \(K = 32\) (Ours) | 62.4 | 60.9 | 63.9 | 42.2 |
| \(K = 64\) | 62.6 | 61.4 | 63.8 | 47.6 |
Key Findings¶
- Q-S-T Transformer is the primary driver of performance: Removing the Q-S-T Transformer triggers the steepest decline (-1.6% \(\mathcal{J}\&\mathcal{F}\)). This verifies that naive feature concatenation or unguided cross-attention fails to overcome intra-class variation, whereas text-guided support refinement ensures consistent cross-modal conditioning.
- Support-text-free setting outperforms legacy dual-text models: On Mini-Ref-YouTube-VOS, QST-SAM discards support-side text annotations (\(\text{Text}_{\text{sup}} = \text{False}\)) yet attains 59.2% \(\mathcal{J}\&\mathcal{F}\), outperforming the prior SOTA CMA-ISM (54.0%) by an impressive margin of +5.2%. This validates that support-side text introduces semantic clutter rather than beneficial signals.
- \(K=32\) offers the optimal performance-efficiency trade-off: Increasing \(K\) from 4 to 32 provides a continuous +2.6% \(\mathcal{J}\&\mathcal{F}\) improvement. Pushing \(K\) to 64 yields only +0.2% gain while increasing memory consumption by 5.4 GB, establishing \(K=32\) as the best operating point.
Highlights & Insights¶
- Principled Supervision Simplification: Identifies the fundamental flaw in prior FSRVOS setups where support-side instance descriptions caused cross-video semantic misalignment. Stripping support-side text restores clean task boundaries and slashes annotation overhead.
- Mask-Gated Instruction Compression: Rather than processing dense, noisy support feature grids, condensing them into \(K=32\) instruction tokens weighted by ground-truth foreground masks effectively isolates pure visual exemplars.
- Adaptive Percentile Sparsification for Prior Transfer: Replacing heuristic absolute thresholds with sample-wise percentile filtering (\(\mathcal{T}_\tau\)) ensures robust visual prior propagation across disparate video distributions.
Limitations & Future Work¶
- Reliance on Foundation Model Pretraining: The framework relies on frozen Hiera-B and RoBERTa backbones; severe distribution shifts (e.g., medical endoscopy, infrared imaging) could limit instruction quality.
- Memory Demands on Long Sequences: While instruction compression cuts support token overhead, the spatiotemporal memory banks and attention operations in SAM 2 still require significant memory (42.2 GB at \(K=32\)), constraining deployment on memory-limited consumer devices.
- Future Directions: Exploring training-free memory pruning for long video sequences, and extending the formulation to open-vocabulary and audio-visual referring segmentation.
Related Work & Insights¶
- vs CMA-ISM (IJCV 2025): CMA-ISM relies on paired text descriptions for both query and support videos. QST-SAM dispenses with support-side text, avoiding semantic cross-video interference, and surpasses CMA-ISM by 5.2% \(\mathcal{J}\&\mathcal{F}\) under weaker supervision.
- vs SAMWISE (CVPR 2025): While SAMWISE is a strong RVOS baseline, a direct adaptation to few-shot VOS yields only a modest +2.0% \(\mathcal{J}\&\mathcal{F}\) gain on Ref-DAVIS17. QST-SAM achieves a +4.8% jump, illustrating the efficacy of explicit support compression and Q-S-T cross-attention.
- vs Pure Few-shot VOS (DANet, HPAN): Traditional FSVOS methods fail when multiple identical-category distractors appear. QST-SAM demonstrates that query-level linguistic intent effectively disambiguates identical instances.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Conceptually redefines the FSRVOS problem formulation; proposes an elegant, modular integration with SAM 2.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 5-fold cross-validation, zero-shot transfer on Ref-DAVIS17, legacy benchmark comparisons, and ablation analysis.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-motivated rationale, clear architectural explanations, and mathematically sound formulations.
- Value: ⭐⭐⭐⭐☆ Provides a pragmatic and effective blueprint for low-supervision video instance tracking and segmentation.