SAM-MT: Real-Time Interactive Multi-Target Video Segmentation¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://henghuiding.com/SAM-MT/
Area: Segmentation
Keywords: Video Object Segmentation, Interactive Segmentation, Multi-Target Tracking, Decoupled Masked Attention, Sparse Memory

TL;DR¶

Built upon the SAM2 framework, SAM-MT transforms multi-target video segmentation into an efficient interactive system by combining decoupled masked attention with a query-based sparse memory, completely decoupling inference latency from target count to achieve over 36 FPS in real-time across 10+ targets.

Background & Motivation¶

Video Object Segmentation (VOS) tracks and segments user-specified objects in open-world videos guided by interactive prompts such as clicks, boxes, or initial masks. Dominant frameworks, spearheaded by Space-Time Memory (STM) networks and recent foundation models like Segment Anything Model 2 (SAM2), maintain rich pixel-level representations in dense memory banks to match against subsequent frames. While these approaches achieve unprecedented segmentation precision and temporal stability on single-object benchmarks, their core designs remain inherently tailored for single-target propagation. When deployed to real-world scenarios requiring simultaneous tracking of dense targets—such as autonomous driving, sports analysis, or multi-robot swarms—the architecture hits a severe scalability bottleneck.

The core tension stems from how existing methods scale to multiple objects: they typically replicate the entire single-target pipeline for every additional instance. Even methods that share backbone feature extraction, such as STCN, XMem, or Cutie, still execute mask decoding and memory encoding independently per object. In SAM2, each object accumulates its own dense pixel-level memory of 4,096 tokens, causing computation and GPU memory footprint to scale linearly with target count. Consequently, runtime frame rates collapse under multi-target workloads: SAM2.1-B+ runs at 37.2 FPS for a single target, but drops precipitously to 17.8 FPS for 3 targets, 12.4 FPS for 5 targets, and 3.7 FPS for 20 targets. An ad-hoc workaround of merging all targets into a single composite mask discards individual identity IDs entirely and forms unnatural, irregular shapes that standard VOS models fail to track accurately.

Real-world applications therefore require an architecture that maintains distinct instance identities while running at near-single-object computational cost regardless of target count. The key insight is that global scene context can be universally shared, whereas individual instance evolution across time can be tracked using compact, explicit query vectors. Core idea: decouple multi-target video segmentation into a shared dense global context and parallel explicit target queries, isolate cross-target interference via decoupled masked attention, and track temporal evolution with a query-based sparse memory and identity transformer to achieve real-time multi-target segmentation with target-independent latency.

Method¶

Overall Architecture¶

SAM-MT inherits SAM2's image encoder and dense memory architecture while comprehensively restructuring interactive prompt handling, decoding, and temporal propagation. The end-to-end framework operates through the following coordinated pipeline:

In the initial interaction frame, the user provides clicks paired with unique target IDs. These clicks are mapped to target-specific prompt queries and concatenated with shared global queries. In the decoder self-attention layers, decoupled masked attention is applied to allow every target query to access the global context while blocking attention between queries belonging to different targets. Following two-way cross-attention with image features, query consolidation uses a lightweight weighting head to compress multiple click queries per target into a single compact Target Query.

During subsequent video propagation frames, a unified first-in-first-out (FIFO) query-based sparse memory maintains historical target queries across a sliding temporal window with minimal memory overhead. At each time step, preceding target queries are routed through an identity transformer with an identity-aware mask that strictly restricts each query to cross-attend only to its own historical trajectory. The updated queries are combined with global queries and decoded into distinct per-target masks. Concurrently, a unified global mask of all combined targets updates the shared dense memory, preserving robust scene-level context without duplicating decoders or memory buffers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-Target Interactive Input<br/>User Clicks + Target IDs"] --> B["Decoupled Masked Attention<br/>Blocks Cross-Target Leakage & Shares Global Context"]
    B --> C["Query Consolidation<br/>Compresses Multi-Click Queries into One Target Query"]
    C --> D["Query-Based Sparse Memory<br/>FIFO Window Stores Historical Query Trajectories"]
    D --> E["Identity Transformer<br/>Per-Target Self-History Retrieval to Prevent Drift"]
    E --> F["Instance & Global Masks Output<br/>Outputs Independent Masks & Updates Memories"]

Key Designs¶

1. Decoupled Masked Attention: Isolating Cross-Target Interference While Sharing Global Context

Allowing queries from different targets to freely attend to one another in self-attention causes cross-instance feature leakage and identity confusion. Conversely, completely isolating all queries prevents targets from understanding the global scene layout and mutual occlusions. SAM-MT introduces decoupled masked attention in the decoder self-attention layers. Let global queries be \(G \in \mathbb{R}^{N_g \times C}\), target queries for target \(q\) be \(Q_q \in \mathbb{R}^{N_q \times C}\), and the concatenated query matrix be \(Q = [G; Q_1; \dots; Q_k] \in \mathbb{R}^{N \times C}\). The masked self-attention is computed as:

\[Q' = \text{softmax}\left(\frac{Q K^\top}{\sqrt{C}} + \mathcal{M}\right) V\]

The attention mask \(\mathcal{M} \in \mathbb{R}^{N \times N}\) enforces asymmetric visibility: rows corresponding to global queries (\(1 \le i \le N_g\)) have full visibility across all columns (entries set to 0); rows corresponding to target \(q\) can attend to all global query columns (\(1 \le j \le N_g\)) and columns within their own index range (\(i, j \in \text{Rng}(q)\)), but entries for columns belonging to any other target \(p \ne q\) are masked to \(-\infty\). This design establishes an explicit instance isolation barrier while preserving shared global visual context.

2. Query Consolidation: Compressing Multi-Point Prompts into a Unique Target Query

In the initial interaction frame, users may supply multiple positive and negative clicks per object (\(N_q \ge 1\)) to delineate complex boundaries. Propagating all \(N_q\) queries across long sequences would inflate temporal computation and introduce prompt weight imbalance over time. SAM-MT introduces a lightweight MLP weighting head \(f_{\text{weight}}\) that assigns importance scores \(w_{q,i}\) across the refined queries \(Q'_{q,i}\):

\[w_{q,i} = \frac{\exp(f_{\text{weight}}(Q'_{q,i}))}{\sum_{j=1}^{N_q} \exp(f_{\text{weight}}(Q'_{q,j}))}, \quad \hat{Q}_q = \sum_{i=1}^{N_q} w_{q,i} \cdot Q'_{q,i}\]

This consolidation occurs exclusively in the initial interaction frame, collapsing multiple prompt points into a single \(1 \times C\) Target Query \(\hat{Q}_q\). In all subsequent propagation frames, each target is represented by this single compact query (\(\hat{Q}_{q, t} = Q'_{q, t}\)), anchoring cross-frame propagation complexity strictly to the number of objects \(k\) rather than the number of interaction points.

3. Query-Based Sparse Memory: Lightweight Object-Level Temporal Buffer

SAM2 stores dense pixel-level memory representations of \(T \times HW \times C\) (typically 4,096 tokens per frame per object), making multi-object replication prohibitively expensive in VRAM. SAM-MT adopts a dual-memory approach: dense memory is computed only once for the combined union mask of all targets to capture shared background and scene changes. For individual target evolution, a query-based sparse memory maintains historical target queries. At frame \(t\), the consolidated queries \(\hat{\mathbf{Q}}_t = [\hat{Q}_{1,t}, \dots, \hat{Q}_{k,t}] \in \mathbb{R}^{k \times C}\) are enqueued into a FIFO memory over window size \(T\), retaining the initial frame and the \(T-1\) most recent frames:

\[M_{\text{sparse}, t} = [\hat{\mathbf{Q}}_t, \hat{\mathbf{Q}}_{t-1}, \dots, \hat{\mathbf{Q}}_{t-T+2}, \hat{\mathbf{Q}}_0] \in \mathbb{R}^{T \cdot k \times C}\]

Because each object consumes only one token per frame, per-object memory footprint decreases by several orders of magnitude. This enables expanding the temporal window to 16 or 32 frames without VRAM spikes, significantly boosting re-identification under long occlusions.

4. Identity Transformer: History Cross-Attention with Identity Guardrails

During temporal propagation, preceding queries \(\hat{\mathbf{Q}}_{t-1} \in \mathbb{R}^{k \times C}\) retrieve relevant temporal context from \(M_{\text{sparse}, t-1}\) to adapt to appearance changes. To prevent target queries from absorbing visual representations of nearby or visually similar distractors, an identity-aware transformer cross-attends to the sparse memory under a structured mask \(\tilde{\mathcal{M}}\):

\[\mathbf{Q}_t = \text{softmax}\left(\frac{\hat{\mathbf{Q}}_{t-1} M_{\text{sparse}, t-1}^\top}{\sqrt{C}} + \tilde{\mathcal{M}}\right) M_{\text{sparse}, t-1} + \hat{\mathbf{Q}}_{t-1}\]

The identity mask \(\tilde{\mathcal{M}}\) enforces that the query for target \(i\) only attends to its own temporal indices \(\Omega_i = \{i + \tau \cdot k \mid 0 \le \tau \le T-1\}\), with all other cross-target positions set to \(-\infty\). This strict identity-restricted cross-attention prevents memory corruption and suppresses temporal identity drift.

Loss & Training¶

SAM-MT employs a two-stage training scheme with dedicated multi-target supervision: - Two-Stage Scheme: Stage I pretrains on static images to master one-shot interactive multi-target segmentation from clicks. Stage II trains on video sequences to stabilize temporal memory retrieval and cross-frame identity consistency. - Strided Sampling: To overcome GPU memory limits during video training, SAM-MT combines consecutive frame sampling with 4-frame strided sampling over a 32-frame temporal span, exposing the network to long-term dynamics, occlusions, and disappearances. - Overlap Prevention Loss: To penalize ambiguous overlapping predictions and enforce mutual exclusivity across targets, an overlap loss is imposed on predicted probability maps \(P_i \in [0, 1]^{H \times W}\):

\[\mathcal{L}_{\text{ovl}, i} = \operatorname{Mean}\left(P_i \odot \sum_{j \ne i} P_j\right)\]

The overall objective combines individual target losses \(\mathcal{L}_I\) (average of Focal, Dice, and Overlap losses across \(k\) targets) and global mask loss \(\mathcal{L}_G\) (Focal, Dice, IoU, and Object score losses): \(\mathcal{L}_{\text{total}} = \mathcal{L}_I + \mathcal{L}_G\).

Key Experimental Results¶

Main Results¶

SAM-MT was quantitatively evaluated across six standard VOS benchmarks. Both SAM-MT and SAM2.1-B+ were initialized with identical interactive prompts (2 positive clicks per target in the initial frame), whereas other baselines received ground-truth masks.

Dataset	Metric	SAM-MT (Ours, Click)	SAM2.1-B+ (Click)	Cutie-base (Mask)	SwinB-DeAOT (Mask)	Gain (vs SAM2.1)
MOSEv2-val	\(\mathcal{J}\&\dot{\mathcal{F}}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	43.0 / 41.4 / 44.5	41.1 / 39.4 / 42.8	42.8 / 41.1 / 44.4	36.7 / 34.9 / 38.6	+1.9 / +2.0 / +1.7
MOSEv1-val	\(\mathcal{J}\&\mathcal{F}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	68.2 / 64.4 / 72.0	65.1 / 60.7 / 69.5	68.3 / 64.2 / 72.3	61.7 / 57.6 / 65.9	+3.1 / +3.7 / +2.5
LVOSv2-val	\(\mathcal{J}\&\mathcal{F}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	76.6 / 73.3 / 80.0	74.6 / 70.9 / 78.3	70.1 / 66.7 / 73.5	72.6 / 69.0 / 76.2	+2.0 / +2.4 / +1.7
LVOSv1-val	\(\mathcal{J}\&\mathcal{F}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	73.6 / 69.0 / 78.3	71.3 / 66.5 / 76.1	66.0 / 61.3 / 70.6	62.6 / 57.4 / 67.8	+2.3 / +2.5 / +2.2
SA-V val	\(\mathcal{J}\&\mathcal{F}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	66.1 / 62.9 / 69.4	65.6 / 62.0 / 69.3	60.7 / 57.7 / 63.7	61.4 / 56.6 / 66.2	+0.5 / +0.9 / +0.1
SA-V test	\(\mathcal{J}\&\mathcal{F}\) / \(\mathcal{J}\) / \(\mathcal{F}\)	66.4 / 63.0 / 69.8	66.1 / 62.4 / 69.8	62.7 / 59.7 / 65.7	61.8 / 57.2 / 66.3	+0.3 / +0.6 / 0.0

Scalability evaluation on a synthetic multi-target benchmark (1 to 20 targets, 1024p resolution, single NVIDIA A6000 GPU):

Target Count	SAM-MT (1024p) FPS	SAM2.1-B+ (1024p) FPS	Cutie-base (480p) FPS	DeAOT-L (1.3×480p) FPS	SAM-MT VRAM (MB)	SAM2.1-B+ VRAM (MB)
1 target	37.2	37.2	31.2	24.7	3094	3043
3 targets	36.8	17.8	30.4	24.7	3247	3356
5 targets	36.5	12.4	27.4	24.2	3312	3702
7 targets	36.4	9.5	23.4	24.2	3357	4089
10 targets	36.3 (9 obj)	7.8 (9 obj)	21.8 (9 obj)	24.1 (9 obj)	3491 (11 obj)	4529 (11 obj)
15 targets	35.7	4.9	18.4	14.0	3627	5436
20 targets	35.4	3.7	15.0	13.7	3785	8585

Ablation Study¶

Ablation experiments conducted on the MOSEv2-val benchmark isolate the contribution of key components:

Variant / Configuration	\(\mathcal{J}\&\dot{\mathcal{F}}\)	\(\mathcal{J}\)	\(\dot{\mathcal{F}}\)	FPS	Note / Empirical Observation
Full Model (SAM-MT)	43.0	41.4	44.5	36.9	Decoupled attention, 3-layer identity transformer, window size 16
Attention: Full Visibility	37.5	35.8	39.1	37.1	Unrestricted cross-target attention destroys identity specificity (-5.5 pts)
Attention: Full Masking	39.3	37.7	40.8	36.9	Isolating global context starves queries of scene awareness (-3.7 pts)
Identity Transformer: w/o Masking	39.1	37.4	40.7	37.1	Unconstrained memory attention causes cross-target history corruption (-3.9 pts)
Identity Transformer Depth: 1	41.9	40.3	43.4	37.2	Insufficient depth for historical temporal refinement
Identity Transformer Depth: 5	43.3	41.6	45.0	36.4	+0.3 pts accuracy gain at the expense of lower FPS; depth 3 selected
Memory Window Size: 8	42.4	40.8	44.0	37.1	Shorter window degrades long-term re-identification
Memory Window Size: 32	43.1	41.6	44.6	36.6	Minor +0.1 gain over 16 with additional compute overhead; 16 chosen
Training: w/o Image Pretrain	40.3	-	-	-	Lacks single-frame interaction tuning, dropping 2.7 pts
Training: w/o Strided Sampling	41.7	-	-	-	Consecutive-only sampling fails on target reappearance (-1.3 pts)
Training: w/o Overlap Loss	42.5	-	-	-	Boundary ambiguity rises in crowded scenarios (-0.5 pts)

Key Findings¶

Latency Decoupling: Scaling target count from 1 to 20 under 1024p resolution causes SAM-MT's frame rate to decline by less than 5% (from 37.2 to 35.4 FPS) and VRAM to grow by only ~691 MB. In contrast, SAM2.1-B+ suffers a 90% FPS collapse (37.2 to 3.7 FPS) and a nearly threefold surge in memory usage (3043 to 8585 MB).
Necessity of Asymmetric Masking: Decoupled attention provides the critical bridge between identity retention and global context awareness. Fully opening attention causes a 5.5-point drop, while completely blocking global queries causes a 3.7-point drop.
Superiority on Long Sequences: On long-term tracking datasets (LVOSv2 and LVOSv1), SAM-MT outperforms SAM2.1-B+ by 2.0 and 2.3 J&F points, proving that lightweight query-based sparse memory excels at mitigating identity drift over extended horizons.

Highlights & Insights¶

Dual-Track Memory Design: By maintaining a single shared dense memory for global context alongside compact 1-token-per-object sparse memories for temporal trajectories, SAM-MT breaks the fundamental computational barrier that has limited multi-object video segmentation.
Structured Masking in a Single Attention Pass: Rather than cascading separate network modules, SAM-MT achieves both cross-target identity isolation and global context injection within a single Transformer self-attention block via asymmetric masking.
Transferable Multi-Target Tracking Pipeline: The combination of prompt query consolidation, decoupled masked attention, and identity-aware sparse memory can be readily applied to interactive 3D scene tracking, multi-arm robotic manipulation, and multi-drone aerial surveillance.

Limitations & Future Work¶

Vision-Only Architecture: Inheriting the SAM backbone, SAM-MT operates purely on visual matching and point prompts without high-level linguistic or reasoning modules, making it unable to handle referring segmentation from complex natural language commands.
Ambiguity Under Dense Near-Identical Targets: In extreme visual ambiguity where numerous identical items (such as swarming insects or indistinguishable balls) frequently cross paths, single-token representations may face representational bottlenecks.
Future Directions: Integrating multimodal foundation models (MLLMs) to provide reasoning guidance and dynamically scaling query capacity per object during severe occlusions are promising avenues.

vs SAM2 (Ravi et al., 2024): SAM2 handles multiple objects by sequentially duplicating the single-target pipeline and maintaining massive per-object dense memories, leading to unbounded latency degradation; SAM-MT achieves a 6x speedup at 10 targets while improving segmentation accuracy.
vs Cutie (Cheng et al., 2023) / XMem (Cheng et al., 2022): Cutie and XMem share backbone features but still rely on independent mask decoders and fixed generic query slots (e.g., 16 queries); SAM-MT introduces fully scalable, identity-aware target queries with a single shared decoder.
vs DeAOT (Yang et al., 2022): DeAOT applies hierarchical decoupled attention but operates at lower baseline frame rates (~13-25 FPS at 480p) and drops sharply past 10 targets; SAM-MT maintains 36+ FPS at native 1024p across 1 to 20 targets.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Architecturally resolves SAM's multi-target bottleneck via elegant asymmetric masking and dual-track memory.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 6 VOS benchmarks, synthetic multi-target scalability stress tests, and granular ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, insightful technical motivation, and transparent reporting of compute trade-offs.
Value: ⭐⭐⭐⭐⭐ Practical and essential contribution that enables real-time, low-memory interactive multi-target video segmentation for production systems.