SAM2Matting: Generalized Image and Video Matting¶

Conference: ECCV 2026
arXiv: 2606.27339
Project: https://henghuiding.com/SAM2Matting
Code: https://github.com/FudanCVL/SAM2Matting
Area: Image Segmentation / Video Understanding
Keywords: Video Matting, Image Matting, SAM2, Decoupled Architecture, ROI Detection

TL;DR¶

SAM2Matting proposes a decoupled "tracker-to-matting" framework that divides video matting into high-level tracking (where a frozen VOS tracker like SAM2/SAM3 ensures temporal consistency) and low-level matting (where a trainable ROI detector + progressive Alpha predictor estimates fine-grained transparency). Trained solely on image matting data, it achieves SOTA in a zero-shot manner on video matting benchmarks. It also supports multiple prompt types, maintains strong temporal consistency, and generalizes well to both human and open-world scenarios.

Background & Motivation¶

Image matting aims to predict pixel-level alpha masks to separate foreground from background, which is a fundamental low-level vision task. When extending this task to video, an explicit target specification (such as a first-frame mask) is typically required to resolve ambiguity and achieve consistent tracking across frames. Therefore, video matting faces a fundamental trade-off: it requires both high-level semantic understanding to robustly track targets similar to video object segmentation (VOS), and low-level fine-grained perception to capture extremely fine details like image matting.

To bridge this gap, existing methods heavily rely on video matting datasets for training or fine-tuning. However, annotating pixel-level alpha values across video frames is extremely expensive, resulting in datasets of limited scale and narrow domains—mainly focusing on human matting scenarios, which are insufficient to represent rich real-world dynamics. Training from scratch on such constrained data fails to establish robust tracking capabilities, while fine-tuning pre-trained VOS models on them degrades their original tracking robustness.

Key Challenge: High-level tracking requires large-scale, diverse video segmentation data to ensure generalization, while low-level matting requires pixel-level alpha annotations for precision. Video matting datasets that satisfy both simultaneously are practically non-existent. This paper revisits this paradigm and argues that video matting is essentially a combination of two independent sub-tasks—high-level tracking (already well-resolved by large-scale pre-trained VOS models) and low-level matting (fully covered by diverse image matting datasets). Core Idea: Decouple the two sub-tasks, freeze the VOS tracker to preserve its tracking robustness, and only train specialized matting components using rich image matting data, thereby achieving highly generalizable video matting without relying on expensive video matting annotations.

Method¶

Overall Architecture¶

The core idea of SAM2Matting is to decouple video matting into two independent parts: high-level tracking and low-level matting. A frozen VOS tracker (SAM2.1-Tiny / SAM2.1-Base+ / SAM3) is responsible for outputting temporally consistent target masks for each frame. A set of trainable matting components receives this mask alongside multi-scale image features, first identifying "key regions" requiring fine-grained matting via an ROI detector, and then progressively generating and refining the alpha mask from coarse to fine using a Progressive Alpha Predictor. The entire matting module is trained solely on image matting data with the tracker frozen. Thus, when generalized to videos in a zero-shot manner, it naturally inherits the temporal consistency of the tracker.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Frame + VOS Mask"] --> B["ROI Detector<br/>Multi-scale Feature Fusion to Predict Key Regions"]
    B --> C["Pseudo-Trimap Generation<br/>Mask and ROI Synthesis into Trimap"]
    C --> D["Progressive Alpha Predictor<br/>Multi-scale Cascade Coarse-to-Fine Refinement"]
    D --> E["Final Alpha Mask"]

Key Designs¶

1. ROI Detector: Replacing Rule-Based Morphological Operations with Learnable Detection

Traditional matting methods typically use morphological dilation/erosion on masks to generate "unknown regions" (ROIs). This implicitly assumes uniform boundary importance, which is prone to missing fine details inside complex structures or containing definite foreground regions that do not require matting. Other methods directly use the raw mask as the ROI, which is similarly coarse.

The ROI Detector reformulates ROI detection as a pixel-level binary classification task, where positive pixels represent key regions requiring fine-grained matting (containing fine details or semi-transparent areas). Specifically, for the \(t\)-th frame, at each scale \(i\), a scale-specific convolutional head \(f_{R,i}\) takes image features \(F_{t,i}\), the scaled frame \(I_{t,i}\), and the scaled mask \(M_{t,i}\) as input to predict the ROI logit map \(L_{t,i}\):

\[L_{t,i} = f_{R,i}(F_{t,i}, M_{t,i}, I_{t,i})\]

The logit maps from different scales are upsampled and concatenated, and then fused by a hierarchical convolutional network \(f_{\varphi}\) to capture global context and structural details, yielding the final logit map \(L_t\). Applying sigmoid to \(L_t\) and binarizing it with a threshold \(\theta=0.65\) yields the ROI prediction \(\mathcal{R}_t\). During training, focal loss is used for pixel-level classification supervision, and smooth-L1 loss is applied to suppress aliasing artifacts. Compared to morphological operations, the learnable ROI Detector adaptively identifies the regions genuinely requiring fine-grained matting (such as wisps of hair, tree leaf gaps, arm gaps) in each frame based on the image content, without missing details or erroneously expanding the scope.

2. Progressive Alpha Predictor: Multi-Scale Cascade Coarse-to-Fine Iterative Refinement

Unlike the parallel multi-scale processing of the ROI Detector, the Alpha Predictor treats alpha estimation as a sequential refinement process. It employs a coarse-to-fine cascade strategy: the alpha prediction from each intermediate scale serves as guidance forwarded to the next finer scale.

At scale \(i\) of the \(t\)-th frame, the composite input \(X_{t,i}\) concatenates the image features \(F_{t,i}\), the scaled pseudo-trimap \(\mathcal{T}_{t,i}\), the scaled frame \(I_{t,i}\), and the upsampled mask \(\mathcal{A}_{t,i-1}\) from the previous scale (except for the first scale). A scale-specific projection layer \(g_{\mathcal{A},i}\) first maps \(X_{t,i}\) to a fixed-dimensional embedding, and then the matting head \(f_{\mathcal{A},i}\) predicts the alpha mask \(\mathcal{A}_{t,i}\) for this scale:

\[\mathcal{A}_{t,i} = \sigma(f_{\mathcal{A},i}(g_{\mathcal{A},i}(X_{t,i})))\]

By default, 3 scales are used, and the output of the finest scale is upsampled back to the original resolution to obtain the final alpha mask. This autoregressive cross-scale information passing enables the model to capture global structure at coarse scales and progressively recover fine details like hair strands at finer scales, yielding significantly higher quality than single-step prediction. The pseudo-trimap is synthesized from the ROI prediction and the VOS mask: pixels where ROI=0 take the mask value (definite foreground/background), while pixels where ROI=1 are labeled as unknown regions (0.5), providing a structured pixel-level spatial prior for the Alpha Predictor.

3. Training Strategy and Loss Design: Frozen Tracker + Image-Only Training + Multi-Layer Supervision

During training, the VOS tracker is completely frozen, and only the matting components are optimized. This ensures that the temporal consistency of the tracker is not corrupted by the domain shift of the matting data. All training is conducted solely on 8 image matting datasets (I-HIM50K, P3M-10k, CelebAHairMask-HQ, AIM-500, Distinctions-646, AM-2K, UHRIM, RefMatte), without using any video matting data.

The supervision signal \(\mathcal{R}_t^{GT}\) for the ROI Detector is obtained by thresholding the ground-truth alpha mask (\(\alpha=0.15, \beta=0.5\)) and applying dilation minus erosion. It is optimized jointly using focal loss \(\mathcal{L}_{focal}\) and smooth-L1 loss \(\mathcal{L}_{sm}\). Deep supervision is applied to the Alpha Predictor across all scales, with L1 loss and Laplacian loss constraints on each scale:

\[\mathcal{L}_{alpha} = \sum_{i=1}^{n} \lambda_i (\mathcal{L}_{L1}(\mathcal{A}_{t,i}, \mathcal{A}_{t,i}^{GT}) + \mathcal{L}_{lap}(\mathcal{A}_{t,i}, \mathcal{A}_{t,i}^{GT}))\]

The loss weights increase with scale (\(\lambda_1=0.3, \lambda_2=0.6, \lambda_3=1.2\)) to encourage higher accuracy at finer scales. Furthermore, a matte-mask consistency penalty \(\mathcal{L}_{con}\) (joint focal + dice segmentation loss) is introduced to anchor the alpha mask to the VOS mask, preventing internal holes in the foreground. The total loss is defined as \(\mathcal{L} = \mathcal{L}_{\mathcal{R}} + \mathcal{L}_{\mathcal{A}}\).

Loss & Training¶

All variants are trained for 5 epochs on 4 NVIDIA A6000 GPUs with a batch size of 32 using the AdamW optimizer, with different learning rates for each variant. Hyperparameters are determined via grid search: ROI threshold \(\theta=0.65\), alpha thresholds \(\alpha=0.15, \beta=0.5\), and the loss weights for the 3 prediction scales are set to 0.3, 0.6, and 1.2, respectively. During inference, both the tracker and the matting components require no fine-tuning and are directly applied to videos in a zero-shot manner.

Key Experimental Results¶

Main Results: Image Matting¶

Dataset	Method	MAD↓	MSE↓	Grad↓	Conn↓	SAD↓
P3M-500-NP	MAM	15.40	9.20	14.22	-	25.82
P3M-500-NP	Matte Anything	-	2.80	17.30	10.00	10.70
P3M-500-NP	SAM2Matting (SAM2.1-T)	3.92	1.07	8.66	6.34	6.78
P3M-500-NP	SAM2Matting (SAM3)	3.83	0.97	8.48	5.84	6.61
AM-2K	MAM	10.10	3.50	10.65	-	17.30
AM-2K	SAM2Matting (SAM2.1-T)	4.57	1.39	7.02	7.22	7.88
PPM-100	MODNet	8.60	4.40	64.26	80.16	94.78
PPM-100	SAM2Matting (SAM2.1-T)	4.51	1.32	49.26	39.56	42.05

Three variants consistently outperform prior methods across three image matting benchmarks. The SAM2.1-Tiny variant reduces the MAD on P3M-500-NP by 11.48 compared to MAM, and reduces Conn on PPM-100 by approximately 40 compared to MODNet.

Main Results: Video Matting (Zero-Shot)¶

Dataset	Method	MAD↓	MSE↓	Grad↓	Conn↓	dtSSD↓
V-HIM60-Medium	MaGGIe (CVPR'24)	13.85	-	6.31	5.11	23.63
V-HIM60-Medium	MatAnyone2 (CVPR'26)	15.12	5.86	6.36	5.43	4.50
V-HIM60-Medium	SAM2Matting (SAM2.1-T)	13.76	4.61	7.78	5.01	4.23
V-HIM60-Medium	SAM2Matting (SAM3)	11.77	3.64	5.92	4.23	3.81
V-HIM60-Hard	MaGGIe (CVPR'24)	21.23	-	7.08	6.89	29.90
V-HIM60-Hard	MatAnyone2 (CVPR'26)	45.75	35.03	8.43	14.75	6.16
V-HIM60-Hard	SAM2Matting (SAM2.1-T)	18.58	8.79	8.03	6.16	5.37
V-HIM60-Hard	SAM2Matting (SAM3)	14.37	5.52	5.85	4.72	4.37
VideoMatte-SD	RVM (WACV'22)	6.08	1.47	0.88	0.41	1.36
VideoMatte-SD	MatAnyone2 (CVPR'26)	4.73	0.55	0.51	0.19	1.12
VideoMatte-SD	SAM2Matting (SAM3)	4.44	0.27	0.23	0.16	1.11

SAM2Matting outperforms SOTA methods trained on video data across all video benchmarks in a zero-shot manner, achieving the lowest dtSSD, which demonstrates extremely strong temporal consistency inherited from the frozen tracker. MatAnyone2 severely degrades on V-HIM60-Hard (MAD 45.75), whereas SAM2Matting remains stable, validating the robustness of the decoupled strategy in difficult scenarios.

Ablation Study¶

Experiment	Configuration	V-HIM60-Hard MAD↓	Grad↓	Conn↓	dtSSD↓
ROI Strategy	Morphological	29.82	11.57	10.37	7.48
ROI Strategy	Mask-only	20.07	9.11	6.68	5.50
ROI Strategy	ROI Detector	18.20	7.39	6.01	5.10
Architecture / Supervision	Prog. + Scaling only	19.43	7.88	6.35	5.30
Architecture / Supervision	+ Consistency Loss	18.65	7.70	6.20	5.18
Architecture / Supervision	+ Consistency + Smooth Loss	18.26	7.45	6.04	5.09
Architecture / Supervision	All (Full model)	18.20	7.39	6.01	5.10

Compared to morphological operations, the ROI Detector reduces MAD from 29.82 to 18.20 (approx. 39% reduction), demonstrating the critical role of learnable ROI in complex scenarios. The progressive multi-scale refinement alone makes a significant contribution, the matte-mask consistency loss fills foreground holes, and the smoothness loss eliminates jagged boundaries—overlapping all three yields complete, high-fidelity matting.

Control Experiment	Condition	P3M-500-NP MAD↓	AM-2K MAD↓
MAM	Original training data + original backbone	15.40	10.10
SAM2Matting	Same training data as MAM	4.05	5.38
Matte Anything	Original training data + original backbone	-	-
SAM2Matting	Same training data as Matte Anything	3.94	5.95
MAM	Replaced with SAM2.1-B+ backbone	12.92	8.64
Matte Anything	Replaced with SAM2.1-B+ backbone	6.00	6.21
SAM2Matting	SAM2.1-B+ backbone	3.81	4.90

Under a fair comparison controlling for training data and backbones, SAM2Matting still significantly outperforms MAM and Matte Anything, proving that the performance gains stem from the architecture and supervision designs themselves rather than larger data or stronger backbones.

Key Findings¶

ROI Detector is the largest single contributor: Replacing it with morphological operations causes the MAD to shoot up from 18.20 to 29.82, showing that learnable key region detection is a prerequisite for high-quality matting.
Fine-tuning on video matting datasets actually harms generalization: Fine-tuning on V-HIM2K5 slightly improves in-domain V-HIM60-Hard performance (MAD 18.20 \(\rightarrow\) 17.90) but degrades performance on AM-2K animal data (MAD 4.90 \(\rightarrow\) 5.23), alongside a noticeable drop in tracking robustness (tracking drift occurs even in simple unoccluded scenes)—indicating that narrow-domain video matting data leads to overfitting.
Robust to tracking inaccuracies: The ROI Detector treats the mask as one of multiple cues rather than a hard constraint, enabling it to recover foreground details missed by the tracker (such as ski poles) and remove incorrectly included background objects (such as tabletops), realizing a fault-tolerant mechanism of "tracker errors corrected by the matting components".
Extremely efficient: The SAM2.1-Tiny variant runs at 40 FPS on 1080p video with less than 4 GB of VRAM. Moreover, the FPS is almost invariant to resolution (720p to 2160p only drops from 40.46 to 40.04), whereas MatAnyone/MatAnyone2 drop below 3 FPS at 1440p and run OOM at 2160p.

Highlights & Insights¶

A paradigm of generalization through decoupling: Splitting video matting into two orthogonal sub-tasks, namely "tracking" and "matting", and addressing them separately using the strongest large-scale pre-trained schemes, is the most core methodological contribution of this paper. This paradigm of "freezing a general backbone + training a lightweight specialized head" can be transferred to any cross-granularity task that simultaneously requires high-level semantics and low-level details (e.g., boundary refinement in video instance segmentation, click response in interactive segmentation).
ROI Detector replacing morphology with binary classification: Replacing hand-crafted morphological operations with a learnable pixel-level classifier essentially answers "which pixels are worth refining" in a data-driven manner. This concept can be transferred to any vision task needing selective refinement (e.g., edge enhancement region selection in super-resolution, damaged region detection in image inpainting).
Achieving zero-shot video generalization trained solely on image data: This is the most elegant byproduct of the decoupled design—requiring absolutely no video annotations during training, while directly enjoying the tracking consistency of the tracker during inference. For fine-grained video tasks with extremely high annotation costs (e.g., video depth estimation, video optical flow refinement), this "image-trained + video zero-shot" strategy holds immense practical value.
Matte-mask consistency loss preventing foreground holes: This design is compact yet yields intuitive benefits: using the VOS mask as a structural anchor for the alpha mask ensures that the matted foreground does not exhibit "holes," serving as a simple and effective way to fuse cross-task supervision signals.

Limitations & Future Work¶

ROI Detector relies on the quality of the tracker mask: Although experiments show some robustness to tracking inaccuracies, if the tracker completely loses the target (e.g., the target reappears at another location after long-term complete occlusion), the ROI Detector cannot recover it out of thin air—it can only correct boundary errors within the mask, not reconstruct completely missing mask regions. Robustness under extreme occlusion scenarios still needs validation.
Inherent limitations of training solely on image data: While zero-shot generalization is an advantage, the model never sees temporal information, which may cause slight frame-to-frame inconsistencies in alpha prediction (although the dtSSD metric is superior, extremely subtle flickering is still qualitatively observable). Introducing unsupervised temporal consistency regularization could further improve results.
Lack of support for prompt-free automatic matting: The current framework requires explicit target specifications (masks/points/boxes/text), making it inapplicable for scenarios like "automatically matting all salient objects in a video." Extending the ROI Detector into a general saliency detector to enable prompt-free mode is a potential direction.
Multi-object matting is not fully explored: This paper focuses on single-target matting, but real-world applications often require matting multiple targets simultaneously (e.g., background replacement in multi-person video conferencing). Extending the framework to support multi-instance matting requires addressing ROI conflicts and instance-level alpha allocation.
Limited evaluation benchmarks: Video matting benchmarks are predominantly human-centric (V-HIM60, VideoMatte-SD), lacking sufficient quantitative evaluation on open-world scenarios (e.g., animals, vehicles, natural objects), although qualitative results demonstrate strong generalization.

vs MaGGIe (CVPR'24) / Mat Anyone (CVPR'25) / MatAnyone2 (CVPR'26): These methods are trained or fine-tuned on video matting data, suffering from limited generalization due to small data scales and domain biases (primarily human matting). SAM2Matting completely bypasses video matting data dependency using a decoupled strategy, performing better in both human and open-world scenarios. A disadvantage is that these end-to-end trained methods may exhibit slightly better temporal consistency in specific human matting scenarios.
vs Matte Anything (CVPR'24): Matte Anything also utilizes SAM-like models for matting, but employs morphological operations for ROI generation and directly uses the original masks for matting prediction. Under the same backbone, SAM2Matting's learnable ROI Detector and progressive refinement strategy outperform it significantly.
vs RVM (WACV'22): RVM is a classic video matting method that does not require explicit target specifications, but its lack of target disambiguation makes it prone to failure in real-world complex scenes. SAM2Matting introduces explicit target specification via the VOS tracker, making it more practical for real-world applications requiring precise target control.
vs VideoMAMA (CVPR'26): VideoMAMA synthesizes video matting data by expanding existing large-scale VOS benchmarks through pseudo-labeling. Although the data scale is larger, it still depends on pseudo-label quality, and direct comparisons with SAM2Matting are not yet available. The two ideas are complementary: VideoMAMA attempts to expand the scale of video matting data, while SAM2Matting argues that video matting data is fundamentally unneeded.

Rating¶

Novelty: ⭐⭐⭐⭐☆ The idea of decoupling tracking and matting is not inherently new (mask-guided matting works existed previously), but systematizing it into a complete framework of "frozen tracker + learnable ROI + progressive refinement + image-only trained zero-shot video generalization," and demonstrating the feasibility of completely bypassing video matting data, presents a clear and powerful methodological contribution.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covering 3 image benchmarks + 3 video benchmarks, comparing with 10+ baselines, and conducting extensive ablations covering ROI strategies, architectural/supervision designs, fair comparisons (controlling data/backbone), fine-tuning analysis, efficiency analysis, and qualitative comparisons make the experimental design extremely comprehensive.
Writing Quality: ⭐⭐⭐⭐☆ Well-structured with fully articulated motivations and detailed method descriptions, rich with figures and tables (14 figures + 7 tables). However, the definitions of some formula variables are scattered across different paragraphs, requiring back-and-forth reference while reading.
Value: ⭐⭐⭐⭐⭐ Offers significant paradigm inspiration for the video matting community by proving that decoupling strategies can eliminate reliance on expensive video matting annotations. Additionally, its real-time performance of 40 FPS and sub-5 GB VRAM footprint give it high engineering value for direct deployment in practical products (e.g., live background replacement, film post-production).