X2SAM: Any Segmentation in Images and Videos¶

Conference: ECCV 2026
Paper: ECCV page
Code: https://github.com/wanghao9610/X2SAM
Area: Semantic Segmentation
Keywords: unified segmentation, video object segmentation, visual prompting, mask memory, multimodal large language model

TL;DR¶

X2SAM uses Qwen3-VL as the reasoning core and the SAM2 mask encoder for fine-grained features, adding a language-conditioned Mask Memory to extend image-style any-segmentation to video; one conversational interface covers 14 image and video segmentation tasks under both textual and visual prompts, achieving 69.9 J&F on ReVOS video reasoning segmentation, 75.8 mIoU on video GCG, and cutting joint-training cost by about 36.5%.

Background & Motivation¶

Two communities in segmentation have long worked in parallel. SAM and SAM2 produce high-quality masks and propagate them across frames, but accept only low-level visual prompts (points and boxes) and cannot understand a conversational instruction like "segment the puppy on the left drinking milk from its mother." Conversely, image segmentation MLLMs such as LISA follow complex textual instructions to produce masks but remain confined to static images, with little support for specifying targets through visual prompts; VISA and VideoLISA extend text-to-mask to video, yet never unify images and videos in one architecture, nor provide a shared visual-prompt interface. The capability table makes this concrete: X-SAM covers 7 image segmentation tasks but 0 video tasks, VISA only 2+2, and no prior model covers 7+7.

Merging these lines raises three concrete challenges. First, prompt integration: the LLM must consume interleaved textual instructions and visual prompts (points/boxes) for both image and video inputs. Second, task formulation: generic, referring, reasoning, GCG, and interactive image-segmentation paradigms each come with their own conventions, so a shared representation is needed for "targets that persist and change over time in video." Third, and most critical, temporal coherence: frame-by-frame independent decoding has no systematic place to store language-guided visual features that have already been confirmed, so masks drift between frames.

The paper splits the segmentation MLLM into "understanding" and "memory-equipped decoding": the LLM unifies all instructions into conditional states, while a newly introduced Mask Memory stores semantically guided visual features for subsequent frames to read. Core idea: textual prompts, visual prompts, and LLM-generated tokens are all converted into mask-aware conditions, and the mask decoder decodes frame by frame over a language-conditioned memory cache, coupling semantic grounding and temporal propagation within the same forward pass rather than cascading image segmentation with SAM2-style propagation.

Method¶

Overall Architecture¶

X2SAM takes a natural-language instruction or a visual prompt (point/box), together with an image or a video, and outputs a language response plus per-frame binary masks. The visual side is dual-branch: the Qwen3-VL vision encoder extracts global representations, while the SAM2 mask encoder extracts high-resolution fine-grained features frame by frame. When a visual prompt is present, a parameter-free Region Sampler extracts region embeddings from the fine-grained features; these, together with global features and text embeddings, feed the LLM. The LLM autoregressively generates the response and produces a hidden embedding for each <SEG> token, projected by the MLLM projector into the prompt-token embedding \(Z_p\). The mask decoder combines \(Z_p\), learnable mask queries, and temporally refined features from the Mask Memory to decode masks frame by frame; the Mask Memory keeps a FIFO cache of "guided vision features" for cross-frame consistency. Training has two stages: class-agnostic segmentor training on SA-1B gives the decoder a stable initialization, followed by unified joint training over the 14 segmentation tasks plus image and video chat data.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image / video input<br/>text instruction or visual prompt"] --> B["Dual-branch visual encoding<br/>Qwen3-VL global branch<br/>SAM2 fine-grained branch"]
    B --> C["Parameter-free Region Sampler<br/>point sampling + adaptive pooling"]
    C --> D["Unified task formulation<br/>LLM emits response and SEG-token embedding"]
    D --> E["Redesigned Mask Decoder<br/>zero-initialized T2I attention"]
    E --> F["Per-frame mask sequence output"]
    E -->|"current-frame features<br/>+ mask logits"| G["Mask Memory<br/>language-conditioned FIFO cache"]
    G -->|"Memory Attention<br/>temporally refined features"| E

The flow mirrors the paper's Fig. 3/4: each decoded frame writes "visual features + mask logits" into memory, and the next frame's Memory Attention reads historical guided features, forming a per-frame loop; an image input (T=1) traverses the same path with an empty memory.

Key Designs¶

1. Unified task formulation: folding 14 segmentation tasks into one conversational interface

Segmentation MLLMs have fragmented because each task carries its own output convention: referring emits a single-target mask, GCG interleaves masks inside a description, interactive segmentation iterates with clicks and boxes. X2SAM extends the notation of X-SAM: <p> and </p> delimit the object condition state (a textual mention or a region encoded from a visual prompt), <SEG> marks where the mask is generated, and the LLM's output representation at <SEG> serves as the dedicated directive for mask decoding, with a set of task templates aligning response formats. Under this formulation, seven image tasks (generic, open-vocabulary, referring, reasoning, GCG, interactive, visual grounded) and their video counterparts (14 in total) all reduce to one pattern: "condition state + context → response + mask," and adding a task means adding a template, not changing the architecture. Visual prompts stop being a SAM-style side input: after Region-Sampler encoding they participate in language modeling as first-class tokens in the sequence.

2. Dual-branch visual encoding and a parameter-free Region Sampler: letting visual prompts enter the LLM like text

A single visual feature can hardly serve both "global semantic understanding" and "pixel-level mask prediction," so X2SAM bolts together two strong encoders: the Qwen3-VL branch patches images/videos (videos augmented with timestamps) into global representations, while the SAM2 mask encoder extracts high-resolution fine-grained features per frame. The Region Sampler is the small bridge: it point-samples regions of interest on the mask encoder's high-resolution features and applies adaptive pooling (kernel size 4) to aggregate sampled points into region-level representations, which enter the LLM as visual-prompt tokens. Having no learnable parameters, this injection adds no training burden and avoids downgrading interaction into preprocessing (as with segmentation-model-then-average pipelines) — semantic integration of the visual prompt is left entirely to the LLM's attention.

3. Redesigned Mask Decoder: dual attention for semantic injection, zero-initialized for stable early training

SAM2's native decoder targets single-object promptable segmentation and hinders parallel multi-mask generation, so X2SAM discards it and follows X-SAM: the decoder has two structured attention modules, Query-to-Image Attention and Token-to-Image Attention, letting the LLM's semantic token embedding \(Z_p\) interact directly with spatial features while mask queries decode in parallel. A key training detail: Token-to-Image Attention parameters are zero-initialized, so the newly injected conditional branch barely perturbs the trunk prediction early in training while the semantic pathway gradually takes effect. The ablation shows this is not fastidiousness — random initialization underperforms the no-module baseline on image tasks (RefCOCO cIoU 82.9 to 82.5), whereas zero initialization surpasses the baseline across the board, lifting V-Ref YT21 from 53.6 to 60.8 J&F.

4. Language-conditioned Mask Memory: writing mask guidance into the temporal cache

Frame-independent decoding necessarily drifts on video, and SAM2's memory serves visual prompts only, blind to language conditions. X2SAM's Mask Memory is a four-step loop: Memory Attention attends to guided vision features of previous frames and produces temporally refined features for the current frame; the mask decoder decodes the current-frame mask and logits from those refined features plus the LLM's SEG token; the Memory Encoder encodes downsampled visual features and the current-frame mask logits into new guided vision features; the Memory Bank stores per-frame guided features with a FIFO policy. The crucial point is that the memory holds not only appearance features but also the current-frame mask logits — the ablation shows this is the watershed between "ineffective" and "effective": a single-scale pure-visual memory moves V-Rea only from 36.5 to 37.6 J&F while slightly degrading V-Gen/V-Ref; adding mask guidance jumps V-Ref YT21 from 53.6 to 63.3, and stacking class guidance and the multi-scale design reaches 65.0. Because semantic conditions flow through the decoder into memory, temporal propagation and semantic grounding are no longer two separate stages — the essential difference from the "frame-wise X-SAM plus cascaded SAM2 propagation" route.

Loss & Training¶

Training has two stages. Stage one, class-agnostic segmentor training: the SAM2 mask encoder is frozen and only the mask decoder trains on SA-1B masks, letting it absorb class-independent shape and boundary priors before meeting multimodal instructions; effective batch size 128, learning rate \(1\times10^{-4}\). The mask loss combines BCE and Dice:

\[\mathcal{L}_{\text{mask}} = \lambda_{\text{bce}}\mathcal{L}_{\text{bce}} + \lambda_{\text{dice}}\mathcal{L}_{\text{dice}}, \quad \lambda_{\text{bce}} = \lambda_{\text{dice}} = 5.0\]

Stage two, unified joint training, optimizes the projectors, the LLM's LoRA parameters, the mask encoder, the mask decoder, and the Mask Memory jointly, switching objectives by sample type: chat samples use only the autoregressive language loss, while segmentation samples add the mask and focal classification losses:

\[\mathcal{L}_{\text{joint}} = \begin{cases} \mathcal{L}_{\text{ar}} & \text{image/video chat} \\ \mathcal{L}_{\text{ar}} + \mathcal{L}_{\text{mask}} + \mathcal{L}_{\text{cls}} & \text{image/video segmentation} \end{cases}\]

Image and video samples differ drastically in temporal length and memory footprint, so joint training relies on three engineering measures. Dimension shifting transposes \(B\times T\times H\times W\times 3\) into \(T\times B\times H\times W\times 3\) and splits it into per-frame tensors, letting video frames reuse the image-level encoding interface while temporal dependencies enter only through the Mask Memory during sequential decoding. Modality-aware batching sets the per-device video batch to 1 and expands image batches with a multiplier; image batches update gradients every step while video gradients accumulate over multiple steps. A temporal-aware sampler groups video clips of the same length into one batch to reduce padding. The mask encoder uses learning rate \(1\times10^{-5}\) and other modules \(1\times10^{-4}\), effective batch 32 for video / 128 for images, AdamW with weight decay 0.05, and dataset-balanced resampling with temperature \(t=0.1\). Main experiments train one epoch on 32 H800 GPUs; ablations use 16 GPUs.

Key Experimental Results¶

Main Results¶

The 14 tasks span image and video sides; the training corpus includes SA-1B (segmentor stage), VIPSeg/VSPW/YT-VIS19 (video generic), YT-RefVOS (video referring), ReVOS (video reasoning), VideoGLaMM (video GCG), and YT-VOS19/DAVIS17 (VOS), plus two V-VGD datasets derived from YT-VIS19 and VIPSeg; the image side follows X-SAM's mixed fine-tuning configuration, and video chat uses VideoInstruct100K. The table below condenses the most informative rows from Tables 6/7/8/9 of the paper (baselines are the strongest reported methods on each benchmark):

Benchmark	Metric	X2SAM	Prior strong baseline	Δ
V-Rea. ReVOS	J&F Overall	69.9	55.7 (HyperSeg)	+14.2
V-GCG VideoGLaMM set	mIoU	75.8	54.3 (VideoGLaMM)	+21.5
V-Ref. YT21 / DV17	J&F	78.5 / 79.0	71.0 / 76.4 (UniPixel-7B)	+7.5 / +2.6
V-VGD YT-VIS19 / VIPSeg (Box)	AP	74.4 / 57.8	54.0 / 40.4 (SAM2-H)	+20.4 / +17.4
V-OV YT-VIS-21 (OOD)	AP / AP50	60.3 / 78.0	53.8 / – (HyperSeg)	+6.5
I-Rea. ReasonSeg Val	cIoU / gIoU	64.5 / 71.1	56.7 / 59.2 (HyperSeg)	+7.8 / +11.9
I-OV ADE20K (OOD)	PQ / mIoU / mAP	31.2 / 38.2 / 20.2	22.6 / 29.9 / 14.4 (ODISE)	+8.6 / +8.3 / +5.8
I-Ref. RefCOCO/+/g	cIoU	84.0 / 78.4 / 81.9	85.1 / 78.0 / 83.8 (X-SAM)	-1.1 / +0.4 / -1.9

Video is X2SAM's main battlefield: referring, reasoning, GCG, and V-VGD all surpass existing MLLM-based video generalists, and V-VGD even beats the SAM2-H specialist by a wide margin. The image side involves trade-offs: open-vocabulary and reasoning segmentation lead clearly, while generic, GCG, and VGD segmentation trade wins and losses with the image-specialized X-SAM; a 14.8-point gap to SAM2-H (88.8 J&F) remains on VOS, which the authors attribute to the cost of a general interface. If any prose value conflicts with the tables, the original paper prevails.

Ablation Study¶

Token-to-Image (T2I) attention in the mask decoder and its initialization strategy (paper's Table 2; ablations run on 16 GPUs; the paper's I-Rea. columns are labeled only Val/Test without specifying the metric — refer to the original):

Config	I-Ref. RefCOCO/+/g cIoU	I-Rea. Val/Test	V-Ref. YT21 J&F	V-Rea. All J&F
Baseline (no T2I, no memory)	82.9 / 78.0 / 79.5	58.6 / 55.9	53.6	36.5
+ T2I (random init)	82.5 / 77.2 / 79.4	62.4 / 56.0	58.2	44.9
+ T2I (zero init)	83.3 / 77.8 / 79.5	63.0 / 56.8	60.8	47.3

Internal design of the Mask Memory, ablated step by step (paper's Table 4):

Config	V-Gen. VPQ/mIoU/mAP	V-Ref. YT21/DV17 J&F	V-Rea. All J&F
Baseline (no memory)	42.9 / 61.1 / 66.3	53.6 / 41.1	36.5
+ Single-scale memory	42.7 / 60.8 / 66.1	52.5 / 41.9	37.6
+ Mask guidance	44.5 / 62.3 / 66.7	63.3 / 49.4	51.1
+ Class guidance	44.8 / 61.9 / 66.8	64.6 / 48.3	51.6
+ Multi-scale (full)	45.0 / 62.5 / 66.6	65.0 / 49.6	53.5

Key Findings¶

Memory must mix in mask information; pure visual memory barely helps. A single-scale memory moves tasks up and down (V-Gen 42.9 to 42.7), but adding mask guidance jumps V-Ref YT21 from 53.6 to 63.3 — cross-frame alignment truly needs the semantic cue of "where the previous frame believed the target was," not raw appearance features.
Zero initialization is the key trick for injecting a new conditional branch. Randomly initialized T2I attention loses points on image tasks (RefCOCO 82.9 to 82.5), while zero initialization is best across the board: the same module, with initialization deciding whether it helps or disturbs.
Unified joint training saves cost without sacrificing quality. Against separate image/video training, unified joint training raises V-OV mAP from 57.1 to 59.1 and V-Gen mIoU from 61.1 to 64.7; against naive joint training, it cuts training cost from about 5.2K to 3.3K GPU hours (-36.5%) while matching or slightly improving performance.
Memory capacity is not the larger the better. Growing capacity from 1 to 6 lifts V-OV mAP to 60.2 and V-Rea All to 57.5; growing it to 8 drops V-Ref to 53.7 and V-Rea to 53.5 (while V-Gen VPQ rises to 45.0). The authors adopt 6 frames, yet the implementation-details section says "memory capacity is fixed at 8 frames" — the two statements are inconsistent, and the capacity actually used in main experiments should follow the original paper.
Generality has a price. On V-Obj, X2SAM (74.0 J&F) clearly trails the VOS specialist SAM2-H (88.8); on the image side, I-Gen PQ and I-GCG mIoU also sit slightly below X-SAM — the unified interface buys coverage at the cost of narrow tasks.

Highlights & Insights¶

The unified formulation — "object = conditional state, instruction = context, = decoding directive" — is the fulcrum of the paper. Once 14 tasks share one token convention, textual and visual prompts become peers inside the LLM sequence, and new segmentation paradigms need only new templates. This is the most reusable architectural abstraction here.
Language-conditioned memory design. SAM2's memory stores visual features only; X2SAM also encodes "mask logits decoded under LLM conditions" into memory, letting temporal propagation carry semantic hypotheses — which explains the V-Ref jump from 53.6 to 63.3 in the mask-guidance ablation. (The paper calls the Mask Memory language-conditioned; this note summarizes per the ablation semantics.)
Zero-initialization, a ControlNet-style practice, works equally well for attention injection in a mask decoder. Any scenario that adds a new attention branch to a pretrained model should first ask: can it be zero-initialized?
Dimension shifting plus modality-aware batching is a general engineering recipe for mixing images and videos. Transposing \(B\times T\) into \(T\times B\) to reuse the image interface per frame, then balancing the two modalities' memory footprints with different batch sizes and gradient accumulation — directly reusable in any multimodal project that co-trains on images and videos.

Limitations & Future Work¶

Limitations admitted by the authors: unified training remains computationally expensive, especially for video samples with high memory cost; the fixed-size FIFO memory is insufficient for long videos with prolonged occlusions, large appearance changes, or sparse target reappearance; as a generalist, X2SAM still lags specialized models on narrow tasks such as optimized VOS or image-only segmentation.
Limitations spotted by this note: the V-VGD benchmark is self-built and self-evaluated (derived from YT-VIS19/VIPSeg) without third-party cross-validation; the I-Gen/I-Ref/I-GCG comparisons against X-SAM are comparisons against "the same authors' prior work," so image-side superiority claims deserve discounting; the 8/6-frame memory-capacity inconsistency suggests loose experiment bookkeeping.
Improvement directions: an adaptive, on-demand memory (proposed by the authors themselves); extending the unified instruction interface to streaming and interactive segmentation; lighter backbones and more efficient video training strategies.

vs X-SAM: X-SAM is image-centric any-segmentation (7 image tasks, no video, no temporal object identity); X2SAM extends the same <p>/<SEG> formulation to video and adds the Mask Memory. The authors stress this is not an X-SAM+SAM2 cascade — textual/visual prompts and SEG tokens all become mask-aware conditions, the memory is language-conditioned, and grounding, decoding, and memory are jointly optimized.
vs SAM2: SAM2's memory propagation is strong, but it takes only point/box prompts, with no language reasoning or grounded conversation; X2SAM bridges semantics through the LLM, letting the dialogue decide what to segment.
vs VISA / VideoLISA / UniPixel-7B: These video segmentation MLLMs support temporal text-to-mask but lack a unified image+video architecture and visual-prompt interface; X2SAM dominates V-Ref/V-Rea (e.g., ReVOS 69.9 vs UniPixel's 63.7).
vs Sa2VA / PSALM / HyperSeg: Sa2VA supports image and video V-Prompts but covers only 3+3 tasks; PSALM supports visual prompts but has weak video capability and no video chat. X2SAM is the only model in Table 1 with a full 7+7 plus image and video chat.
Inspiration: the "language-conditioned memory" idea transfers to other temporally consistent dense-prediction tasks such as video editing and video depth estimation; the unified formulation plus task templates is a ready-made recipe for any work stuffing multiple output heads (detection, tracking, depth) into one MLLM.

Rating¶

Novelty: 4/5. The combination — unified 14-task image/video coverage, a visual-prompt interface, and a language-conditioned memory — is new, but the core components are respectively inherited from X-SAM, SAM2, and Qwen3-VL; high-quality system integration rather than a single-point invention.
Experimental Thoroughness: 5/5. Fourteen tasks, nine main tables, OOD coverage plus a self-built V-VGD benchmark, and four interlocking ablations (decoder initialization, joint training, memory design, memory capacity) — among the most complete ablations in this direction.
Writing Quality: 4/5. Structure and figures are clear, but the memory-capacity 8/6-frame contradiction and ambiguous metric labels in one ablation table cost a star.
Value: 4/5. Provides a reproducible strong baseline with open-source code for "unified pixel-level spatio-temporal understanding," with significant video-side gains; the image-side trade-off against specialists quantifies the cost of generalization.