Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

Conference: ECCV 2026
Paper: ECCV Original
Code: None (No official public code repository provided in the paper)
Area: Segmentation
Keywords: Unsupervised Learning, Segment Anything, Perceptual Grouping, Contrastive Learning, Zero-Shot Segmentation

TL;DR

MoSA introduces a three-stage unsupervised framework that extracts multi-granularity motion pseudo-labels from unlabeled videos, internalizes a static appearance-driven objectness prior via contrastive perceptual grouping, and adapts to whole-image and interactive segment-anything architectures without any manual annotations.

Background & Motivation

Foundation segmentation models such as SAM have established a new paradigm for zero-shot generalization in computer vision, yet their empirical success relies on massive datasets with hundreds of millions of human-annotated masks (e.g., SA-1B). This dependence on costly manual annotations imposes a fundamental bottleneck for model scaling and impedes the construction of foundation models in annotation-scarce domains. While unsupervised segmentation seeks to bypass manual supervision, existing paradigms exhibit distinct shortcomings: image-based methods (such as CutLER and UnSAM) rely heavily on clustering static self-supervised features (e.g., DINO) without continuous temporal grounding, whereas prior video-based approaches overfit to salient moving objects, failing to comprehend multi-granularity structures (e.g., sub-parts) and struggling to generalize to static scene elements (e.g., background furniture or roof tiles).

The central tension arises from the intrinsic sparsity and locality of motion-derived pseudo-supervision: while motion offers a natural perceptual boundary separating objects from the background, moving regions only cover a minor fraction of any frame. Conventional dense segmentation objectives like Binary Cross-Entropy (BCE) or Dice loss severely penalize unlabeled, static foreground entities by wrongly treating them as background. Furthermore, motion signals extracted across consecutive frames are typically coarse or single-blob, falling short of capturing multi-level hierarchical representations spanning from fine-grained parts to whole objects.

Inspired by developmental psychology and Gestalt principles of common fate, humans naturally internalize universal concepts of objectness from dynamic environments and seamlessly extrapolate them to static entities. Core idea: decompose large-scale unlabeled videos into multi-granularity motion pseudo-labels, train a multi-head Perceptual Grouping Model via patch-level contrastive learning to internalize appearance-driven objectness without motion dependency, and transfer this prior to high-resolution interactive and whole-image segment-anything architectures.

Method

Overall Architecture

MoSA is structured into three progressive stages. In Stage 1, a Multi-Granularity Motion Segmentation (MGMS) pipeline is trained on synthetic video data and deployed over large-scale unlabeled videos to generate part-level and foreground-level pseudo-labels filtered by a dedicated quality score. In Stage 2, a ViT-based Perceptual Grouping Model (PGM) is trained using Perceptual Grouping Contrastive Learning (PGCL) with scale-specialized projection heads, compelling the model to group coherent visual regions into object representations without penalizing static regions. In Stage 3, the frozen PGM executes multi-scale sliding-window inference on unlabeled high-resolution static images, providing pseudo-masks to train a whole-image segmentation model (Mask2Former) and a promptable segmentation model (Semantic-SAM).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Large-Scale Unlabeled Videos<br/>Kinetics-700 / BDD100K / YouTube-8M"] --> B["Stage 1: Multi-Granularity Motion Pseudo-Labels Generation<br/>ROSM + MFSM Optical Flow Segmentation + Quality Filtering"]
    B --> C["Stage 2: Learning Objectness from Motion via Contrastive Grouping<br/>Multi-Scale Heads + Patch Anchor Coherence Contrast"]
    C --> D["Stage 3: Adaptation to Segment Anything<br/>Multi-Scale Tiling Inference + Mask2Former / Semantic-SAM"]
    D --> E["Output: Zero-Shot General-Purpose Segmentation<br/>Whole-Image Multi-Granularity Instances & Point Prompts"]

Key Designs

1. Multi-Granularity Motion Pseudo-Label Generation: Disentangling Rigid Parts from Moving Foreground Real-world videos encompass intricate camera motions, perspective changes, and non-rigid deformations, rendering it difficult for a single module to capture both fine-grained components and entire non-rigid objects. To address this multi-scale challenge, two complementary modules are trained on synthetic videos with precise ground truth: the Rigid Object Segmentation Module (ROSM) employs a SOLOv2 architecture over bidirectional optical flow to segment entities following perspective transformations (e.g., wheels, limbs, or rigid parts); the Motion Foreground Segmentation Module (MFSM) utilizes a Res-UNet to delineate all moving pixels in the frame, effectively capturing complex non-rigid bodies like walking humans. Generated candidate masks are then pruned via a composite quality score combining mean prediction confidence \(S_{\text{maskness}}\) and boundary gradient sharpness \(S_{\text{sharpness}}\) followed by NMS, yielding 21 million high-fidelity pseudo-labels from 10,000 video hours.

2. Perceptual Grouping Contrastive Learning: Overcoming Sparse Motion Supervision Penalties Because motion pseudo-labels are naturally sparse across static frames (e.g., an animal with only one moving paw detected), applying standard BCE or Dice losses incorrectly treats the rest of the body as negative background. Meanwhile, slot-centric architectures suffer from fixed slot capacity in open-world scenes and lack multi-granularity flexibility. To resolve this, Perceptual Grouping Contrastive Learning (PGCL) is introduced: \(K\) linear projection heads are attached atop a ViT backbone, each dedicated to a specific bounding box scale range. For an assigned pseudo-mask, an anchor patch embedding \(f_t^k\) is sampled within the mask, and its cosine similarity \(s_{t,j}^k\) against all image patch embeddings \(f_j^k\) from that head is calculated. The probability \(p_{t,j}^k\) that patch \(j\) belongs to the same object is modulated by a learnable temperature \(\tau_k\) and a sigmoid function. By exclusively constructing positive pairs within the mask and negative pairs across its boundary, PGCL drives intra-object feature compactness and inter-object separability, encouraging the network to discover and group unannotated static entities based on appearance coherence alone.

3. Adaptation to Segment Anything: High-Resolution Tiling and Prompt Simulation Because PGM is trained on low-resolution (\(512 \times 512\)) video frames and lacks prompt interaction capabilities, an adaptation pipeline transfers its learned objectness prior to high-resolution static domains. Frozen PGM processes overlapping multi-scale tiles on static images, computes pairwise cosine similarities, sharpens boundaries via Dense CRF, and merges candidates using NMS. With perceptual grouping masks generated over just 1% of the unlabeled SA-1B dataset, two downstream architectures are trained following UnSAM: a Mask2Former model with a ResNet-50 backbone for automated whole-image segmentation, and a Semantic-SAM model with a Swin-Tiny backbone for promptable interactive segmentation, where positive point prompts are simulated by sampling foreground coordinates within pseudo-masks.

Loss & Training

During Stage 2 PGM pre-training, AdamW is adopted with a learning rate of \(1 \times 10^{-4}\) and batch size 32. Copy-Paste data augmentation is applied to build robustness against occlusions. Across \(N\) patch tokens and \(N_a\) sampled anchor patches, the PGCL objective is averaged over all \(K\) heads:

\[ \mathcal{L}_{\text{PGCL}} = - \frac{1}{K N_a N} \sum_{k=1}^{K} \sum_{t=1}^{N_a} \sum_{j=1}^{N} \Big[ m_{t,j} \log p_{t,j}^k + (1 - m_{t,j}) \log (1 - p_{t,j}^k) \Big] \]

where \(m_{t,j} \in \{0, 1\}\) equals 1 if and only if both anchor patch \(t\) and patch \(j\) reside within the pseudo-mask \(M_i\). In Stage 3, the Mask2Former whole-image model is trained for 8 epochs with learning rate \(5 \times 10^{-5}\), while the Semantic-SAM interactive model is trained on simulated point prompts without any human-annotated masks.

Key Experimental Results

Main Results

Evaluated on seven challenging benchmarks spanning whole entities and fine-grained parts (COCO, LVIS, ADE20K, EntitySeg, SA-1B, PartImageNet, PACO) using Average Recall (AR) as the primary open-world metric, MoSA (ResNet-50, 23M parameters) sets a new state of the art among unsupervised approaches and matches or surpasses fully supervised SAM (ViT-B, 85M parameters).

Dataset / Task	Metric	MoSA (Ours, RN-50)	UnSAM (RN-50)	SAM (Supervised, ViT-B)	Gain over Prev. SOTA (UnSAM)
Average (7 Datasets Avg.)	AR	42.1	39.2	42.1	+2.9
COCO Whole-Image	AR	43.5	40.5	49.6	+3.0
LVIS Whole-Image	AR	42.2	37.7	46.1	+4.5
ADE20K Whole-Image	AR	38.4	35.7	45.8	+2.7
EntitySeg Whole-Image	AR	41.1	39.6	45.9	+1.5
SA-1B Whole-Image	AR	48.2	41.9	60.8	+6.3
PartImageNet Parts	AR	52.7	51.6	28.3	+1.1
PACO Fine-Grained Parts	AR	28.4	27.5	18.1	+0.9
COCO Point Prompt (MaxIoU)	MaxIoU	41.6%	40.3%	52.1%	+1.3%
COCO Point Prompt (OracleIoU)	OracleIoU	63.4%	59.5%	68.2%	+3.9%

Note: UnSAM relies on CascadePSP, a refiner trained on manual supervision, whereas MoSA maintains a strictly unsupervised learning pipeline.

Ablation Study

Ablations on 1,000 SA-1B images evaluate PGM training objectives and MGMS component contributions across overall \(AR_{1000}\), small \(AR_s\), medium \(AR_m\), and large \(AR_l\) scales:

Experiment Group	Configuration	\(AR_{1000}\)	\(AR_s\)	\(AR_m\)	\(AR_l\)	Note / Key Insight
PGM Loss Formulation	Standard BCE + Dice	3.8	1.2	2.8	6.6	Conventional loss penalizes static foreground as background
PGM Loss Formulation	Slot Attention (256 slots)	8.2	5.2	8.6	8.7	Fixed capacity limits open-world scaling and multi-granularity
PGM Loss Formulation	PGCL (Ours)	28.3	14.7	26.6	36.1	Contrastive grouping learns robust appearance objectness
MGMS Module Selection	ROSM only (Rigid Object)	28.8	20.6	31.1	28.7	Excels at fine parts and small objects but misses global context
MGMS Module Selection	MFSM only (Moving Foreground)	34.7	13.7	34.9	43.1	Captures large non-rigid entities but lacks fine-grained details
MGMS Module Selection	ROSM + MFSM (Full Model)	36.6	20.6	35.4	45.1	Complementary multi-scale motion cues achieve optimal performance

Key Findings

• Contrastive Learning Resolves Pseudo-Label Sparsity: Standard segmentation losses collapse under sparse motion labels (yielding only 3.8 AR), whereas PGCL achieves 28.3 AR, confirming that feature-space compactness provides an effective supervisory signal for unannotated object discovery.
• Superior Fine-Grained Part Comprehension: Benefiting from relative local motions in dynamic scenes, MoSA outperforms fully supervised SAM by +24.4% on PartImageNet (52.7% vs. 28.3%) and +10.3% on PACO (28.4% vs. 18.1%), demonstrating that motion cues reveal hierarchical boundaries frequently omitted by human annotators.
• Scalability with Heads and Unlabeled Video: Increasing projection heads from 1 to 4 boosts AR from 27.9% to 36.6% before saturating at 5 heads (36.5%). Furthermore, expanding pre-training video data from 100h to 1,000h and 10,000h yields monotonic non-saturating gains (16.8% \(\rightarrow\) 28.3% \(\rightarrow\) 36.6%), highlighting strong foundation-model scaling behavior.

Highlights & Insights

• From Dynamic Motion Priors to Static Cognitive Grouping: Implements the Gestalt principle of common fate into a scalable machine learning pipeline, proving that foundational static object perception can be acquired solely by observing raw video dynamics without human labels.
• Strictly Unsupervised Benchmark Superiority: Unlike prior leading methods such as UnSAM that incorporate supervised boundary refiners (CascadePSP), MoSA establishes a genuinely unsupervised pipeline and attains performance competitive with supervised SAM while using 3.4x fewer parameters and 100x less image data.
• Modular Multi-Scale Feature Contrast: The multi-head PGCL framework disentangles scale specialization and patch-level clustering, offering a reusable template for other sparse self-supervised grouping problems.

Limitations & Future Work

• Optical Flow Dependency: The quality of motion pseudo-labels depends on bidirectional optical flow estimators, which are susceptible to severe camera shake, reflections, and extreme low-light environments.
• Multi-Stage Training Pipeline: The end-to-end framework requires sequential execution across synthetic pre-training, video pseudo-labeling, PGM training, and downstream adaptation (Mask2Former/Semantic-SAM), rather than an integrated end-to-end self-supervised system.
• Future Directions: Integrating video self-supervised representations with generative diffusion priors to establish a unified, single-stage online self-evolving segmentation foundation model from continuous video streams.

Related Work & Insights

• vs CutLER / UnSAM: CutLER and UnSAM derive pseudo-masks via clustering static DINO features on individual images, with UnSAM relying on a supervised CascadePSP refiner; MoSA grounds objectness in physical motion across unlabeled videos, preserving a purely unsupervised pipeline while excelling on fine-grained parts.
• vs DyStaB / DIOD: Existing motion-bootstrapping works focus on discovering specific moving foreground masks, leading to severe motion bias on static scenes; MoSA's PGCL decouples motion supervision from appearance grouping, enabling zero-shot generalization across static imagery and abstract hand-drawn sketches.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant operationalization of Gestalt perceptual grouping into a scalable three-stage unsupervised Segment Anything framework with novel PGCL.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive zero-shot evaluation across seven whole-image and part-level benchmarks, complete with promptable interaction tests, multi-scale ablations, and data scaling curves.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear, compelling narrative with rigorous structure and insightful connections between biological perception and computer vision architectures.
• Value: ⭐⭐⭐⭐⭐ Demonstrates that large-scale unlabeled video dynamics offer a viable, annotation-free alternative to massive manual datasets for building foundation segmentation models.