Learning Accurate Segmentation Purely from Self-Supervision

Conference: ECCV 2026
Paper: ECCV Official
Code: https://geshang777.github.io/Selfment/
Area: Segmentation
Keywords: self-supervised learning, unsupervised saliency detection, camouflaged object detection, normalized cut, iterative patch optimization

TL;DR

Selfment introduces a fully self-supervised foreground segmentation framework that operates without manual labels, pretrained segmentation priors (e.g., SAM), or CRF/bilateral post-processing: it generates an initial coarse partition via Normalized Cut on patch affinity graphs, refines it via Iterative Patch Optimization (IPO) in feature space, and trains a lightweight projection head with contrastive and region-consistency objectives to achieve new state-of-the-art results across salient and camouflaged object detection benchmarks.

Background & Motivation

Object segmentation has fundamentally relied on dense, pixel-level manual masks. However, creating pixel-wise human annotations is prohibitive in cost and time, inherently limiting scalability while injecting subjective human inductive biases into the model. Recent weakly supervised works have sought to ease this burden using sparse cues (points, scribbles, motion vectors) or prompt-based off-the-shelf segmentation models such as SAM. Nevertheless, these strategies remain tethered to either manual prompts or models pretrained on massive human-annotated datasets, evading the fundamental research challenge: can a computer vision model discover and segment precise object masks purely from raw, unlabeled images?

Self-supervised vision foundation models—most prominently the DINO family and DINOv3 with Gram Anchoring—have demonstrated that dense patch embeddings spontaneously cluster by semantic similarity without any supervision. Prior approaches like TokenCut construct patch-level affinity graphs from these representations and apply Normalized Cut (NCut) to partition foreground from background. However, due to heuristic graph thresholding and continuous spectral relaxation, raw NCut partitions are inherently noisy and lack spatial coherence. Achieving acceptable segmentation quality has conventionally demanded heavy post-processing pipelines (e.g., dense CRFs, bilateral solvers, or morphological heuristics), which undermine the autonomy of self-supervised learning and cause severe degradation when scaling to high-resolution images.

The core insight of this paper is that the dense feature manifold of self-supervised representations contains sufficient topological clarity to self-purify segmentation masks internally without external heuristic smoothing. While the Fiedler vector from NCut offers a solid semantic seed, its spectral noise can be iteratively rectified directly in feature space via patch clustering and orientation locking. Core idea: leverage self-supervised feature affinity to obtain an initial coarse NCut bipartition, refine it into high-fidelity masks via feature-space Iterative Patch Optimization (IPO) with orientation consistency, and distill the resulting pseudo-labels into a lightweight projection head using contrastive and region-consistency objectives.

Method

Overall Architecture

The Selfment pipeline consists of four synergistic stages: self-supervised feature extraction, graph-based NCut initialization, Iterative Patch Optimization (IPO), and self-supervised lightweight head training. An input image is first fed into a frozen self-supervised backbone (e.g., DINOv3-7B) to extract dense patch representations. An affinity graph is constructed over these embeddings, and solving the generalized eigenvalue problem yields the Fiedler vector, which produces a coarse binary foreground-background mask. Next, IPO refines patch assignments via dynamic feature centroid clustering and orientation consistency constraints. Finally, a lightweight two-layer projection head is trained using the refined pseudo-labels under a composite objective of contrastive alignment, soft Dice loss, and binary cross-entropy, enabling fast, end-to-end inference at arbitrary resolutions without post-processing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image (H×W×3)"] --> B["Self-Supervised Feature Extraction<br/>Frozen DINOv3 dense patch embeddings"]
    B --> C["NCut Affinity Graph Bipartition<br/>Construct graph & compute Fiedler vector"]
    C --> D["Iterative Patch Optimization (IPO)<br/>Feature-space clustering & orientation lock"]
    D --> E["Self-Supervised Head Training<br/>Train projection head via Con/Dice/BCE loss"]
    E --> F["High-Resolution Segmentation Output"]

Key Designs

1. Normalized Cut Initialization: Spectral Seed Discovery on Feature Affinity Graphs

To identify object candidates without human priors, an undirected weighted affinity graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) is constructed over normalized patch features \(\{f_i\}_{i=1}^N\). Pairwise edge weights measure cosine similarity under a threshold \(\tau = 0.2\):

\[A_{ij} = \begin{cases} \langle f_i, f_j \rangle, & \text{if } \langle f_i, f_j \rangle > \tau \\ \epsilon, & \text{otherwise} \end{cases}\]

where \(\epsilon\) is a minimal constant maintaining graph connectivity. With diagonal degree matrix \(D_{ii} = \sum_j A_{ij}\), the NCut objective is relaxed into the generalized eigenvalue formulation \((D - A)\mathbf{x} = \lambda D \mathbf{x}\). The second-smallest eigenvector \(x_2\) (the Fiedler vector) defines the optimal continuous graph bipartition. The initial binary mask \(y^{(0)}\) is obtained by thresholding \(x_2\) at its mean value and extracting the connected component containing the seed patch with the maximum absolute magnitude in \(x_2\). This produces an initial, object-centric semantic seed directly from self-attention geometry.

2. Iterative Patch Optimization (IPO): Feature Manifold Self-Purification

Because spectral relaxation and threshold truncation introduce boundary noise and fragment artifacts, IPO refines the coarse partition entirely within the self-supervised feature space rather than relying on external filtering. Patch embeddings are first \(\ell_2\)-normalized as \(\tilde{f}_i = f_i / \|f_i\|_2\). Given the current foreground partition \(\mathcal{F}^{(t)}\) and background partition \(\mathcal{B}^{(t)}\), dynamic cluster centroids are computed as:

\[\mu_f^{(t)} = \frac{1}{|\mathcal{F}^{(t)}|} \sum_{i\in\mathcal{F}^{(t)}} \tilde{f}_i, \quad \mu_b^{(t)} = \frac{1}{|\mathcal{B}^{(t)}|} \sum_{i\in\mathcal{B}^{(t)}} \tilde{f}_i\]

At each step \(t\), patch labels are reassigned according to relative similarity to the centroids: \(y_i^{(t+1)} = 1\) if \(\langle \tilde{f}_i, \mu_f^{(t)} \rangle > \langle \tilde{f}_i, \mu_b^{(t)} \rangle\), and \(0\) otherwise, followed by updating the cluster means. The procedure repeats for \(T=20\) iterations (empirically stabilizing within ~10 iterations). To prevent label flipping during iterative updates, a fixed reference vector \(r = \mu_f^{(0)} - \mu_b^{(0)}\) is tracked; if \(\langle (\mu_f^{(t+1)} - \mu_b^{(t+1)}), r \rangle < 0\), the labels are reversed to preserve persistent foreground-background semantics.

3. Self-Supervised Projection Head: Contrastive-Geometric Representation Learning

While IPO produces high-fidelity masks, executing spectral graph cuts and iterative clustering per image at test time is computationally demanding. Selfment addresses this by distilling the refined pseudo-labels into a lightweight segmentation head \(\phi_\theta\) (0.54M parameters, consisting of a two-layer MLP with ReLU followed by a linear classifier \(W_c\)). The head maps frozen patch features \(f_i\) into embeddings \(z_i \in \mathbb{R}^d\) and classification logits \(l_i\). The head is optimized via three complementary objectives: - Patch-level Contrastive Loss \(\mathcal{L}_{\text{con}}\): Inspired by InfoNCE, normalized embeddings \(z_i\) are compared using pairwise similarity \(S_{ij} = z_i^\top z_j / \tau\). Patches sharing the same pseudo-label form positive pairs \(\mathcal{P}_i\), pulling intra-region patches together while pushing foreground and background embeddings apart; - Soft Dice Loss \(\mathcal{L}_{\text{Dice}}\): Operating on predicted foreground probabilities \(p_i = \sigma(l_i^{(1)})\) and pseudo-labels \(y_i\), it drives continuous spatial compactness and alleviates foreground-background area imbalance:

\[\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum_i p_i y_i + \epsilon}{\sum_i p_i^2 + \sum_i y_i^2 + \epsilon}\]

Coupled with standard patch-wise binary cross-entropy \(\mathcal{L}_{\text{BCE}}\), the total self-supervised objective is \(\mathcal{L}_{\text{total}} = 0.1 \mathcal{L}_{\text{con}} + 1.0 \mathcal{L}_{\text{Dice}} + 1.0 \mathcal{L}_{\text{BCE}}\). At inference time, the frozen backbone and lightweight head produce crisp masks in a single feed-forward pass.

Loss & Training

• Corpus & Optimization: Trained using only 1,000 unlabeled images randomly sampled from the DUTS training set for 3 epochs with Adam (\(1 \times 10^{-3}\) learning rate).
• Efficiency & Caching: The DINOv3-7B backbone remains completely frozen; patch embeddings are pre-extracted and cached to disk, allowing 8 NVIDIA A100 GPUs to finish training in just 27.6 minutes.
• Resolution Generalization: Despite training at \(768 \times 768\), the model generalizes seamlessly to \(1280 \times 1280\) and \(2048 \times 2048\) during inference, capturing finer structural details without retraining.

Key Experimental Results

Main Results

Evaluated across four standard unsupervised salient object detection (USOD) benchmarks without any post-processing, Selfment sets new state-of-the-art baselines. Furthermore, it demonstrates remarkable zero-shot transfer on challenging camouflaged object detection (COD) tasks.

Table 1: Comparison of unsupervised saliency detection methods (from original Table 1, no post-processing)

Method	ECSSD \(F_{\max} \uparrow\)	ECSSD IoU \(\uparrow\)	DUTS \(F_{\max} \uparrow\)	DUTS IoU \(\uparrow\)	HKUIS \(F_{\max} \uparrow\)	HKUIS IoU \(\uparrow\)	PASCAL-S \(F_{\max} \uparrow\)	PASCAL-S IoU \(\uparrow\)
TokenCut-768 [48]	87.8	75.9	73.0	60.5	82.2	64.0	76.2	61.3
TokenCut-1280 [48]	86.7	75.0	68.1	55.9	79.6	64.9	80.0	60.2
SelfMask-768 [40]	91.9	77.2	79.4	62.4	89.8	74.4	86.0	61.8
FOUND-768 [42]	91.4	78.9	76.4	64.8	87.7	68.4	85.4	63.7
Selfment-768 (Ours)	95.3	82.4	85.1	66.6	93.9	80.0	91.5	71.2
Selfment-1280 (Ours)	95.9	84.3	86.4	68.4	94.4	81.6	91.7	71.6

Table 2: Zero-shot performance on camouflaged object detection benchmarks (from original Table 2)

Paradigm	Method	CHAMELEON \(S_m \uparrow\)	CHAMELEON \(F_\beta^\omega \uparrow\)	CAMO \(S_m \uparrow\)	CAMO \(F_\beta^\omega \uparrow\)	COD10K \(S_m \uparrow\)	COD10K \(F_\beta^\omega \uparrow\)	NC4K \(S_m \uparrow\)	NC4K \(F_\beta^\omega \uparrow\)
Fully-Supervised	SINetv2 [10]	.888	.816	.820	.743	.815	.680	.847	.770
Fully-Supervised	FSPNet [17]	.908	.851	.856	.799	.851	.735	.879	.816
Fully-Supervised	BiRefNet [54]	.929	.911	.932	.914	.913	.874	.914	.894
Unsupervised	TokenCut [48]	.654	.496	.633	.498	.658	.469	.725	.615
Unsupervised	UCOD-DPL [50]	.864	.825	.793	.747	.834	.763	.850	.818
Pure Self-Supervised	Selfment (Ours)	.910	.843	.869	.792	.873	.754	.902	.836

Ablation Study

Table 3: Cumulative module contribution on ECSSD (trained on 500 DUTS images, from original Table 3)

Configuration	BCE	Dice	Con.	\(F_{\max} \uparrow\)	IoU \(\uparrow\)	Acc \(\uparrow\)
NCut (baseline)	-	-	-	74.7	63.9	86.2
+ IPO	-	-	-	79.5	73.2	87.8
+ Self-supervised training	✓	-	-	88.3	80.4	94.4
+ Dice loss	✓	✓	-	88.9	81.3	94.7
+ Contrastive loss	✓	-	✓	88.9	81.4	94.7
+ Dice & Contrastive loss (Full)	✓	✓	✓	89.1	81.5	94.8

Key Findings

• Substantial Gains from IPO: Adding IPO directly on top of NCut without any learning increases \(F_{\max}\) from 74.7% to 79.5% and IoU from 63.9% to 73.2% (+9.3%), proving that feature-space centroid clustering effectively cleans spectral relaxation errors.
• Robustness to Model Scaling: While prior methods like TokenCut and FOUND fail to benefit from or even collapse when scaling to DINOv3-7B (FOUND fails completely with \(F_{\max}=0\) due to fine-grained background seed sensitivity), Selfment consistently improves as the backbone scales from DINO-Base to DINOv3-7B.
• Positive High-Resolution Scaling: Traditional graph cuts suffer severe degradation at higher resolutions due to graph instability (e.g., TokenCut drops from 73.0% to 68.1% on DUTS when resolution grows from 768 to 1280). In contrast, Selfment improves from 85.1% to 86.4% on DUTS, successfully capturing minute details.

Highlights & Insights

• Self-Purification within Representation Manifold: Rather than turning to external spatial regularizers (CRFs or bilateral solvers), Selfment exploits the intrinsic semantic geometry of self-supervised representations through IPO, achieving boundary precision in milliseconds.
• True Annotation-Free Paradigm: Bypasses the widespread reliance on SAM prompt-engineering, achieving competitive performance entirely from bottom-up visual representation learning using only 1,000 unlabeled training images.
• Backbone-Agnostic Generalization: Extends robustly beyond the DINO lineage; evaluations on Perception Encoder (PE) and TIPSv2 demonstrate over 30% absolute \(F_{\max}\) improvements over TokenCut across different ViT architectures.

Limitations & Future Work

• Single Primary Connected Component Assumption: NCut seed extraction assumes the primary object corresponds to the maximum magnitude in the Fiedler vector, which can miss disjoint secondary instances in cluttered multi-object scenes.
• Backbone Feature Footprint: DINOv3-7B feature extraction incurs substantial GPU memory requirements; while mitigated via offline caching during training, edge deployment demands smaller backbones.
• Future Directions: Extending the formulation to multi-eigenvector spectral clustering for unsupervised panoptic/multi-instance segmentation, and integrating IPO clustering directly into pretraining objectives.

Related Work & Insights

• vs TokenCut [48]: TokenCut relies on raw NCut on ViT features, requiring bilateral filter post-processing and suffering severe drops at high resolution; Selfment replaces heuristic post-processing with feature-space IPO and a lightweight distillable head, maintaining stability across resolutions.
• vs FOUND [42] / SelfMask [40]: FOUND relies on background seed heuristics that cause training failure on fine-grained DINOv3-7B representations; Selfment dynamically updates foreground and background centroids globally, scaling reliably with larger models.
• vs SAM-Guided Pseudo-Labeling: Methods leveraging SAM inherit strong human prior supervision from SA-1B; Selfment remains strictly self-supervised from pretraining to segmentation prediction.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Replaces heuristic post-processing and SAM dependency with elegant feature-space iterative clustering and self-supervised distillation]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across USOD and COD benchmarks, backbone ablations, resolution scaling, and component breakdowns]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, structured pipeline narrative, and honest empirical analysis]
• Value: ⭐⭐⭐⭐⭐ [Establishes a practical, low-cost milestone for autonomous, label-free segmentation]

Related Papers

• [CVPR 2026] Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding
• [ECCV 2026] SkelEM: Training-Signal Decoupling of Skeleton and Diffusion for Self-supervised Axial Super-Resolution in Volume Microscopy
• [ECCV 2024] Representing Topological Self-Similarity Using Fractal Feature Maps for Accurate Segmentation of Tubular Structures
• [ECCV 2026] Unsupervised Semantic Segmentation Facilitates Model Understanding
• [ECCV 2026] Thermo-JEPA: Learning a Geometry-Grounded Thermal World Model via Cross-Modal Privileged Masking