INSID3: Training-Free In-Context Segmentation with DINOv3

Conference: CVPR 2026
arXiv: 2603.28480
Code: GitHub
Area: Segmentation
Keywords: In-context segmentation, DINOv3, training-free, self-supervised, positional bias correction

TL;DR

This paper proposes INSID3, a training-free in-context segmentation method relying solely on frozen DINOv3 features. Through a three-stage pipeline consisting of positional bias elimination, fine-grained clustering, and seed cluster aggregation, it outperforms methods relying on SAM or fine-tuning across semantic, part, and personalized segmentation tasks using a single self-supervised backbone, achieving an average mIoU gain of +7.5%.

Background & Motivation

In-context segmentation (ICS) aims to segment arbitrary concepts (objects, parts, or personalized instances) in a target image given a single labeled example. Existing methods generally follow two paradigms:

1. Fine-tuning approach (e.g., SegIC, DiffewS): These train segmentation decoders on VFMs or fine-tune diffusion models. While effective in-domain, they exhibit poor generalization.
2. Training-free approach (e.g., Matcher, GF-SAM): these combine DINOv2 with SAM. They offer strong generalization but involve complex architectures and high computational overhead.

Key Challenge: Prior methods depend on specific segmentation priors (SAM pre-training or downstream fine-tuning), preventing a truly "purely self-supervised" segmentation.

As the latest self-supervised VFM, DINOv3 produces dense local features with strong spatial structures due to massive data scaling and the Gram anchoring objective. Core Idea: DINOv3's dense self-supervised features inherently possess semantic matching and segmentation capabilities, requiring no additional decoders, fine-tuning, or model ensembling.

Method

Overall Architecture

INSID3 addresses a straightforward question: given a labeled reference image and a target image, can the same concept be extracted using only frozen DINOv3 features without training a decoder? The pipeline updates no parameters and follows three steps: first, removing "positional noise" from DINOv3 features to clarify cross-image matching; second, decomposing the target image into semantically coherent region candidates; finally, identifying target candidates using the reference labels and aggregating them into a complete mask. The process utilizes two sets of features: debiased features for cross-image matching and original features for internal image clustering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    REF["Ref Image + Mask"] --> ENC["DINOv3 Frozen Encoder<br/>Extract dense patch features"]
    TGT["Target Image"] --> ENC
    NOISE["Gaussian Noise Image"] --> ENC
    ENC --> DEBIAS["Positional Bias Removal<br/>SVD on noise features for positional subspace B<br/>Project real features to orthogonal complement"]
    ENC --> CLUS["Fine-grained Hierarchical Clustering<br/>Agglomerative clustering on original target features<br/>Decompose into K region candidates"]
    DEBIAS -->|Debiased features for cross-image matching| SEED
    CLUS -->|K region candidates| SEED
    subgraph SEED["Seed Cluster Selection & Aggregation"]
        direction TB
        LOC["Candidate Localization<br/>Reverse correspondence: Keep if NN falls within ref mask"] --> SEL["Seed Selection<br/>Candidate with highest cross-image similarity as seed"]
        SEL --> AGG["Cluster Aggregation<br/>Aggregate clusters where cross-img × self-similarity ≥ α"]
    end
    SEED --> OUT["CRF Boundary Refinement<br/>Output full mask"]

Key Designs

1. Positional Bias Removal: Eliminating the "same position = same object" false signal

The authors discovered a counter-intuitive phenomenon where unrelated images show strong DINOv3 feature matching at identical spatial locations. This indicates a signal tied solely to patch coordinates rather than content, causing false correspondences. To isolate this, Gaussian noise \(\mathbf{I}^{noise} \sim \mathcal{N}(0,1)\) is fed into the encoder. Since it lacks semantic content, the extracted features mainly represent the positional signal. SVD is performed on these noise features to obtain the basis \(\mathbf{B}\) for the "positional subspace," and real features are projected onto the orthogonal complement:

\[\tilde{\mathbf{F}} = \mathbf{F}(\mathbf{1}_D - \mathbf{B}\mathbf{B}^\top)\]

This debiasing is applied only to cross-image matching. Intra-image clustering retains original features as spatial information serves as a useful prior within a single image.

2. Fine-grained Hierarchical Clustering: Region candidates

To decompose the target image into "potential concepts," the authors use agglomerative (hierarchical) clustering on original target features \(\mathbf{F}^t\). This merges adjacent similar patches from the bottom up into \(K\) non-overlapping spatial regions \(\{\mathcal{G}_1, ..., \mathcal{G}_K\}\). Unlike K-means, this does not require a pre-defined number of clusters, and it is more stable in high-dimensional space than DBSCAN.

3. Seed Cluster Selection & Aggregation: Locking and growing the mask

To identify target candidates among background or distractors, the reference labels are used. Candidate Localization uses reverse correspondence: a target patch is kept only if its nearest neighbor in the reference image falls inside the reference mask, yielding a candidate set \(\mathcal{C}_{cand}\). Seed Selection picks the candidate with the highest cross-image similarity \(s_k^{cross}\) between the candidate prototype and the reference prototype in the debiased feature space as the seed \(\mathcal{G}^*\). Cluster Aggregation then incorporates any cluster where the product of cross-image similarity and intra-image self-similarity \(s_k^{intra}\) exceeds a threshold: \(S_k = s_k^{cross} \cdot s_k^{intra} \geq \alpha\).

Loss & Training

Ours is entirely training-free and does not involve any loss functions or training procedures. During inference, CRF is used for mask refinement, and input images are resized to 1024×1024.

Key Experimental Results

Main Results

Dataset	Metric	Ours	Prev. SOTA (GF-SAM)	Gain
LVIS-92i (Semantic)	mIoU	41.8%	35.2%	+6.6
COCO-20i (Semantic)	mIoU	57.6%	58.7%	-1.1
ISIC (Skin Lesion)	mIoU	54.4%	48.7%	+5.7
Chest X-Ray	mIoU	78.8%	51.0%	+27.8
iSAID (Remote Sensing)	mIoU	52.1%	47.1%	+5.0
PASCAL-Part (Part)	mIoU	50.5%	44.5%	+6.0
PACO-Part (Part)	mIoU	38.7%	36.3%	+2.4
PerMIS (Personalized)	mIoU	67.0%	54.1%	+12.9
Mean of 9 Datasets	mIoU	55.1%	47.6%	+7.5

Parameter count: INSID3 uses 304M parameters vs. GF-SAM's 945M (3× reduction).

Ablation Study

Configuration	COCO mIoU	PASCAL-Part mIoU	Note
Thresholded similarity map	44.2%	35.4%	No clustering baseline
Coarse clustering (τ=0.5) w/o agg.	50.6%	31.1%	Better for object-level
Fine clustering (τ=0.6) w/o agg.	42.8%	36.2%	Better for part-level
Clus. + Cross-image aggregation	54.6%	48.5%	Cross similarity only
Clus. + Cross + Self-similarity agg.	57.6%	50.5%	Full method

Key Findings

• DINOv3 exhibits systematic positional bias where features at the same spatial coordinates produce false matches; this may stem from the Gram anchoring objective.
• Positional debiasing is generally effective for semantic correspondence, improving PCK by +1.4~6.6 on SPair-71k.
• Fine-tuning methods like SegIC achieve high in-domain mIoU (76.1% on COCO) but fail across domains (e.g., 46.1% on iSAID), whereas INSID3 remains stable across all domains.

Highlights & Insights

• Minimalist Philosophy: A single frozen self-supervised backbone can perform in-context segmentation without decoders, fine-tuning, or model ensembling.
• Positional Bias Correction: The paper identifies the positional bias in DINOv3 and provides a simple SVD-based solution that generalizes to other tasks like semantic correspondence.
• Reverse Correspondence: This mechanism effectively treats unlabeled regions in the reference image as implicit negative samples, solving the issue of distractor instances in personalized segmentation.

Limitations & Future Work

• Performance on COCO-20i is slightly lower than GF-SAM (57.6% vs 58.7%), suggesting self-supervised features may not yet match SAM's mask priors on in-domain data.
• Dependence on DINOv3-Large (1024×1024 input) entails high computational costs.
• Thresholds \(\tau\) and \(\alpha\) are fixed across tasks and may not be optimal for all scenarios.
• Extensions to few-shot (multi-example) scenarios were not explored.

Related Work & Insights

• vs GF-SAM: GF-SAM uses DINOv2 matching points as prompts for SAM, discarding much dense feature information; INSID3 performs matching and segmentation in a unified space.
• vs SegIC: SegIC achieves strong in-domain performance through decoder training but suffers from poor generalization.
• vs DINOv2: DINOv2 has weaker positional bias than DINOv3, possibly because DINOv3's training objective unintentionally amplifies spatial correlation.

Rating

• Novelty: ⭐⭐⭐⭐
• Experimental Thoroughness: ⭐⭐⭐⭐⭐
• Writing Quality: ⭐⭐⭐⭐
• Value: ⭐⭐⭐⭐

Related Papers

• [CVPR 2026] The Power of Prior: Training-Free Open-Vocabulary Semantic Segmentation with LLaVA
• [CVPR 2026] PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentation
• [CVPR 2026] Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
• [CVPR 2026] Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
• [CVPR 2026] ReAttnCLIP: Training-Free Open-Vocabulary Remote Sensing Image Segmentation via Re-defined Attention in CLIP