Unsupervised Semantic Segmentation Facilitates Model Understanding¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Kainmueller-Lab/ssl-rep-seg
Area: Segmentation
Keywords: self-supervised learning, interpretability, positional effect, locality bias, vision transformer
TL;DR¶
Addressing the lack of intuitive visualization tools for internal representations in self-supervised Vision Transformers, this work establishes an unsupervised semantic segmentation and mutual-information probing protocol, revealing that dominant positional effects in Key/Query embeddings trigger downstream locality bias in Masked Image Modeling and identifying severe spatial boundary artifacts as a key driver of inverse scaling in DINOv3-Large.
Background & Motivation¶
Self-supervised learning (SSL) has catalyzed a diverse family of Vision Transformers (ViTs) that capture expressive patch-level and image-level semantic representations without human annotations. To inspect the mechanics underpinning these representations, numerous prior works have analyzed self-attention mechanisms and latent feature spaces, establishing distinct behaviors across contrasting pretraining regimes, most notably between contrastive learning (CL, such as DINO and MoCov3) and masked image modeling (MIM, such as MAE and iBOT). However, findings from this literature remain prone to over-generalization: insights uniquely discovered in CL frameworks—such as the presumption that Key embeddings intrinsically encode clean semantic structure, or that spatial position information steadily dissolves in deeper ViT layers—are frequently and erroneously assumed to hold universally across all SSL paradigms.
A fundamental obstacle to broader model understanding lies in the lack of an intuitive, comparative visualization and diagnostic framework that operates consistently across multiple images. Existing visualization tools (such as attention heatmaps and attention matrices) typically evaluate intra-image relations—capturing relative spatial dependencies termed locality bias—while failing to isolate absolute spatial coordinate dependencies that recur systematically across different images, termed positional effect. Crucially, in the context of model scaling, larger models (such as ViT-Large) frequently underperform their Base counterparts on dense prediction tasks. While existing hypotheses cite degraded patch consistency or pretext-downstream task misalignment, this inverse scaling behavior has never been systematically examined through the lens of detrimental, image-wide positional biases.
The angle of attack in this paper is to repurpose standard unsupervised semantic segmentation via non-parametric clustering as an unadulterated, tuning-free probing protocol. By completely avoiding downstream task fine-tuning that inevitably reshapes and confounds the native embedding manifold, across-image clustering surfaces both semantic structure and spurious spatial biases in an immediately interpretable, colorized format. Core idea: repurpose unsupervised semantic segmentation as a tuning-free probing protocol, coupling across-image k-means clustering with mutual-information-based spatial metrics to disentangle semantic structure from absolute positional effects across SSL ViTs, demonstrating that dominant positional effects in Keys and Queries act as the upstream cause of MIM locality bias and revealing that boundary artifacts and spurious spatial clusters underpin the inverse scaling of DINOv3-Large.
Method¶
Overall Architecture¶
The proposed protocol avoids task-specific decoders or supervised fine-tuning that distort the geometry of learned representations. It uses frozen ViT backbones across arbitrary layers \(l \in [1, L]\) and canonical embedding components, including Keys, Queries, and Values extracted from Multi-Head Attention (MHA) blocks (concatenated along attention dimensions across all heads) as well as patch Tokens output by Feed-Forward Networks (FFN). It then applies non-parametric online k-means clustering across images to assign discrete cluster IDs and project dense semantic segmentation maps onto test images.
The diagnostic workflow is built upon four coordinated pillars: sample-level segmentation visualization, dataset-wide aggregate heatmaps, an information-theoretic positional effect metric, and Hungarian-matched mIoU for semantic quality. The overall dataflow and module interactions are illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Unannotated Image Set"] --> B["Across-Image Cluster Probing<br/>Extract Multi-Layer K/Q/V/Token Embeddings & Online k-means"]
B --> C["Dual-Level Spatial Visualization<br/>Sample Image Palettes + Global Spatial Probability Map"]
C --> D["Mutual Information & Quality Evaluation<br/>Compute Global NMI / Class MI & Hungarian-Matched mIoU"]
D --> E["Model Diagnostic & Mechanistic Attribution<br/>Disentangle Positional Effect vs. Locality Bias & Trace Artifacts"]
Key Designs¶
1. Across-Image Cluster Probing: Tuning-Free Probing of Intrinsic Representations Standard evaluation techniques that train linear probes or fine-tune downstream decoders introduce extensive optimization hyperparameters and structural assumptions, inadvertently warping the intrinsic manifold of SSL embeddings. In contrast, this design applies batch-wise online k-means clustering directly to frozen patch embeddings \(F_i^{(l)} \in \mathbb{R}^{N \times d}\) across an input batch \(\{x_i\}_{i=1}^B\), where \(N = H_p \times W_p\) is the patch count and \(d\) is embedding dimensionality. The codebook of \(K\) cluster centroids \(C = \{c_k\}_{k=1}^K \subset \mathbb{R}^d\) is iteratively updated under cosine similarity across batches to ensure consistent assignment across diverse images. To generate dense pixel-level predictions without learnable decoders, patch features are bilinearly upsampled to native resolution \(\hat{F}_i \in \mathbb{R}^{H \times W \times d}\), assigning each pixel \((h, w)\) to its nearest centroid: $$ \hat{y}{hw} = \arg\max $$ Because centroids are updated globally across image batches, any persistent spatial regularities or semantic clusters emerge naturally in discrete cluster indices without complex self-training or post-processing artifacts.} \frac{\hat{\mathbf{f}}_i^{(hw)} \cdot \mathbf{c}_k}{|\hat{\mathbf{f}}_i^{(hw)}| |\mathbf{c}_k|
2. Dual-Level Spatial Visualization: Disentangling Semantic Structures from Positional Artifacts Visualizing individual images in isolation is insufficient to confirm whether a localized color patch reflects genuine semantic segmentation or an invariant spatial coordinate bias. The protocol therefore provides two complementary visual perspectives: exemplary individual image sets and dataset-wide aggregate views. On the exemplary image level, predicted clusters are mapped to ground truth classes via global Hungarian matching and colorized according to standard class palettes, immediately highlighting if identical image coordinates receive uniform classes across varied scenes. On the dataset level, a global class probability tensor \(W \in \mathbb{R}^{K \times H \times W}\) is computed by averaging predicted class probabilities across all test samples, normalized per pixel as \(\tilde{W}_{k,i,j} = W_{k,i,j} / \sum_{k'} W_{k',i,j}\), and rendered as an aggregate blended map \(A \in \mathbb{R}^{H \times W \times 3}\): $$ A_{i,j} = \sum_{k=1}^K \frac{W_{k,i,j}}{\sum_{k'} W_{k',i,j}} \cdot v_k $$ where \(v_k\) represents the RGB color assigned to class \(k\). While pure semantic representations yield smoothly distributed, diffuse aggregate maps, strong positional effects manifest as sharp, high-saturation color blocks locked to fixed spatial regions across the entire benchmark.
3. Mutual Information & Quality Evaluation: Information-Theoretic Quantification of Positional Bias To establish rigorous cross-model and cross-layer comparisons beyond subjective visual inspections, the protocol formulates positional bias using the mutual information between the discrete spatial location variable \(P\) and predicted class variable \(C\). It computes class-wise mutual information \(I_c(P; c) = \sum_p \mathbb{P}(p,c) \log \frac{\mathbb{P}(p,c)}{\mathbb{P}(p)\mathbb{P}(c)}\), aggregating into Normalized Mutual Information (NMI): $$ I'(P, C) = \frac{\sum_c I_c(P; c)}{H(C)} = \frac{I(P, C)}{-\sum_c \mathbb{P}(c) \log \mathbb{P}(c)} $$ where \(H(C)\) denotes the entropy of predicted class assignments. Ground-truth semantic masks serve as the natural empirical baseline, capturing intrinsic scene layout statistics (e.g., sky typically above ground). Analyzing NMI against attention matrix locality mutual information \(\hat{I}(Q, K)\) and segmentation mIoU demonstrates that Key/Query positional bias is strongly correlated with attention locality bias (\(\rho = 0.70, p < 0.001\)), establishing it as an upstream causal factor, while exhibiting a pronounced negative correlation with segmentation mIoU (\(\rho = -0.59, p < 0.001\)), confirming that excessive absolute positional bias directly degrades dense semantic representations.
Loss & Training¶
The proposed probing protocol requires no gradient backpropagation or learnable parameters. The k-means clustering centroids are initialized using principal component analysis (PCA) on extracted patch embeddings and updated iteratively via online mini-batch clustering until convergence on the evaluation split. For all ViT models, evaluation is conducted under frozen weights, and final-layer Token embeddings are systematically extracted immediately after the terminal LayerNorm layer to maintain uniform scaling across architectures.
Key Experimental Results¶
Main Results¶
On the COCO-Stuff 27-class benchmark, the probing protocol evaluates 8 representative self-supervised models alongside 2 supervised baselines (Supervised ViT and CLIP) across all layers and canonical feature representations (Keys, Queries, Values, Tokens). The main comparison below highlights the peak-performing layer performance (mIoU) alongside the global positional effect metric (NMI).
| Pretraining Paradigm | Model (ViT-B) | Best Component | Peak Layer mIoU (%) | Global Positional NMI | Key Characteristic Representation Behavior |
|---|---|---|---|---|---|
| Contrastive Learning (CL) | DINO | Token / Key | 31.8 / 26.2 | 0.082 | Positional effect confined to early layers (L2-3); deep layers exhibit clean semantics |
| Contrastive Learning (CL) | MoCov3 | Token | 24.5 | 0.089 | Locality bias and positional effects rapidly decay in deeper layers; abstract semantics |
| Multi-Granular SSL | Mugs | Token | 28.1 | 0.095 | Balanced performance across layers; Token representations substantially outperform Keys |
| Masked Modeling (MIM) | MAE | Token / Key | 18.2 / 6.4 | 0.176 | Keys/Queries exhibit extreme positional clustering; image corners cluster into single classes |
| Masked + Contrastive | iBOT | Token / Key | 32.4 / 14.7 | 0.134 | High Token semantic fidelity, yet Key/Query embeddings retain prominent positional biases |
| Joint-Embedding SSL | DINOv2 | Token | 36.8 | 0.142 | Crisp semantic contours but exhibits persistent high-frequency block artifacts |
| Register-Augmented | DINOv2+reg | Token | 35.9 | 0.108 | Registers effectively suppress block artifacts and spatial bias; minor high-frequency noise |
| Rotary Pos. Embedding | DINOv3 | Token / Key | 37.1 / 9.8 | 0.165 | Key embeddings dominated by spatial bias across all layers; final Token shows sudden spike |
| Vision-Language Align | CLIP | Token | 25.4 | 0.112 | Gradual loss of fine-grained spatial localization in deep layers; mild block artifacts |
| Supervised Baseline | Supervised ViT | Token | 27.6 | 0.105 | Early layers preserve edges; deep layers capture class concepts but lose fine detail |
| Empirical Ground Truth | Ground Truth (GT) | - | 100.0 | 0.071 | Empirical baseline reflecting natural photography composition (e.g., sky at top, ground at base) |
Ablation Study¶
Ablations and scaling experiments analyze the impact of model scaling (ViT-Base versus ViT-Large) and layer progression on learned internal representations and downstream dense semantic performance.
| Evaluation Config | Architecture Scale | Best Component / Layer | Semantic mIoU (%) | Positional NMI | Observed Representation & Artifact Characteristics |
|---|---|---|---|---|---|
| DINOv3 Scaling Study | ViT-Base | Token (Peak Layer) | 37.1 | 0.165 | Moderate positional clustering at bottom border; clean foreground semantic masks |
| DINOv3 Scaling Study | ViT-Large | Token (Peak Layer) | 32.5 | 0.228 | Severe inverse scaling: Upper/lower borders misassigned as spurious ceiling/floor clusters |
| DINOv2 Scaling Study | ViT-Base | Token (Peak Layer) | 36.8 | 0.142 | Localized block artifacts visible; object contours remain well-defined |
| DINOv2 Scaling Study | ViT-Large | Token (Peak Layer) | 33.1 | 0.210 | Significant increase in global positional effect, locking large spatial regions into fixed clusters |
| DINOv2+reg Scaling Study | ViT-Base | Token (Peak Layer) | 35.9 | 0.108 | Positional bias largely mitigated; clean background and object separation |
| DINOv2+reg Scaling Study | ViT-Large | Token (Peak Layer) | 34.2 | 0.115 | Positional bias stays suppressed, but patch-level consistency drops with fragmented noise |
| iBOT Scaling Study | ViT-Base | Token (Peak Layer) | 32.4 | 0.134 | Coarser object part groupings; lower central region shows faint spurious vehicle cluster |
| iBOT Scaling Study | ViT-Large | Token (Peak Layer) | 34.6 | 0.141 | Positive scaling: Fine-grained, semantically consistent body parts (heads, legs) across species |
| DINO Layer Progression | ViT-Base | Key (Layer 2-3) | 12.1 | 0.155 | Early layers exhibit transient positional effect before purifying into semantic features |
| DINO Layer Progression | ViT-Base | Key (Layer 11-12) | 26.2 | 0.084 | Deep layers completely transition into semantic representations, approaching GT NMI |
| MAE Layer Progression | ViT-Base | Key (Layer 5-12) | 6.4 | 0.245 | Intermediate layers exhibit massive positional lock-in; near-zero cross-image semantic utility |
Key Findings¶
- Universal Superiority of Token Embeddings: Across all evaluated models (with the lone edge case of MAE-Large), Token embeddings consistently and substantially outperform Key, Query, and Value representations in unsupervised semantic segmentation. This refutes the conventional belief—stemming from isolated observations on DINO—that Key embeddings are inherently superior dense visual descriptors.
- Causal Link from Key/Query Positional Effect to Attention Locality Bias: Normalized mutual information on Keys and Queries correlates strongly with attention matrix locality mutual information (\(\rho = 0.70, p < 0.001\)). Because attention weights are computed directly via normalized dot products of Queries and Keys, spatial coordinate clustering in these components forms the direct mechanistic upstream driver of diagonal locality attention.
- Negative Impact of Positional Bias on Semantic Segmentation: Across all models, layers, and components, positional effect NMI exhibits a significant negative correlation with segmentation mIoU (\(\rho = -0.59, p < 0.001\)). Any surplus positional effect beyond the empirical ground-truth baseline constitutes structural segmentation error.
- Anatomy of DINOv3 Inverse Scaling: Although DINOv3 adopts Rotary Position Embeddings (RoPE) intended to induce relative rather than absolute positional encoding, DINOv3-Large exhibits severe spurious clustering along top, bottom, and lateral boundaries (systematically misclassified as ceiling, floor, or solid classes), driving higher NMI and lower mIoU compared to DINOv3-Base.
Highlights & Insights¶
- Repurposing Unsupervised Segmentation as a Mechanistic Probe: The work transforms standard non-parametric clustering from a competitive downstream task into a clean, tuning-free analytical lens, offering immediate visual diagnosis of internal ViT properties without optimization confounds.
- Formal Decoupling of Locality Bias from Absolute Positional Effect: By combining intra-image attention metrics with cross-image mutual information, the paper formally disentangles short-range attention locality from absolute spatial coordinate lock-in.
- Unveiling the RoPE Paradox in Scaled Self-Supervised Models: It provides surprising empirical evidence that relative position encodings (RoPE) can still yield severe absolute positional boundary artifacts when scaled under large-scale self-distillation, providing actionable diagnostic targets for future architecture designs.
Limitations & Future Work¶
- Reliance on Hungarian Matching against Fixed Taxonomies: The protocol evaluates cluster fidelity against fixed semantic labels; if a model discovers valid visual concepts at a finer hierarchical granularity than standard ground-truth annotations, Hungarian mIoU may penalize these emergent representations.
- Disentangling Confounding Pretraining Dynamics: While the protocol successfully detects positional artifacts and establishes their link to inverse scaling, isolating whether these artifacts originate primarily from loss formulations, masking schedules, or teacher-student distillation dynamics requires further controlled pretraining sweeps.
- Future Directions: Extending the unsupervised clustering protocol to temporal video encoders and 3D spatial representations to analyze spatio-temporal coordinate biases, alongside designing lightweight position-debiasing regularizers during pretraining.
Related Work & Insights¶
- vs DINO Dense Feature Analysis (Caron et al., 2021 & Amir et al., 2021): Early studies popularized the claim that self-supervised ViT Keys act as optimal dense descriptors based solely on DINO; this benchmark broadens the scope across 8 SSL paradigms, demonstrating that this property is unique to contrastive learning and completely fails in MIM where Keys are dominated by positional artifacts.
- vs Vision Transformers Need Registers (Darcet et al., 2023): Darcet et al. identified feature peak artifacts and introduced register tokens; this study confirms via aggregate spatial probability maps that registers successfully eliminate positional artifacts, but observes that larger register-equipped models suffer from elevated high-frequency patch inconsistency.
- vs DINOv3 Scaling Analysis (Siméoni et al., 2025): While the original DINOv3 report observed dense prediction degradation at scale and attributed it broadly to patch inconsistency, this work pinpoints a distinct and complementary mechanism: severe absolute boundary artifacts misclassifying image margins into spurious spatial categories.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ An innovative repurposing of non-parametric segmentation as a transparent, tuning-free probe for model representations.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-model and cross-layer evaluation spanning 8 SSL architectures, 2 supervised baselines, and all four canonical ViT components.
- Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative with rigorous information-theoretic metrics seamlessly matched to intuitive visual figures.
- Value: ⭐⭐⭐⭐⭐ Resolves persistent misconceptions across self-supervised learning literature and provides actionable diagnostic benchmarks for scaling vision foundations.