Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention¶

Conference: ECCV 2026
OpenReview: https://eccv.ecva.net/virtual/2026/poster/4702
Paper: CVF Open Access
Code: https://github.com/Zhou-Weichen/Geosubprobe
Area: Interpretability
Keywords: Subspace Intervention, Geometric Representations, Self-Supervised Vision Transformers, Task Affinity, Readability Gap

TL;DR¶

This paper introduces a controlled subspace intervention framework utilizing Singular Value Decomposition (SVD) on converged linear probe weights to decode geometric representations in self-supervised ViTs, proving that DINOv2 neatly aligns explicit geometry into a highly compressible low-rank subspace while MAE disperses geometric signals across spatial and channel dimensions.

Background & Motivation¶

Modern dense prediction frameworks increasingly rely on self-supervised Vision Transformers (ViTs) as foundational backbones. Empirical evidence demonstrates that even without explicit 3D supervision, self-supervised representations naturally yield rich single-view geometric priors. However, the academic community has conventionally evaluated these representations via downstream decoding accuracy—a black-box probing paradigm that obscures how geometric primitives are topologically encoded within the high-dimensional latent space. It remains fundamentally uncertain whether geometric signals are aligned into linearly accessible coordinates or dispersed across complex non-linear manifolds.

The central tension lies in the diagnostic ambiguity of black-box probing: a failure in linear decoding conflates the genuine absence of information with decoding inability. When a linear probe yields sub-optimal accuracy, it cannot discern whether geometric features are truly missing, non-linearly entangled within individual tokens, or fragmented across disjoint spatial patches. Existing mechanistic interpretability studies predominantly target semantic concepts, dictionary learning, or individual attention heads, lacking a rigorous, training-free causal methodology to isolate spatial geometric structure and quantify intrinsic linear dimensionality.

To resolve this ambiguity, this work develops a systematic diagnostic hierarchy that first disentangles local non-linear folding from global spatial dispersion through a tiered probing ladder, and subsequently isolates task-aligned geometric directions directly via post-hoc matrix decomposition. Core idea: perform Singular Value Decomposition (SVD) on converged linear probe weights to extract task-aligned bases, and apply controlled orthogonal subspace projections and residual ablations on frozen representations without additional training, thereby quantifying geometric compressibility, cross-layer energy routing, and layer-wise task affinities.

Method¶

Overall Architecture¶

The proposed diagnostic framework consists of two main stages: (1) Readability Gap Analysis, which establishes a three-tier probing hierarchy (Linear Probe, Token-wise 1×1 MLP, and multi-scale DPT Decoder) to decouple local non-linear entanglement from global spatial fragmentation; and (2) Subspace Intervention, which decomposes converged linear weights via SVD to construct task-aligned subspaces, random orthogonal baselines, and orthogonal residual spaces. Features are projected onto these subspaces and evaluated through the frozen linear head, isolating geometric capacity without modifying or retraining model weights.

The complete workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Frozen Backbone Features<br/>Z in R^(N x D)"] --> Tiers["Three-Tier Probing Ladder<br/>Linear vs MLP vs DPT"]
    Tiers --> SVDDec["Probe Weight Decomposition<br/>W = U Sigma V^T for Task-Aligned Basis"]
    SVDDec --> SubInt["Controlled Subspace Projections<br/>Aligned Sk / Random Rk / Orthogonal Residual S_perp"]
    SubInt --> EPerf["Spectral-Weighted Energy & Task Affinity<br/>Layer Energy Ei & Normal/Depth/Semantic Evolution"]
    EPerf --> Out["Topological Representation Diagnostics<br/>Alignment, Compressibility & Routing Guidelines"]

Key Designs¶

1. Three-Tier Probing Ladder: Decoupling Local Non-Linear Entanglement and Spatial Fragmentation

To resolve the conflation between feature absence and decodability, the framework introduces three probes of strictly controlled capacity and receptive field. The baseline Linear Probe employs a $1 \times 1$ projection mapping frozen patch tokens directly to target depth bins ($C=256$), quantifying explicit geometric cues readable without inter-patch communication. The intermediate 1×1 MLP Probe introduces point-wise non-linear activations (GELU) while strictly preserving the token-wise receptive field; the performance gap between Linear and MLP probes isolates local non-linear entanglement. Finally, the DPT Decoder incorporates multi-scale feature aggregation across spatial tokens, where its performance gain over the MLP probe isolates spatial fragmentation—revealing whether the backbone scatters geometric primitives across spatial tokens that require global context to assemble.

2. Probe Weight Decomposition: Training-Free Task-Aligned Subspace Projection

Given a converged linear probe weight matrix $\mathbf{W}^{(l)} \in \mathbb{R}^{C \times D}$, its mapping rank is upper-bounded by the target dimensionality $C \ll D$. Performing SVD directly yields: $$\mathbf{W}^{(l)} = \mathbf{U}^{(l)} \mathbf{\Sigma}^{(l)} (\mathbf{V}^{(l)})^\top$$ The right singular vectors in $\mathbf{V}^{(l)} \in \mathbb{R}^{D \times C}$ form an orthonormal basis spanning the task-aligned representation space. Selecting the top-$k$ principal directions spans the low-rank subspace $\mathcal{S}_k^{(l)} = \text{span}\{\mathbf{v}_1^{(l)}, \dots, \mathbf{v}_k^{(l)}\}$. The original feature tensor is then orthogonally projected via $\tilde{\mathbf{Z}}_k^{(l)} = \mathbf{Z}^{(l)} \mathbf{V}_k^{(l)} (\mathbf{V}_k^{(l)})^\top$. Evaluating $\tilde{\mathbf{Z}}_k^{(l)}$ directly through the locked linear head ensures that performance variations stem strictly from the geometric capacity of the chosen $k$-dimensional subspace. Two rigorous control subspaces are paired for validation: a random orthonormal subspace $\mathcal{R}_k$ and the orthogonal residual complement $\mathbf{Z}_{\text{res}}^{(l)} = \mathbf{Z}^{(l)} - \tilde{\mathbf{Z}}_k^{(l)}$, verifying that explicit geometric signals are strictly concentrated along the top singular directions.

3. Spectral-Weighted Energy Routing and Layer-Wise Task Affinity: Mapping Representations from Geometry to Semantics

When evaluating multi-layer concatenated features with global weight $\mathbf{W}_{\text{global}} \in \mathbb{R}^{C \times 4D}$ across layers $l_6, l_{12}, l_{18}, l_{24}$, each right singular vector $\mathbf{v}_m \in \mathbb{R}^{4D}$ is partitioned into four layer-specific blocks $\mathbf{v}_m = [(\mathbf{v}_m^{(l_6)})^\top, (\mathbf{v}_m^{(l_{12})})^\top, (\mathbf{v}_m^{(l_{18})})^\top, (\mathbf{v}_m^{(l_{24})})^\top]^\top$. The spectral-weighted energy contribution $E_i$ for layer $i$ is formalized as: $$E_i = \frac{\sum_{m=1}^C \sigma_m^2 \|\mathbf{v}_m^{(l_i)}\|_2^2}{\sum_{j=1}^4 \sum_{m=1}^C \sigma_m^2 \|\mathbf{v}_m^{(l_j)}\|_2^2}$$ where $\sigma_m$ weights each singular direction by its explanatory variance. Concurrently, identical linear probes are trained on surface normal estimation, depth estimation, and 40-class semantic segmentation. Tracking normalized accuracy trajectories across network depths reveals the empirical transition of task affinities, tracing the shift from low-level local geometry to mid-level spatial layout and terminal semantic abstraction.

Key Experimental Results¶

Main Results¶

On the NYU Depth V2 benchmark using ViT-Large backbones, the three-tier probing hierarchy evaluates geometric representation accessibility and isolates entanglement/fragmentation gaps across self-supervised paradigms:

Backbone	Pre-training Objective	Linear (SA-$\delta_1$ ↑)	1×1 MLP (SA-$\delta_1$ ↑)	DPT (SA-$\delta_1$ ↑)	Local Entanglement Gap	Spatial Fragmentation Gap
DINOv2-Large	Self-Distillation	0.9157	0.9325	0.9483	+0.0168	+0.0158
iBOT-Large	Hybrid MIM + Distill	0.8198	0.8376	0.8524	+0.0178	+0.0148
MAE-Large	Masked Autoencoding	0.6033	0.6390	0.7022	+0.0357	+0.0632

Ablation Study¶

Spectral-weighted energy contribution $E_i$ (%) across four representative network depths ($l_6, l_{12}, l_{18}, l_{24}$) in global linear probe weights (baseline uniform distribution is 25.0%):

Model	Layer 6 Energy ($E_6$)	Layer 12 Energy ($E_{12}$)	Layer 18 Energy ($E_{18}$)	Layer 24 Energy ($E_{24}$)	Representation Routing Profile
DINOv2-Large	17.2%	35.8%	36.7%	10.3%	Strongly concentrated in intermediate layers ($l_{12}+l_{18} > 72\%$), steep drop at terminal layer
iBOT-Large	26.9%	28.4%	22.4%	22.3%	Balanced around the 25% uniform baseline, preserving global and local cues
MAE-Large	19.5%	28.4%	32.7%	19.4%	Diffusely distributed across layers, requiring global cross-depth aggregation

Key Findings¶

Alignment vs. Dispersion Across SSL Objectives: DINOv2 exhibits exceptional linear accessibility (0.9157 SA-$\delta_1$), where adding non-linear MLP (+0.0168) or global DPT (+0.0158) yields minimal gains, demonstrating that self-distillation rectifies geometric primitives into explicit linear coordinates. In stark contrast, MAE suffers from severe spatial dispersion; its DPT decoder achieves a +0.0989 leap over linear probing, with spatial fragmentation (+0.0632) dominating local entanglement (+0.0357).
Extreme Low-Rank Compressibility: When projecting features onto random subspaces $\mathcal{R}_k$ or residual spaces $\mathcal{S}_k^\perp$, performance immediately collapses to the noise floor (SA-$\delta_1 < 0.18$). Conversely, on task-aligned subspaces, MAE recovers >98% of its linear capacity at rank $k=32$, while DINOv2 reaches saturation at $k \ge 64$. Explicit geometric representations are remarkably compressible, indicating dense prediction heads can be constrained to low-rank manifolds.
Layer-Wise Task Affinity Transition: In DINOv2, surface normal estimation peaks early at Layer 18, monocular depth estimation peaks at Layer 21, and semantic segmentation reaches its maximum at Layer 24. Rather than losing overall capacity, terminal layers deliberately transition from metric geometry toward semantic abstraction.

Highlights & Insights¶

Post-Hoc Training-Free Causal Probing: Decomposing converged linear weights via SVD isolates task-aligned coordinate frames without requiring secondary probe optimization, avoiding optimization artifacts and hyperparameter bias.
Unpacking Black-Box Probing Ambiguity: The combination of low-rank subspace interventions, random controls, and orthogonal residuals provides an unambiguous diagnostic toolkit, answering precisely along which dimensional directions explicit geometry resides.
Actionable Guidelines for Dense Predictor Design: Demonstrating that terminal ViT features are suboptimal for geometric prediction provides rigorous empirical justification for multi-level feature tapping (intermediate layers for geometry, terminal layers for semantics) and lightweight low-rank predictive heads.

Limitations & Future Work¶

Linear Boundaries on Non-Linear Manifolds: SVD subspace intervention primarily characterizes linearly accessible structures and cannot unfold deeply folded, highly non-linear geometric representations.
Dataset Constraint for Multi-Task Ground Truth: Controlled task affinity analysis strictly necessitates co-registered ground truth for depth, surface normals, and semantics, which limits large-scale evaluation primarily to indoor datasets like NYUv2.
Scale Validation Limits: Empirical evaluations focused on ViT-Base and ViT-Large architectures due to compute limits, leaving the subspace dynamics of billion-parameter models (such as ViT-Giant) for future exploration.

vs. Conventional Linear Probing (Alain & Bengio 2016): Standard probes output scalar accuracy, conflating feature absence with decodability; this work dissects probe weight topology via SVD to causally isolate directional geometric subspaces.
vs. Visual Representation Studies (Amir et al. 2021, Banani et al. 2024): Previous works qualitative visualize features or rely on black-box fine-tuning; this paper establishes formal readability gap metrics and spectral energy distribution formulas, uncovering structural differences between self-distillation and masked image modeling.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ An elegant, mathematically grounded training-free subspace intervention methodology.
Experimental Thoroughness: ⭐⭐⭐⭐☆ Thorough analysis across three SSL paradigms, multi-scale depths, and multi-task affinities on NYUv2 with stability ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, crisp mathematical definitions, and cohesive deductive narrative.
Value: ⭐⭐⭐⭐⭐ Offers crucial architectural insights for feature selection and lightweight decoder design in dense computer vision tasks.