title: >- [Paper Note] Φeat: Physically-Grounded Material Feature Representation description: >- [ECCV 2026][Self-Supervised Learning][Material Representation] A material-grounded vision backbone fine-tuned from DINOv3 using physically based rendering under controlled lighting and geometry variations to encode intrinsic reflectance and mesostructure. tags: - ECCV 2026 - Self-Supervised Learning - Material Representation - Physical Priors - DINOv3 - Contrastive Learning date: 2026-09-19 content_hash: b6b4db71a3ce91dc
Φeat: Physically-Grounded Material Feature Representation¶
Conference: ECCV 2026
Paper: ECCV 2026 Official Page
Area: Self-Supervised Learning
Keywords: Material Representation, Self-Supervised Learning, Physical Priors, DINOv3, Contrastive Learning
TL;DR¶
Addressing the limitation of semantic foundation models that overlook physical material reflectance and mesostructure, Φeat fine-tunes DINOv3 using large-scale physically based renderings under controlled geometry and illumination variations, yielding representations invariant to extrinsic factors while highly sensitive to intrinsic material identity.
Background & Motivation¶
Modern visual representation learning has advanced dramatically through self-supervised foundation models, particularly the DINO family. These models map pixel spaces into robust semantic manifolds where high-level object category identity remains invariant across viewpoints and styles. However, because standard self-supervised training exclusively optimizes for high-level semantic abstractions, these representations struggle to isolate the fundamental physical factors that determine a real-world scene's appearance. In computer vision tasks such as intrinsic image decomposition, physical material acquisition, and robotic tactile-visual interaction, correctly estimating high-level semantic categories (e.g., "chair") is frequently less informative than determining intrinsic physical material properties (e.g., "polished oak"). Standard backbones invariably conflate distinct materials based on macro-object context or degrade under non-uniform illumination.
This limitation stems from a fundamental conflict between conventional self-supervised data augmentation and the physics of material appearance. Standard pretraining protocols heavily rely on photometric augmentations, such as color jittering and solarization. These aggressive image-space operations destroy delicate physical optical cues—such as specular highlights, surface roughness, and geometric mesostructures—forcing models to ignore intrinsic material properties to preserve object-level invariance. Conversely, existing material-centric approaches depend on dense, labor-intensive pixel-level annotations or narrow acquisition settings (e.g., co-located flash photography), severely limiting their generalizability across diverse real-world environments.
This paper addresses this bottleneck by replacing arbitrary image-space augmentations with physically grounded weak supervision. By rendering identical physical materials across varying macro-geometries and diverse high dynamic range (HDR) illumination conditions, the authors construct "physical triplets" where intrinsic properties remain invariant while extrinsic context fluctuates. Core idea: fine-tune a self-supervised ViT using physically rendered triplets under controlled lighting and geometry variations, steering the representation from high-level semantics toward intrinsic material properties while preserving dense spatial structure.
Method¶
Overall Architecture¶
Φeat aims to produce visual embeddings that are invariant to extrinsic perturbations (macro-geometry, surface orientation, and environment lighting) while remaining exquisitely sensitive to intrinsic physical attributes (surface reflectance and mesostructure). The training pipeline samples rendering pairs of identical materials under distinct physical conditions from a curated synthetic dataset of approximately one million high-fidelity path-traced images. These images undergo multi-crop extraction into global and local views before passing into a ViT student network and an exponential moving average (EMA) teacher network. The model optimizes five joint objectives: image-level Sinkhorn prototype assignment, patch-level masked latent reconstruction, KoLeo feature dispersion regularization, Gram anchoring, and a cross-material InfoNCE contrastive loss.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Physical Rendering Pairs of Same Material<br/>Controlled lighting & geometry variations"] --> B["Semantically Aligned Synthetic Data Generation<br/>Substance 3D procedural assets + matched templates + HDR path tracing"]
B --> C["Multi-Crop Multi-Scale View Generation<br/>Global (40-100%) + Local (10-40%) crops"]
C --> D["ViT Encoder & Structured Feature Extraction<br/>DINOv3 ViT-B/16 + RoPE + Register Tokens"]
E["Multi-Level Physical Constraint Joint Optimization<br/>Sinkhorn prototype alignment + Masked patch reconstruction + Contrastive loss"]
D --> E
E --> F["Intrinsic Material-Aware Representation<br/>Decoupled from lighting/shape / sensitive to reflectance & mesostructure"]
Key Designs¶
1. Semantically Aligned Synthetic Data Generation: eliminating physically implausible pairings Prior datasets for inverse rendering or material similarity typically combine 3D shapes, textures, and illumination environments at random. Such naive pairings often produce severe perceptual artifacts (e.g., applying rigid corkboard textures to intricately folded garments), corrupting realistic mesostructure cues. The authors leverage the Adobe Substance 3D Assets repository, spanning over 9,500 procedural materials across 21 appearance categories (including wood, stone, marble, fabric, metal, and plastic). By varying procedural parameters across artist-crafted presets, they synthesize roughly 36,000 unique sets of Physically Based Rendering (PBR) parameter maps. Crucially, materials are paired only with semantically compatible geometric templates and rendered under 4 randomly selected HDR environment maps from a pool of 20 using Monte Carlo GPU path tracing (128 samples per pixel with displacement tessellation and denoising), compiling a corpus of approximately one million photorealistic renderings.
2. ViT Encoder & Structured Feature Extraction: uniting global material identity with local dense structure Φeat adopts the Vision Transformer backbone from DINOv3 using a ViT-B configuration with a patch size of \(16 \times 16\). Input images are tokenized into non-overlapping patch embeddings alongside a prepended [CLS] token and a dedicated set of learnable register tokens. These register tokens act as high-capacity global memory buffers that prevent background tokens from dominating feature norms and creating spatial artifacts. Rotary Positional Embeddings (RoPE) are integrated into the self-attention blocks, naturally facilitating sequence length extrapolation and flexible input resolutions during inference. The backbone outputs two complementary feature levels: the global [CLS] embedding representing overall material identity and dense patch tokens capturing fine-grained physical mesostructure.
3. Multi-Level Physical Constraint Joint Optimization: disentangling extrinsic factors while preserving separability To instill physical invariance without inducing representation collapse or erasing inter-material distinctions, Φeat unifies five complementary objectives: - Sinkhorn Image-Level Alignment (\(\mathcal{L}_{\text{image}}\)): Global and local views from student and teacher networks are mapped onto \(K\) learnable prototypes. Teacher probabilities \(q_k(v_t)\) are balanced via Sinkhorn-Knopp normalization over the batch, supervising the student softmax predictions \(p_k(v_s)\): $\(\mathcal{L}_{\text{image}} = -\frac{1}{|V_t|\,|V_s|}\sum_{v_t\in V_t}\sum_{v_s\in V_s}\sum_{k=1}^{K}q_k(v_t)\log p_k(v_s)\)$ - Masked Patch-Level Latent Reconstruction (\(\mathcal{L}_{\text{iBOT}}\)): Between 10% and 50% of student patch tokens are masked out. A lightweight projection head \(h_\theta\) forces the masked embeddings to reconstruct the corresponding unmasked patch embeddings from the teacher: $\(\mathcal{L}_{\text{iBOT}} = \frac{1}{M}\sum_{m=1}^{M}\|h_\theta(p^s_m) - p^t_m\|_2^2\)$ - Feature Dispersion Regularization (\(\mathcal{L}_{\text{KoLeo}}\)): Based on the Kozachenko-Leonenko differential entropy estimator, this loss maximizes nearest-neighbor pairwise distances \(\rho_i\) across normalized global student tokens in the batch, promoting uniform coverage across the hypersphere: $\(\mathcal{L}_{\text{KoLeo}} = -\frac{1}{B}\sum_{i=1}^{B}\log(\rho_i + \varepsilon)\)$ - Second-Order Gram Anchoring (\(\mathcal{L}_{\text{Gram}}\)): During the final 2,000 iterations, patch-level second-order correlation matrices from a frozen Gram teacher \(P_G\) supervise the student patch matrix \(P_s\) via Frobenius norm minimization, stabilizing spatial relations. - Cross-Material Contrastive Loss (\(\mathcal{L}_{\text{contrast}}\)): Because self-supervised alignment across varied physical renderings can blur boundaries between different materials, an in-batch InfoNCE loss is applied to normalized global student vectors. For anchor \(z_i\), views of the identical material form positive set \(\mathcal{P}(i)\), while all other material instances serve as negatives: $\(\mathcal{L}_{\text{contrast}} = -\frac{1}{N}\sum_{i=1}^{N}\log\frac{\sum_{j\in\mathcal{P}(i)}\exp(\operatorname{sim}(z_i,z_j)/\tau)}{\sum_{k\neq i}\exp(\operatorname{sim}(z_i,z_k)/\tau)}\)$ The composite objective balances all terms: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{image}} + \lambda_p \mathcal{L}_{\text{iBOT}} + \lambda_k \mathcal{L}_{\text{KoLeo}} + \lambda_g \mathcal{L}_{\text{Gram}} + \lambda_c \mathcal{L}_{\text{contrast}}\)$
Loss & Training¶
The backbone is initialized from pretrained DINOv3 ViT-B/16 checkpoints and optimized using AdamW with a base learning rate of \(0.001\) and weight decay of \(0.05\). The teacher network is updated via an exponential moving average (EMA) scheduled with a cosine trajectory from \(0.996\) to \(1.0\). The batch size is set to 512 render pairs, with each pair comprising two distinct renders of the same material under different environmental conditions. Multi-crop augmentation samples 2 global crops (40%–100% scale) and 8 local crops (10%–40% scale) per view. Loss hyperparameters are configured as \(\lambda_p = 1.0\), \(\lambda_k = 0.1\), \(\lambda_g = 0.7\), and \(\lambda_c = 0.25\), with temperatures \(\tau_s = 0.1\) and \(\tau = 0.1\), and \(\varepsilon = 10^{-6}\). Pretraining is executed across 16 NVIDIA A100 GPUs using mixed precision over 10,000 iterations (~100 GPU hours), with Gram anchoring activated for the final 2,000 steps.
Key Experimental Results¶
Main Results¶
Φeat is evaluated against leading general-purpose visual foundation models on two core benchmarks: dense patch-level material selection (DuMaS benchmark, thresholding reference-patch cosine similarity maps at 0.5 against ground-truth segmentation masks) and non-parametric feature separability (\(k\)-NN retrieval across 972 material classes in 24 lighting/geometry variations, totaling 23,328 test images).
| Model | Supervision Paradigm | Selection \(\ell_1 \downarrow\) | Selection mIoU \(\uparrow\) | Selection F1 \(\uparrow\) | \(k\)-NN Acc. \(\uparrow\) | \(k\)-NN Prec. \(\uparrow\) | \(k\)-NN F1 \(\uparrow\) |
|---|---|---|---|---|---|---|---|
| Materialistic (Sharma et al.) | Task-Specific Supervised (Upper Bound) | 5.7 | 85.8 | 90.6 | — | — | — |
| Guerrero et al. 2025 | Task-Specific Supervised (Upper Bound) | 3.0 | 89.6 | 93.5 | — | — | — |
| CLIP (ViT-B) | Weakly Supervised (VLM) | 55.7 | 21.6 | 31.6 | 39.9 | 36.6 | 31.2 |
| SigLIP 2 (ViT-B) | Weakly Supervised (VLM) | 53.1 | 24.4 | 35.2 | 33.5 | 30.8 | 25.1 |
| PE-core | Weakly Supervised (VLM Features) | 37.3 | 40.5 | 54.6 | 44.2 | 37.0 | 33.9 |
| PE Spatial | Agglomerative (Dense Feature) | 25.2 | 61.4 | 72.3 | 50.3 | 44.0 | 39.0 |
| RADIOv3 | Agglomerative (Multi-Teacher Distillation) | 25.6 | 63.5 | 74.7 | 35.9 | 32.9 | 28.8 |
| DINOv2 | Self-Supervised (DINO + iBOT) | 26.7 | 56.5 | 69.3 | 64.3 | 56.9 | 53.3 |
| DINOv3 | Self-Supervised (Baseline Backbone) | 27.6 | 59.7 | 72.2 | 66.9 | 60.0 | 55.4 |
| Φeat (Ours) | Physically Grounded Weak Supervision | 25.4 | 72.0 | 80.8 | 71.8 | 75.0 | 66.6 |
Ablation Study¶
A step-by-step ablation starting from the base DINOv3 encoder validates the critical necessity of multi-render physical variations and explicit cross-material contrastive learning.
| Configuration | Selection \(\ell_1 \downarrow\) | Selection mIoU \(\uparrow\) | Selection F1 \(\uparrow\) | \(k\)-NN Acc. \(\uparrow\) | Core Observation & Mechanism |
|---|---|---|---|---|---|
| DINOv3 (Base Backbone) | 27.6 | 59.7 | 72.2 | 66.9 | Baseline semantic encoder with respectable global clustering but weak physical localization |
| + single render (Material SSL) | 26.5 | 69.4 | 78.1 | 34.5 | Elevates local material sensitivity but causes severe collapse of global separability (Acc drops 32.4%) |
| + multi render (Physical Variations) | 26.1 | 69.9 | 79.2 | 51.3 | Exposure to varied illumination and viewpoints partially restores cluster stability |
| + contrastive (Cross-Material InfoNCE) | 25.4 | 72.0 | 80.8 | 71.8 | Enforces compact intra-material clusters and clean margins (+12.3% mIoU, +4.9% Acc over DINOv3) |
Robustness & Semantic Tradeoff Analysis¶
Invariance to extrinsic disturbances is quantified via average pairwise Hamming distance of \(k\)-NN predictions across identical materials under varied environments: - Illumination Perturbation Hamming Distance: CLIP scores \(0.403\), DINOv2 scores \(0.284\), DINOv3 achieves \(0.240\), and Φeat reaches the lowest distance of 0.221. - Geometry Perturbation Hamming Distance: CLIP scores \(0.534\), DINOv2 scores \(0.435\), DINOv3 achieves \(0.365\), and Φeat achieves the lowest distance of 0.305. - Semantic Specialization Tradeoff (Linear Probing): On ImageNet classification (Top-1 Acc), ADE20K segmentation (mIoU), and Cityscapes segmentation (mIoU), DINOv3 scores \(83.5\%\), \(51.5\%\), and \(71.5\%\) respectively, whereas Φeat obtains \(80.1\%\), \(46.2\%\), and \(66.1\%\). The model sacrifices modest semantic classification performance (a \(3.4\%\) drop on ImageNet) to secure dramatic enhancements in physical material reasoning.
Key Findings¶
- Fine-tuning foundation encoders on material data using standard single-view self-supervision induces severe feature manifold degradation (\(k\)-NN accuracy plunging from \(66.9\%\) to \(34.5\%\)), as crops from the same image fail to establish discriminative inter-class margins.
- Incorporating an explicit in-batch InfoNCE contrastive objective (\(\mathcal{L}_{\text{contrast}}\)) is indispensable for restoring cluster separability, propelling \(k\)-NN accuracy from \(51.3\%\) to \(71.8\%\) and substantially tightening patch-level boundary delineation.
- Vision-language models (CLIP and SigLIP 2) struggle on dense material selection (mIoU between \(21.6\%\) and \(24.4\%\)), demonstrating that language supervision strongly couples representations to semantic object identities while remaining blind to optical reflectance properties.
Highlights & Insights¶
- Physical Multi-View Rendering as Principled Supervision: Rather than relying on ungrounded photometric color jittering that erases subtle reflectance phenomena, Φeat leverages GPU Monte Carlo path tracing across varied geometry and lighting to directly supervise physical invariance.
- Semantically Governed Material-Geometry Pairing: Avoiding arbitrary random object-material combinations preserves contextual realism, ensuring that procedural materials reside on mechanically and visually plausible geometric supports.
- Versatile Zero-Shot Material Backbone: Without task-specific classification heads or specialized decoder modules, raw patch embeddings from Φeat support video-rate material tracking and unsupervised K-means clustering out of the box.
Limitations & Future Work¶
- Lack of Explicit Latent Disentanglement: The learned feature representation bundles reflectance, micro-roughness, and mesostructure into an entangled latent space, preventing direct scalar manipulation of individual PBR channels like albedo or metallic properties.
- Synthetic-to-Real Domain Discrepancy: Pretraining exclusively leverages synthetic Substance 3D assets. Addressing sensor noise, lens blur, and complex in-the-wild illumination via unsupervised real-world domain adaptation remains an open problem.
- Subsurface Scattering and Volumetric Transport: The underlying rendering pipeline relies primarily on microfacet surface reflectance models, leaving more complex optical phenomena—such as translucency, refraction, and volumetric subsurface scattering—underrepresented.
Related Work & Insights¶
- vs DINOv3 / DINOv2: While the DINO series clusters features along semantic object boundaries, Φeat reconfigures the feature manifold to group regions by intrinsic reflectance and surface texture across object instances.
- vs PE Spatial / RADIOv3: Agglomerative models prioritize dense geometric prediction over semantic grouping but still conflate physical material boundaries with macro-shape cues, underperforming Φeat in fine-grained material classification (\(35.9\% \sim 50.3\%\) vs. \(71.8\%\)).
- vs Materialistic / Guerrero et al.: These fully supervised dense segmentation networks achieve slightly higher raw selection numbers by relying on dense pixel-level supervision and specialized heads. Φeat serves as a general-purpose, label-free backbone that can substitute DINOv3 within any downstream pipeline.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ An elegant formulation replacing synthetic image corruptions with physically based rendering triplets to induce genuine physical invariance in foundation backbones.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across dense selection, non-parametric clustering, illumination/shape robustness, and semantic utility tradeoffs.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, rigorous mathematical definitions, and compelling qualitative visualizations.
- Value: ⭐⭐⭐⭐⭐ Provides a foundational drop-in backbone for material acquisition, neural inverse rendering, and embodied robotic manipulation.