SiPhy: Single-Image Physical Property Reasoning

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/DominoAI-Lab/SiPhy-ECCV-2026
Area: Multimodal VLM
Keywords: Physical Reasoning, Vision Language Model, Single-Image Understanding, Pseudo-Voxelization, Material Alignment

TL;DR

SiPhy tackles the inability of single-image approaches to perceive 3D volume and material semantics by integrating depth-adaptive pseudo-voxel sampling, VLM-driven part-based contrastive alignment, and heaviness-aware thickness refinement, outperforming multi-view reconstruction methods on mass, density, and Young's modulus prediction without requiring multi-view captures.

Background & Motivation

Humans effortlessly infer how heavy, rigid, or flexible an object is from a single glance. A metal mug and a foam cup evoke completely distinct expectations about mass, stiffness, and density even in a static image without motion cues or physical contact. This single-view physical intuition plays a vital role in human perception, supporting motor actions such as robotic grasping, dexterous tool use, and dynamic trajectory planning. However, inferring physical properties from visual observations remains exceptionally difficult because physical quantities are latent attributes: they cannot be observed as mere surface pixels and must instead be inferred from delicate visual cues, 3D spatial volumes, and material semantics.

Existing research has long been trapped in a dilemma. On one hand, multi-view 3D reconstruction frameworks (such as NeRF2Physics, PUGS, and GaussianProperty) leverage neural radiance fields or 3D Gaussian Splatting to construct precise geometry, infusing semantic embeddings into spatial primitives to integrate volumetric mass. While accurate, they rely heavily on dense multi-view captures and time-consuming per-scene optimization, rendering them impractical for real-world scenarios where only a single image is available. On the other hand, existing single-image efforts either treat the problem as 2D pixel-level material recognition (e.g., MINC) while remaining completely oblivious to 3D spatial structures, or directly regress mass from RGB pixels through end-to-end networks (e.g., Image2Mass). Lacking explicit physical knowledge and 3D volumetric grounding, such data-driven regressions suffer from severe generalization failure when encountering novel object categories, textures, and unseen compositions.

The core tension lies in the trade-off between 3D geometric consistency and single-view practicality: multi-view methods achieve structured geometry at the expense of requiring dozens of input views, whereas single-view methods lack explicit 3D volumetric grounding and grounded material commonsense. This paper's angle is that the essential strengths of multi-view reasoning—spatial voxelization, part consistency, and volumetric aggregation—can be faithfully approximated from a single view by pairing monocular metric depth priors with camera geometry. Core idea: propose SiPhy, a 3D-aware vision-language physical reasoning framework that constructs non-overlapping pseudo-voxels via depth-adaptive 2D sampling, aligns visual patches with VLM-derived material candidates via part-constrained contrastive learning, and accurately integrates total mass using heaviness-aware thickness refinement.

Method

Overall Architecture

Given a single RGB image \(I \in \mathbb{R}^{H \times W \times 3}\), SiPhy aims to infer both pixel-level physical property maps (such as material segmentation, mass density, and Young's modulus) and global object-level quantities (such as total mass). The pipeline consists of three interconnected processing stages: first, fine-tuned SiPhy VLM proposes top-\(K\) candidate materials and physical attribute priors for each SAM-derived part mask; second, a 3D-aware sampling module adaptively spaces 2D sample points over the object surface using estimated depth and camera intrinsics to form non-overlapping pseudo-voxels; third, CLIP visual features extracted from each sample are aligned with text embeddings through a part-restricted self-attention and contrastive learning mechanism to generate material likelihoods; finally, physical fields are estimated as expectations, and total mass is integrated using a heaviness-aware thickness refinement module.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Single RGB Image + Monocular Depth"] --> B["SiPhy VLM Part Material Proposal"]
    A --> C["3D-Aware Pseudo-Voxel Sampling<br/>Depth-adaptive spacing s for non-overlapping grid"]
    B --> D["Part-Based Contrastive Alignment<br/>SAM-restricted self-attention + supervised contrastive loss"]
    C --> D
    D --> E["Pixel & Voxel Physical Expectation<br/>Density / Young's modulus / base thickness expectation"]
    E --> F["Heaviness-Aware Thickness Refinement<br/>Heavy/light coarse gating to adjust thickness prior"]
    F --> G["Physical Property Output<br/>Pixel-level property maps + integrated total mass"]

Key Designs

1. 3D-Aware Pseudo-Voxel Sampling: Eliminating Single-View Projection Overlap and Redundancy Predicting global mass requires a spatial partitioning of the object into discrete volumetric units. While multi-view methods achieve this via 3D point cloud voxelization, naive uniform 2D pixel sampling from a single perspective results in extreme distortion: distant surface regions become overly dense and heavily overlap when projected into 3D space, whereas foreground regions become sparse, severely biasing the total volume integration. To resolve this, SiPhy defines a virtual 3D unit cube with predefined edge length \(d\). Leveraging the monocular depth estimate \(z\) (from Depth Anything v2) and camera intrinsics \(f_x, f_y\), the 2D pixel spacing \(s\) is computed adaptively:

\[s = d \cdot \sqrt{\frac{f_x f_y}{z}}\]

Using this depth-adaptive spacing \(s\), the model samples \(N\) non-overlapping 2D points across the object surface. Around each point, a square patch of side length \(s\) is cropped and encoded by a frozen CLIP ViT-B/16 encoder, producing visual tokens \(\{f_i\}_{i=1}^N \in \mathbb{R}^D\). This adaptive formulation ensures that the sampled 2D patches approximate a uniform, non-overlapping surface voxel grid in 3D metric space, forming a geometrically grounded substrate for subsequent volume and mass integration.

2. SiPhy VLM and Part-Based Contrastive Alignment: Enforcing Intra-Part Coherence Directly computing cosine similarity between raw CLIP patch features and free-form text introduces severe visual noise, violating the physical principle that a single structural part usually shares homogeneous material. To inject reliable physical priors, SiPhy first employs a fine-tuned VLM (frozen CLIP visual backbone, Vicuna-7B-v1.5 with LoRA, and a projection MLP). For each SAM-derived part mask, the VLM takes the part mask, cropped part image, full object image, and a GPT-4 generated description to predict top-\(K\) (default \(K=5\)) candidate materials alongside their physical reference values (density, Young's modulus, thickness).

To enforce part-level semantic smoothness, a part-restricted self-attention module is introduced. For each sample \(i\), attention is strictly confined within its SAM-derived neighborhood \(\mathcal{N}(i)\):

\[\tilde{\mathbf{f}}_i = \operatorname{softmax}\left(\frac{(\mathbf{f}_i W_Q)(\mathbf{F}_{\mathcal{N}(i)} W_K)^\top}{\sqrt{d_a}}\right) \mathbf{F}_{\mathcal{N}(i)} W_V\]

The attended feature is concatenated with candidate text embeddings and mapped to embedding \(\mathbf{z}_i\) through an MLP. To pull representations of identical parts together while separating distinct parts without collapsing CLIP's original discriminative geometry, the network is trained with a joint objective combining a supervised contrastive loss \(\mathcal{L}_{\text{CL}}\) and a teacher alignment regularizer \(\mathcal{L}_{\text{align}}\):

\[\mathcal{L}_{\text{CL}} = -\frac{1}{N} \sum_i \frac{1}{|\mathcal{P}(i)|} \sum_{p \in \mathcal{P}(i)} \log \frac{\exp(\operatorname{sim}(\mathbf{z}_i, \mathbf{z}_p)/\tau)}{\sum_a \exp(\operatorname{sim}(\mathbf{z}_i, \mathbf{z}_a)/\tau)}\]

\[\mathcal{L}_{\text{align}} = \frac{1}{NK} D(S_{\text{MLP}}, S_{\text{CLIP}})\]

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CL}} + \lambda \mathcal{L}_{\text{align}}\]

Here \(\mathcal{P}(i)\) denotes points in the same part as \(i\), \(\tau = 0.1\), and \(D(\cdot, \cdot)\) measures the distance between the learned similarity matrix \(S_{\text{MLP}}\) and the raw teacher matrix \(S_{\text{CLIP}}\). Normalizing the resulting similarities yields the material posterior probability matrix \(P \in \mathbb{R}^{N \times K}\).

3. Heaviness-Aware Thickness Refinement: Mitigating Underestimation on Dense Objects For each sampled pseudo-voxel \(i\), physical properties are computed as expectations over candidate attributes: \(\hat{V}_i = \sum_{j=1}^K p_{i,j} V_j\), where \(V_j\) corresponds to density \(\rho_j\), Young's modulus \(E_j\), or thickness \(h_j\). Values across unsampled pixels are propagated via \(k\)-nearest neighbor interpolation. However, the authors uncovered an empirical phenomenon: material classification accuracy exhibits a surprisingly weak correlation with final mass error. Instead, total mass accuracy is dominated by thickness and volume estimation, where standard material-dependent thickness priors severely underestimate heavy and bulky objects.

To overcome this bottleneck, SiPhy introduces the Heaviness-Aware Thickness (HAT) refinement module. Using initial coarse mass estimates, objects are first categorized into "heavy" or "light" classes. This discrete prior conditions a GPT-4 guided adjustment that scales the predicted thickness distribution toward physically plausible volumetric regimes. Aggregating the product of refined thickness, voxel area, and expected density across all pseudo-voxels yields the final object mass, effectively eliminating the massive error typical of monolithic objects under single-view perspective foreshortening.

Loss & Training

Training is divided into two decoupled stages. For SiPhy VLM, the projection module is first pre-trained on ~10k PhysXNet and MVImgNet samples while keeping CLIP and the LLM frozen. In the second stage, Vicuna-7B-v1.5 is fine-tuned using LoRA jointly with the projector; an attention-masked mean-pooled hidden state is passed to a linear classification head optimized via cross-entropy over \(K\) material classes. In the alignment stage, the part-based attention module (\(d_a = 64\)) and MLP are trained using the Adam optimizer with a learning rate of \(1 \times 10^{-4}\) and temperature \(\tau = 0.1\), balancing intra-part clustering and CLIP embedding fidelity.

Key Experimental Results

Main Results

Evaluation is conducted across three benchmarks: ABO-500 test set (real everyday objects with ground-truth mass), MVImgNet-100 (real-world multi-view dataset evaluated from a single front-facing view), and PhysXNet-100 (synthetic dataset with dense per-pixel mass density and Young's modulus annotations). Metrics include Absolute Difference Error (ADE), Absolute Log Difference Error (ALDE), Absolute Percentage Error (APE), and Minimum Ratio Error (MnRE, higher is better, considered the primary scale-invariant metric).

Dataset	Property / Metric	Input View	SiPhy (Ours)	Prev. SOTA / Baseline	Gain
ABO-500	Mass MnRE (↑)	Single (S)	0.58	0.55 (NeRF2Physics, Multi-view M)	+5.5% vs. M-view (+87.1% vs. single-view LLaVA 0.31)
ABO-500	Mass ADE (↓) (kg)	Single (S)	7.78	8.74 (NeRF2Physics, M) / 30.30 (PUGS, M)	-0.96 kg vs. NeRF2Physics (-74.3% error vs. PUGS)
ABO-500	Mass ALDE (↓)	Single (S)	0.74	0.78 (NeRF2Physics, M)	-0.04 log error
PhysXNet-100	Mass Density MAE (↓) (\(\text{kg/m}^3\))	Single (S)	1315	2044 (NeRF2Physics, M)	-35.5% MAE reduction vs. NeRF2Physics (near PUGS 1297)
PhysXNet-100	Young's Modulus MAE (↓) (GPa)	Single (S)	52	68 (NeRF2Physics / PUGS, M)	-23.5% error reduction
PhysXNet-100	Material Seg M-mIoU (↑)	Single (S)	0.31	0.21 (GaussianProperty, M)	+47.6% relative M-mIoU

Ablation Study

Ablations on ABO-500 examine the contributions of the 3D-aware adaptive sampling strategy, part alignment, and the Heaviness-Aware Thickness (HAT) module across overall, heavy, and light object splits.

Variant	Sampling Strategy	HAT Module	ABO-500 Overall ADE (↓)	ABO-500 Overall MnRE (↑)	Heavy Split ADE (↓)	Heavy Split MnRE (↑)
Uniform 2D Sampling	Fixed pixel spacing	Included	11.94	0.39	-	-
Mask Average Pooling	Naive SAM mask averaging	Included	10.23	0.54	-	-
Without HAT Refinement	Depth-adaptive sampling	Excluded	9.84	0.53	18.86	0.57
SiPhy (Full Model)	Depth-adaptive sampling	Included	7.78	0.58	15.59	0.65

Key Findings

• Thickness accuracy dictates mass precision over material classification: Ablation reveals that material errors have minimal direct impact on mass error, whereas thickness errors multiply directly across volume integration. On heavy objects (ABO-500 Heavy), omitting HAT causes ADE to surge to 18.86. Adding HAT brings ADE down to 15.59 (a 17.3% improvement) and boosts MnRE from 0.57 to 0.65.
• Critical necessity of depth-adaptive sampling: Replacing adaptive spacing with uniform 2D grid sampling causes severe voxel overlap and redundancy, causing ADE to spike from 7.78 to 11.94 and MnRE to collapse from 0.58 to 0.39, validating the geometric spacing formulation.
• Real-world hand-object interaction and acoustic synthesis: On HO3D and ARCTIC hand-object interaction datasets featuring real occlusions, SiPhy achieves MnRE of 0.67 and 0.57, outperforming LLaVA (0.58 and 0.53). When serving as material priors for Make-An-Audio, SiPhy corrects visual appearance illusions (such as a metal locker mistaken for wood), generating physically authentic, crisp metallic impact audio spectrograms.

Highlights & Insights

• Bridging 2D pixels and 3D voxels without 3D inputs: By establishing a depth-adaptive inversely scaled spacing formula, SiPhy approximates a metric surface voxelization directly from single-view monocular depth, completely bypassing the massive computational burden of NeRF and 3DGS pipelines.
• Gated thickness rectification: Rather than relying on unconstrained end-to-end regression, the model employs a coarse heavy/light categorization to gate physical thickness bounds, offering an elegant solution to perspective volume collapse.
• Downstream cross-modal physical grounding: Demonstrates that inferring latent mechanical properties (density, stiffness) serves as a potent bridge for audio synthesis and embodied manipulation, establishing single-view physical reasoning as a practical data annotation engine.

Limitations & Future Work

• Vulnerability to specular reflections and transparency: Monocular depth estimation and SAM segmentation degrade significantly under strong specular highlights, mirror reflections, and transparent glass surfaces, leading to distorted voxel spacing and corrupted CLIP features.
• Coarse discreteness of heavy/light categorization: The current HAT module relies on a binary heavy/light threshold, which can produce boundary discontinuities for objects lying near the critical transition threshold. Future work could develop continuous or compositional volumetric priors.
• Blindness to internal hollows or heterogeneous cores: A single RGB-D perspective cannot capture interior cavities or hidden dense internal components (such as a hollow plastic shell weighted with lead), demanding deeper causal physical reasoning and inductive priors.

Related Work & Insights

• vs. NeRF2Physics (CVPR 2024): NeRF2Physics requires dozens of calibrated multi-view images and tens of minutes of neural volumetric training per scene. SiPhy performs feedforward single-image inference, yet surpasses NeRF2Physics in ABO-500 mass MnRE (0.58 vs. 0.55) and PhysXNet-100 Young's modulus MAE (52 vs. 68 GPa).
• vs. PUGS (2025) & GaussianProperty (2024): While 3DGS baselines require full multi-view optimization, SiPhy achieves comparable density MAE (1315 vs. 1297 \(\text{kg/m}^3\)) and superior Young's modulus accuracy from a single view thanks to structured part-contrastive alignment and VLM semantic priors.
• vs. Image2Mass (CoRL 2017) & LLaVA (2023): Image2Mass directly regresses mass using end-to-end CNNs, leading to poor generalization on unseen categories. Standard VLMs like LLaVA produce severe hallucinations without explicit 3D voxelization (MnRE of only 0.31). SiPhy demonstrates that explicit 3D pseudo-voxelization combined with physical attribute expectation is essential for faithful physical reasoning.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly adapts multi-view volumetric physical inference into single-view geometry-grounded reasoning.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation spanning synthetic benchmarks, real multi-view datasets, real-world hand-object interactions, and acoustic generation.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology, clean mathematical formulations, and thorough ablation analyses.
• Value: ⭐⭐⭐⭐⭐ Highly valuable as a lightweight physical annotation engine for embodied AI, robotics manipulation, and multimodal simulation.