AutoPhyX: Automatic Text-Condition Physics Property Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://hb-pencil-zero.github.io/AutoPhyX
Area: 3D Vision
Keywords: physics simulation, physical parameter prediction, cross-modal modulation, 3D voxel field, text conditioning

TL;DR¶

AutoPhyX introduces the first text-conditioned feed-forward framework for 3D physical property prediction, leveraging volume-rendered OpenCLIP voxels and FiLM cross-modal modulation to regress dense Young's modulus, Poisson's ratio, and density fields in a single forward pass, resolving the fundamental physical ambiguity of purely visual methods.

Background & Motivation¶

High-fidelity 4D physical simulation serves as a core cornerstone for interactive digital environments, embodied AI robotics policy training, and virtual reality. Whether utilizing Finite Element Methods (FEM) for elastoplastic solid deformation or Material Point Methods (MPM) for fracture and fluid-structure dynamics, synthesizing realistic physical behaviors strictly requires accurate physical parameters, specifically Young's modulus $E$, Poisson's ratio $\nu$, and density $\rho$. However, assigning these physical parameters has historically relied on manual trial-and-error design by human experts. When dealing with complex scenes characterized by multi-part, spatially varying materials, manual tuning becomes entirely unscalable.

To automate material acquisition, existing methodologies confront critical bottlenecks. On one hand, test-time optimization methods (such as PhysDreamer, DreamPhysics, and OmniPhysGS) utilize differentiable physics engines and video diffusion motion distillation to iteratively update material fields; nevertheless, they suffer from prohibitive computational costs—often demanding hours of per-scene optimization—and memorize single scenes without generalizing. On the other hand, recent feed-forward approaches like Pixie and VoMP achieve rapid inference by directly regressing material parameters from visual representations. However, relying purely on visual cues introduces severe appearance ambiguity: objects with identical visual appearances can possess drastically different mechanical behaviors, such as a flexible rubber toy versus a stiff plastic figurine, or resilient fresh plant stems versus brittle dry wood. Pure vision offers no explicit semantic mechanism to disambiguate such physical discrepancies.

This paper's angle of attack is that natural language conveys direct, high-level material priors and user intent that can resolve visual ambiguity. The core idea is to build AutoPhyX, a text-conditioned feed-forward framework based on 3D voxel fields that lifts multi-view features via volume rendering to avoid occlusion contamination, fills internal voids via hierarchical ray casting, and modulates intermediate 3D U-Net features via FiLM cross-modal conditioning to regress dense, simulation-ready physical fields in a single 2-second forward pass.

Method¶

Overall Architecture¶

The AutoPhyX pipeline proceeds in two primary phases: robust 3D feature field extraction with internal volumetric filling, followed by text-conditioned cross-modal physical parameter regression. Given multi-view observations and a textual description specifying physical characteristics, the system first constructs an occlusion-free 3D OpenCLIP feature grid via differentiable volume rendering, and employs hierarchical ray-casting inside-outside checks to populate unobservable interior voxels with solid material semantics. Next, textual descriptions are encoded via a frozen OpenCLIP text encoder and passed through dedicated MLPs to predict channel-wise affine modulation parameters for the 3D U-Net intermediate layers. Finally, the decoder directly outputs a dense volumetric grid of physical parameters $(E, \nu, \rho)$ across all occupied voxels, which can be plugged directly into downstream solvers including 3D Gaussian Splatting (3DGS) with MPM, or tetrahedral meshes with FEM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multi-view Images + Text Prompt"] --> B["3D Feature Extraction: Volume Rendering<br/>Accumulated transmittance suppresses occlusion"]
    B --> C["Hierarchical Ray-Casting: Voxel Filling<br/>6-dir boundary check + even-odd topology"]
    C --> D["Cross-Modal FiLM Modulation: 3D U-Net<br/>Text dynamically modulates channel scale & shift"]
    D --> E["Voxel Physical Field Output: Regress E / ν / ρ<br/>Plug-and-play into MPM / FEM Solvers"]

Key Designs¶

1. Differentiable Volume Rendering and Hierarchical Even-Odd Internal Filling: Eliminating Geometric Contamination for Solid Dynamics

Prior feature lifting approaches (such as VoMP) rely on naive multi-view averaging, which ignores line-of-sight occlusion and causes severe feature contamination (e.g., foreground leaves erroneously projecting green semantic features onto an occluded tree trunk behind them). To resolve this, AutoPhyX treats 2D OpenCLIP features as continuous feature radiance $\Phi: \mathbb{R}^3 \to \mathbb{R}^D$ and performs end-to-end differentiable volume rendering guided by pre-trained NeRF density fields $\sigma$: $$ \hat{F}(r) = \int_{t_n}^{t_f} T(t) \sigma(r(t)) \Phi(r(t)) dt, \quad \text{where } T(t) = \exp\left(-\int_{t_n}^t \sigma(r(s)) ds\right) $$ Because the accumulated transmittance $T(t)$ naturally decays to zero once a surface is penetrated, occluded regions are automatically masked, ensuring that 3D surface features are supervised only from genuinely visible viewpoints. Furthermore, continuous physics engines (e.g., MPM/FEM stress propagation) mandate volumetric interior properties rather than hollow surfaces. AutoPhyX introduces a two-tier interior voxel filling strategy: Condition 1 conducts coarse enclosure detection via six-direction boundary checking, while Condition 2 executes topological refinement using the even-odd intersection rule across multiple rays cast to infinity. The successfully identified interior voxels then inherit semantic features via nearest-neighbor interpolation from the surface, establishing a structurally solid and semantically coherent volumetric substrate.

2. Text-Conditioned FiLM Cross-Modal Modulation: Resolving Appearance Ambiguity in Physical Parameters

Visual appearance alone cannot differentiate objects sharing identical textures but possessing opposite stiffness or density. To introduce explicit physical priors, AutoPhyX encodes the user's natural language description using a frozen OpenCLIP text encoder into semantic embedding $e_{\text{text}}$. Because both the 3D visual voxels and the textual embeddings share the pre-aligned OpenCLIP latent space, no artificial cross-modal projection layer is required. AutoPhyX passes $e_{\text{text}}$ through dedicated multi-layer perceptrons (MLPs) to predict layer-specific channel-wise affine transformation parameters—scaling factor $\gamma_i$ and shifting factor $\beta_i$—for the $i$-th layer of the 3D U-Net: $$ F_i' = \gamma_i(e_{\text{text}}) \cdot F_i + \beta_i(e_{\text{text}}) $$ By interleaving these FiLM modulation layers across multiple scales of the 3D U-Net, linguistic cues dynamically amplify or suppress specific visual feature channels. Consequently, when presented with prompts such as "flexible rubber" versus "rigid brittle cardboard", the network modulates intermediate feature distributions to alter downstream stiffness regression without perturbing the underlying 3D geometry.

3. Part-Controllable Text2PhyX Dataset Generation Pipeline: Grounding with Engineering Databases and Simulation Checks

Training a generalizable feed-forward model requires large-scale 3D assets paired with dense, part-aware physical properties and textual descriptions. The authors establish an automated data generation pipeline: multi-view images are rendered across 15 upper-hemisphere viewpoints and fed into a VLM to generate candidate part keyword lists, which are then segmented onto 3D OpenCLIP voxels via text-visual cosine similarity across 5 candidate samplings to avoid atypical vocabulary failures. Next, the VLM generates diverse textual descriptions for subsets of parts and queries authoritative engineering repositories (MatWeb, Wikipedia, and The Engineering Toolbox) to retrieve genuine physical parameter ranges (e.g., bounding mahogany density between 800 and 1300 $\text{kg/m}^3$). Ground-truth parameters are then sampled within these validated bounds. Finally, a 100-frame physical simulation is executed for each asset to automatically filter out samples exhibiting visual artifacts, inaccurate part boundaries, or dynamics inconsistent with the prompt. The resulting Text2PhyX benchmark comprises 1,700 3D assets (1,000 Objaverse meshes and 700 SAM 3D Gaussian Splats) providing 13,600 annotated asset-text-physics triplets.

Loss & Training¶

Supervision is applied strictly over occupied solid voxels $\mathcal{G}$. With target physical properties $\mathcal{K} = \{E, \nu, \rho\}$, the volumetric loss function is formulated as: $$ \mathcal{L}{\text{phys}} = \frac{1}{N(p)|^2 $$ where }}} \sum_{p \in \mathcal{G}} \sum_{k \in \mathcal{K}} |\hat{k}(p) - k_{\text{GT}$N_{\text{occ}}$ denotes the total count of valid occupied voxels, and $\hat{k}(p)$ and $k_{\text{GT}}(p)$ represent predicted and ground-truth parameters at voxel $p$. Because Young's modulus $E$ and density $\rho$ span multiple orders of magnitude across materials, they are transformed into $\log_{10}$ space, and all $\log E$, $\nu$, and $\log \rho$ values are normalized into the range $[-1, 1]$. AutoPhyX is trained using the Adam optimizer with an initial learning rate of $10^{-4}$ on 2 NVIDIA RTX A800 GPUs for 100 epochs with a batch size of 4 per GPU, taking approximately 2 days.

Key Experimental Results¶

Main Results¶

Evaluated on the Text2PhyX test set (125 objects, 1,000 testing samples), performance is measured across physical simulation rendering quality (PSNR, SSIM, LPIPS) and parameter prediction error (average physical error, $\log E$ error, $\nu$ error, and $\log \rho$ error):

Method	PSNR ↑	SSIM ↑	LPIPS ↓	Avg. Phys. err ↓	$\log E$ err ↓	$\nu$ err ↓	$\log \rho$ err ↓
DreamPhysics (50 ep)	19.223	0.881	0.178	—	1.362	—	—
OmniPhysGS (5 ep)	17.880	0.884	0.305	—	—	—	—
NeRF2Physics	18.554	0.889	0.245	0.841	0.585	0.455	0.982
Gemini (avg. of 5)	21.124	0.869	0.205	0.218	0.227	0.235	0.192
Pixie (Clip, uncond.)	23.421	0.908	0.092	0.078	0.062	0.065	0.108
Ours (w/ Clip)	25.105	0.916	0.091	0.069	0.058	0.061	0.089
Ours (w/ OpenClip)	26.012	0.921	0.089	0.059	0.042	0.051	0.083

On the out-of-distribution real-world ABO-500 mass and density estimation benchmark (500 objects, evaluated using the default prompt This is a [object name]):

Method	ALDE (↓)	ADE (↓)	ARE (↓)	MnRE (↑)
NeRF2Physics	0.732	11.826	0.989	0.573
Pixie	0.782	12.565	0.971	0.568
Ours	0.652	9.433	0.901	0.582

Ablation Study¶

Ablation analysis on feature representation and extraction methods on Text2PhyX:

Config	PSNR ↑	SSIM ↑	Avg. Phys. err ↓	Note
Pixie (Occupancy)	17.901	0.868	0.122	Pure occupancy grid; completely lacks surface appearance and semantics
Pixie (RGB)	18.695	0.863	0.121	Raw RGB voxels; vulnerable to lighting variations and surface textures
Pixie (Clip)	23.421	0.908	0.078	Pre-trained visual semantics without text modulation; limited by visual ambiguity
Ours (w/ Clip)	25.105	0.916	0.069	Incorporating FiLM text modulation reduces physical error by 11.5%
Ours (w/ OpenClip)	26.012	0.921	0.059	Rich OpenCLIP semantics combined with volume-rendered lifting achieves top results

Key Findings¶

Cross-modal modulation resolves physical ambiguity: Compared to the feed-forward baseline Pixie, adding text conditioning via FiLM increases simulation rendering PSNR by ~2.6 dB (23.421 to 26.012) and reduces overall physical parameter error from 0.078 to 0.059 (a 24.4% reduction). In particular, the $\log E$ error for Young's modulus drops sharply from 0.062 to 0.042.
Superior real-world zero-shot transfer: On the out-of-distribution ABO-500 dataset, AutoPhyX decreases Average Displacement Error (ADE) to 9.433, outperforming Pixie (12.565) by over 20% (24.9% relative reduction), demonstrating that the learned parameter distribution generalizes to real-world objects without per-scene calibration.
Orders-of-magnitude faster inference: AutoPhyX completes feed-forward parameter inference in under 2 seconds, eliminating the hours-long iterative optimization required by DreamPhysics or OmniPhysGS while substantially improving dynamical realism.

Highlights & Insights¶

Repurposing VLMs as grounded reasoning tools rather than direct numerical guessers: The authors highlight an inherent asymmetry—VLMs excel at forward part generation and query formulation, but exhibit high variance and hallucinations when asked to predict physical numbers directly. Prompting VLMs to query verified engineering databases (MatWeb, Wikipedia) builds an accurate, repeatable training ground truth.
Occlusion-aware lifting paired with topological filling for true solid mechanics: Using accumulated transmittance to cull hidden surfaces prevents multi-view projection contamination, while even-odd ray intersections convert hollow graphic shells into solid physical volumes capable of continuous stress propagation.
Universal downstream engine portability: By formulating physical properties in a unified voxel field, predictions immediately transfer to Gaussian Splatting under MPM solvers (for tearing, bending, and fracturing) or tetrahedral meshes under FEM in Isaac Sim (for robotics manipulation and collision).

Limitations & Future Work¶

Expansion of physical parameter dimensions: AutoPhyX targets the primary solid parameters ($E$, $\nu$, $\rho$), but does not yet estimate surface friction coefficients, contact damping, or plastic yield stress limits.
Voxel discretization resolution constraints: Fine, high-aspect-ratio geometries (e.g., thin cables or paper-thin shells) can suffer from voxel quantization artifacts under uniform grids. Exploring adaptive octrees or continuous implicit neural representation fields represents a promising avenue for sub-millimeter dynamics.

vs NeRF2Physics: NeRF2Physics relies on raw GPT parameter guesses mapped to point clouds via CLIP text-image similarity, suffering from hallucinated values and blurred part boundaries; AutoPhyX uses curated engineering databases and end-to-end 3D U-Net regression, achieving crisp part separation and lower error.
vs Pixie / VoMP: Pixie and VoMP operate purely on visual features, unable to distinguish physically dissimilar but visually identical materials; AutoPhyX introduces text-driven FiLM modulation, granting direct user control over material behavior.
vs DreamPhysics / OmniPhysGS: Diffusion-based test-time optimization takes hours per scene and fails to generalize; AutoPhyX transfers physical priors into a feed-forward architecture, achieving superior fidelity in under 2 seconds.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First text-conditioned feed-forward 3D physics framework, featuring occlusion-aware lifting and 3D FiLM modulation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across Text2PhyX, ABO-500, multiple solvers (MPM/FEM), and representations (3DGS/Mesh).
Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation, comprehensive ablation studies, and insightful analysis of VLM limitations.
Value: ⭐⭐⭐⭐⭐ Bridges the critical gap between static 3D generative assets and realistic interactive dynamics for embodied robotics and simulation.