PhysConvex: Physics-Informed Dynamic Convex Fields for Reconstruction and Simulation¶
Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://github.com/DanWang-98/PhysConvex
Area: 3D Vision
Keywords: Dynamic 3D Reconstruction, Physical Simulation, Dynamic Convex Primitives, Continuum Mechanics, Model Order Reduction
TL;DR¶
PhysConvex establishes a unified 4D representation by treating dynamic convex hulls as differentiable rendering elements, deformation carriers, and continuum physical supports simultaneously, coupling boundary-driven kinematics with mesh-free neural skinning reduced-order simulation for highly efficient dynamic video reconstruction and system identification.
Background & Motivation¶
Reconstructing deformable 3D objects from video observations stands as a core challenge across computer vision and graphics. A truly capable 4D representation must not only synthesize high-fidelity novel views but also adhere strictly to governing physical laws, enabling temporal forward prediction of future motions and plausible generalization under modified material properties, external forces, or boundary conditions. Classical numerical simulation frameworks, such as Finite Element Methods (FEM) and Material Point Methods (MPM), deliver physically grounded dynamics yet depend heavily on explicit volumetric meshes, clean initial geometries, and known material constitutive parameters—assumptions that fail in real-world visual observations. Conversely, purely neural kinematic models, including dynamic NeRFs and 4D Gaussian Splatting variants, fit observed trajectories directly from data without underlying physical constraints, leading to catastrophic divergence when rolling out into unobserved futures or interactive scenarios.
Recent advances have attempted to bridge this gap by coupling differentiable simulation frameworks (e.g., MPM or mass-spring systems) with neural radiance or Gaussian fields. However, these methods suffer from an inherent representation mismatch between visual rendering and mechanical simulation. Existing pipelines predominantly rely on voxels or symmetric Gaussian ellipsoids that were engineered for static rendering efficiency rather than continuous mechanical deformation. In hybrid Eulerian-Lagrangian setups, these systems regress to center-driven dynamics that update primitives solely via central particles or pooled neighbor velocities, entirely missing non-uniform intra-primitive deformations. Predefined bindings between visual primitives and simulation particles disregard geometric and constitutive relationships, while ellipsoidal primitives suffer from sphere-packing gaps on planar regions and lack the capacity to express evolving sharp boundaries under MPM grid interactions.
The key insight of this paper is that the visual rendering primitive must serve natively as the material support for physical deformation and continuum mechanics. The core idea is to represent dynamic deformable objects as boundary-driven 4D dynamic convex fields, where convex hull vertices and dual half-space supports directly undergo physical advection, integrated seamlessly with a mesh-free reduced-order simulation driven by neural skinning eigenmodes for end-to-end differentiable optimization.
Method¶
Overall Architecture¶
The PhysConvex framework operates across a two-stage pipeline uniting geometry initialization and differentiable dynamic-physics optimization. In the first stage, an undeformed static convex field \(\mathcal{C}(0)\) is reconstructed within the reference material configuration \(\Omega_0\) using multi-view observations of the initial timestep. Next, a lightweight neural network learns continuous neural skinning eigenmodes in a data-free manner via elastic strain energy minimization, constructing a compact physical deformation subspace. In the second stage, an implicit Lagrangian Newton solver computes the time-stepping of reduced degrees of freedom under video supervision, explicitly advecting convex hull vertices while implicitly updating dual half-space support functions. Differentiable convex splatting renders the dynamic scene, enabling joint end-to-end system identification of physical parameters (Young's modulus \(E\), Poisson's ratio \(\nu\)) alongside neural deformation parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Multi-view Video Sequences & Camera Poses"] --> B["Stage 1: Reference Convex Field Initialization"]
B --> C["Boundary-Driven Dynamic Convex Representation"]
C --> D["Neural Skinning Eigenmode Reduced-Order Simulation"]
D --> E["Unified Rendering-Deformation-Physics Coupling"]
E --> F["Output: Physically Consistent View Synthesis & Future Rollouts"]
Key Designs¶
1. Boundary-Driven Dynamic Convex Representation: Overcoming Center-Driven Anisotropic and Non-Uniform Deformation Bottlenecks
Standard 3D Gaussian dynamics treat primitives as rigid or affine-constrained kernels, lacking spatial sensitivity to non-uniform local deformations and sharp boundary motion. PhysConvex formulates the scene as a collection of dynamic convex polytopes \(\mathcal{C}(t) = \{\mathcal{K}_n(t), \mathbf{c}_n, \sigma_n, \alpha_n, \beta_n\}_{n=1}^N\), parameterizing each primitive as the convex hull of a time-varying 3D point set \(\mathcal{K}(t) = \{\mathbf{x}_k(t)\}_{k=1}^K\). By advecting individual hull vertices under continuum mechanics rather than shifting a single primitive center, each convex primitive actively stretches, shears, and alters its active supporting faces, providing compact volumetric coverage without sphere-packing vacuum gaps. For differentiable splatting, the dual implicit surface dynamics express the hull via \(H\) time-varying planes \(f_h(\mathbf{x}, t) = \mathbf{n}_h(t) \cdot \mathbf{x} + d_h(t)\), producing a continuous time-dependent occupancy function: $\(\mathcal{O}(\mathbf{x}, t) = \text{Sigmoid}\left(-\beta \max_{h=1}^H (\alpha f_h(\mathbf{x}, t))\right)\)$ This formulation preserves sharp geometric boundaries while ensuring smooth gradient back-propagation across the entire volumetric domain.
2. Neural Skinning Eigenmode Reduced-Order Simulation: Mesh-Free Continuum Mechanics in Low-Dimensional Subspaces
Conventional FEM requires expensive tetrahedral meshing, while MPM introduces grid dissipation and unstable gradient flows during neural rendering. PhysConvex designs a mesh-free Model Order Reduction (MOR) simulation scheme, parameterizing the displacement map of convex hull vertices via continuous neural skinning eigenmodes \(W^\theta(\mathbf{X}) \in \mathbb{R}^3 \to \mathbb{R}^M\) contracted with time-varying reduced control handle matrices \(\mathbf{Z}(t) \in \mathbb{R}^{3 \times 4 \times M}\): $\(\mathbf{x}_k(t) = \phi(\mathbf{X}_k, \mathbf{Z}(t), W^\theta) = \sum_{m=1}^M W_m^\theta(\mathbf{X}_k) \mathbf{Z}_m(t) [\mathbf{X}_k, 1]^\top\)$ The continuous MLP field \(W^\theta\) is trained prior to dynamics optimization in a completely data-free manner by minimizing hyperelastic strain energy \(\Psi\) alongside an orthogonality regularizer over random configurations: $\(\min_\theta \int \varphi(\mathbf{X})\Psi(\phi(\mathbf{X}, \mathbf{Z}, W^\theta)) d\mathbf{X} + \lambda_{\text{ort}} \sum_{i=1}^M \sum_{j=1}^M \int \varphi(\mathbf{X}) \left(W_i^\theta W_j^\theta - \delta_{ij}\right)^2 d\mathbf{X}\)$ High strain energy penalizes non-physical distortions, while orthogonality prevents collapse into trivial rigid transformations. Forward simulation optimizes the flattened \(12M\)-dimensional reduced coordinate vector \(\mathbf{z}(t)\) via an implicit incremental potential formulation and standard Newton solver, executing stable large-deformation elastodynamics without background grids or manual bindings.
3. Unified Rendering-Deformation-Physics Coupling: Eliminating Particle Bindings for End-to-End System Identification
Existing physics-informed neural rendering methods attach Gaussian or voxel primitives to external simulation particles using heuristics like KNN binding, introducing hyper-parameter sensitivity and dynamic mismatch. In PhysConvex, each convex primitive serves simultaneously as a rasterization primitive, a deformation carrier, and a numerical quadrature support where mass, strain energy, and external forces are directly evaluated. For rendering, 3D dynamic convex vertices are projected onto 2D image planes and rasterized via depth-sorted \(\alpha\)-blending. For physics identification, a pre-trained VideoMAE backbone extracts temporal motion priors from a single viewpoint, with lightweight MLP heads regressing initial estimates of Young's modulus \(E\) and Poisson's ratio \(\nu\). Photometric rendering errors then back-propagate directly through the differentiable simulation solver and convex geometry, end-to-end refining both constitutive parameters and motion fields.
Loss & Training¶
The framework is optimized through a two-stage objective: 1. Static Reference Field Reconstruction: Optimizing undeformed convex parameters from the initial multi-view frames: $\(\mathcal{L}_{\text{static}} = (1 - \lambda)\mathcal{L}_1 + \lambda \mathcal{L}_{\text{D-SSIM}} + \beta_m \mathcal{L}_m\)$ where \(\mathcal{L}_1\) and \(\mathcal{L}_{\text{D-SSIM}}\) enforce photometric and structural fidelity, while \(\mathcal{L}_m\) regularizes the total primitive count. 2. Dynamic Physics Optimization: Differentiably refining physical parameters \(\{E, \nu\}\) and skinning weights \(\theta\) via mean squared error across temporal observations: $\(\mathcal{L}_{\text{render}} = \frac{1}{SP}\sum_{s=1}^S \sum_{p=1}^P \|\mathbf{C}(\mathbf{q}_p, t_s) - \mathbf{C}_{\text{gt}}(\mathbf{q}_p, t_s)\|_2^2\)$ Optimized with Adam using learning rates of \(5 \times 10^{-3}\) for \(E\), \(1 \times 10^{-3}\) for \(\nu\), and \(5 \times 10^{-7}\) for \(\theta\) over 400 iterations, converging within approximately 6 minutes per scene.
Key Experimental Results¶
Main Results¶
Evaluated on the 12 Google Scanned Objects (GSO) benchmark scenes featuring complex non-linear elastodynamics, models observe 16 frames to identify physical parameters and reconstruct dynamic sequences, followed by rolling out 8 unobserved frames for future state prediction. Baselines include MPM-based NeRF (PAC-NeRF), spring-mass Gaussians (Spring-GS), continuum Gaussians (GIC), and reduced-order Gaussians (Vid2Sim w/o LGM).
| Method | Dynamic PSNR (dB) ↑ | Dynamic SSIM ↑ | Dynamic FoVVDP ↑ | Physics \(\log(E)\) MAE ↓ | Physics \(\nu\) MAE ↓ | Future PSNR (dB) ↑ | Training Time (min) ↓ |
|---|---|---|---|---|---|---|---|
| PAC-NeRF | 22.06 | 0.924 | 6.674 | 2.50 | 0.21 | 20.11 | 51 |
| Spring-Gaus | 20.91 | 0.920 | 6.410 | - | - | 18.32 | 32 |
| GIC | 21.01 | 0.934 | 6.544 | 2.01 | 0.16 | 19.20 | 69 |
| Vid2Sim (w/o LGM) | 28.57 | 0.952 | 7.794 | 0.51 | 0.06 | 25.07 | 8 |
| PhysConvex (Ours) | 30.00 | 0.962 | 8.247 | 0.37 | 0.04 | 25.71 | 6 |
Ablation Study¶
To isolate the contributions of the dynamic convex representation and boundary-driven kinematics under identical reduced-order simulation (MOR), the authors conducted a rigorous module breakdown (Tab. 5):
| Model Config | 3D Primitive | Kinematics Mode | Simulation | Dynamic PSNR (dB) ↑ | Dynamic SSIM ↑ | Dynamic FoVVDP ↑ | \(\log(E)\) MAE ↓ | \(\nu\) MAE ↓ |
|---|---|---|---|---|---|---|---|---|
| Model A | 3D Gaussian | Center-Driven | MOR | 28.65 | 0.954 | 7.895 | 0.54 | 0.07 |
| Model B | 3D Convex | Center-Driven | MOR | 29.16 | 0.958 | 8.173 | 0.57 | 0.05 |
| Model C (Full) | 3D Convex | Boundary-Driven | MOR | 30.00 | 0.962 | 8.247 | 0.37 | 0.04 |
Furthermore, architectural sensitivity analyses on vertex capacity \(K\) (Tab. 6) and reduced handle count \(M\) (Tab. 7) indicate: - Varying \(K\) from 3 to 6 improves dynamic PSNR from 24.90 dB to 30.00 dB, with performance plateauing beyond \(K=6\) (\(K=8\) achieves 30.19 dB but incurs a 49% increase in primitive count and higher training time). - \(M=10\) provides the ideal balance between deformation flexibility and subspace stability, achieving a future prediction PSNR of 25.71 dB compared to 25.05 dB (\(M=8\)) and 24.76 dB (\(M=12\)).
Key Findings¶
- Boundary-driven kinematics is essential for accurate deformation and physics: Transitioning from center-driven convex motion (Model B) to boundary-driven vertex advection (Model C) improves dynamic PSNR by 0.84 dB and drops the \(\log(E)\) estimation error significantly from 0.57 to 0.37, confirming the importance of boundary deformation in capturing true material strain.
- Superior primitive efficiency and compactness: PhysConvex requires only 23,681 convex primitives per scene compared to 31,363 Gaussians in Vid2Sim (a 24.5% reduction) while achieving superior rendering fidelity, converging in just 6 minutes on a single RTX A6000 GPU.
- Extensive generalization to novel materials and boundaries: PhysConvex successfully generalizes to stiff elastic (\(E=10^7, \nu=0.49\)), soft elastic (\(E=8000, \nu=0.4\)), plasticine (\(\tau_Y=500\)), and granular sand (\(\theta_f=10^\circ\)) materials, as well as handling continuous external lateral forces and sliding motion over dynamic floor boundaries without geometry retraining.
Highlights & Insights¶
- Rendering Primitives as Mechanical Domains: Eliminates artificial separation between rendering geometry and simulation particles by directly computing continuum mechanics and strain energy over deformable convex hulls.
- Data-Free Continuous Neural Skinning: Discovers low-dimensional physical deformation eigenmodes across complex 3D geometries via strain energy minimization and orthogonality regularization without requiring manual rigs or trajectory data.
- Duality of Explicit Vertices and Implicit Half-Spaces: Leverages explicit vertex advection for flexible physical mechanics alongside dual half-space smooth signed distance fields for continuous, differentiable volume rendering.
Limitations & Future Work¶
- Reliance on Posed Videos and Foreground Masks: The optimization pipeline requires pre-calibrated camera poses and clean object segmentations produced by SAM, which can introduce artifacts in heavy occlusion or dynamic camera environments.
- Simplified Environmental Boundary Constraints: Experiments assume known flat ground planes and gravity vectors; handling complex multi-body frictional contact and non-convex self-collisions requires deeper contact barrier mechanics.
- Future Directions: Exploring feed-forward dynamic convex initialization, explicit contact barrier potential integration into reduced simulation, and extending the representation to multi-object interactive dynamic scenes.
Related Work & Insights¶
- vs PAC-NeRF / GIC: PAC-NeRF and GIC couple neural representations with background Eulerian MPM grids, suffering from boundary dissipation and long training times (51–69 minutes); PhysConvex uses mesh-free reduced-order convex fields to achieve 6-minute training with sharp boundaries.
- vs Spring-Gaus: Spring-Gaus is restricted to elastic mass-spring networks between Gaussian centers; PhysConvex models true continuum strain energy densities, supporting elastoplasticity, sand, and non-uniform boundary evolution.
- vs Vid2Sim: While Vid2Sim adopts reduced-order simulation, it remains tethered to center-driven Gaussian ellipsoids with inherent packing gaps; PhysConvex's boundary-driven dynamic convex fields yield higher PSNR (+1.43 dB) and better future extrapolation (+0.64 dB).
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering unification of dynamic convex fields as rendering primitives and continuum simulation supports with boundary-driven kinematics]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across physical parameter identification, dynamic reconstruction, future prediction, ablations, and real-world scenes]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous continuum mechanics formulations, clear pipeline figures, and coherent structural progression]
- Value: ⭐⭐⭐⭐⭐ [Establishes a highly efficient and geometrically faithful paradigm for video-based differentiable physics and dynamic view synthesis]