ReconPhys: Reconstruct Appearance and Physical Attributes from Single Video

Conference: ECCV 2026
Paper: ECCV paper
Code: https://github.com/chuanshuogushi/ReconPhys
Area: 3D Vision
Keywords: non-rigid reconstruction, physical attribute estimation, Gaussian Splatting, spring-mass system, self-supervised learning

TL;DR

ReconPhys connects a frozen single-image 3D Gaussian reconstructor and a video-based physical parameter predictor to differentiable spring-mass simulation, learning simulation-ready assets from monocular free-fall videos through image reconstruction error and improving synthetic future-prediction PSNR from Spring-Gaus's 13.27 to 21.64 with reported feedforward inference under 1 second.

Background & Motivation

Dynamic 3D reconstruction can render a deforming object convincingly without explaining why it deforms that way. NeRF and dynamic 3D Gaussian Splatting (3DGS) typically learn relationships between time, appearance, and geometry. Continuing motion beyond the observations or applying a new squeezing action is different: a deformation field fitted to an existing video need not respond plausibly. Robotics simulation needs an object with a dynamic response, not just an animation that can be replayed.

Physics-aware reconstruction connects rendering to simulation and estimates material parameters from observation errors. Spring-Gaus drives Gaussians with a spring-mass system to represent elastic deformation, but typically requires multi-view capture and per-scene optimization. ReconPhys changes how these parameters are obtained: a network learns a cross-object mapping from visual motion to physics so that each new test object does not require hours of optimization. Its principal setting is gravitational falling, collision, and rebound, rather than general physical inference from arbitrary real-world videos.

A falling-object video contains both shape and texture cues and evidence about compression, rebound, and energy dissipation after contact. The authors use a pretrained reconstructor to supply a canonical shape and concentrate learning on physical attributes that explain the motion. Core Idea: predict spring-mass parameters in one forward pass, compare the video produced by differentiable simulation and Gaussian rendering with the observations, and use motion reconstruction error to train the feedforward physical estimator.

Method

Overall Architecture

The input is a single-view video, using the first 20 frames in the experiments. Its initial reference image enters a frozen 3DGS predictor to produce canonical Gaussians. The video also enters a physical prediction branch that outputs mass, stiffness, damping, and ground friction; these parameters drive a sparse spring-mass system constructed from the canonical geometry, and a binding mechanism transfers its motion to Gaussian centers.

The resulting asset includes canonical appearance, geometry, and physical attributes for subsequent simulation, rather than only future images. Training requires simulation, rendering, and reconstruction-error feedback. At test time, the feedforward asset prediction supports further simulation or interaction without scene-specific parameter optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Monocular video"] -->|Initial reference image| Canonical["Frozen Canonical<br/>Reconstruction"]
    Input --> Physics["Video-based<br/>Physical Estimation"]
    Canonical --> Binding["Sparse Dynamics<br/>Binding"]
    Physics -->|Predicted physical parameters| Binding
    Binding --> Rollout["Self-Forcing<br/>Simulation Supervision"]
    Input -.->|Training image supervision| Rollout
    Rollout -.->|Training predictor update| Physics
    Rollout --> Output["Dynamic Gaussians and<br/>future simulation"]

Key Designs

1. Frozen Canonical Reconstruction: fix the visual foundation so that physics must explain motion

The 3DGS predictor produces Gaussian centers, rotations, colors, scales, and opacities from the initial image. The paper explicitly uses off-the-shelf pretrained weights and freezes this predictor throughout physical training. Thus, the abstract's joint inference of appearance and physics should not be read as training both branches jointly from scratch. During simulation rollout, only Gaussian centers change; the other attributes remain fixed. This supplies a stable visual representation and prevents the physical model from hiding incorrect motion behind changing colors or opacities.

This constraint also leaves a source of error. If single-image geometry is inaccurate, the physical branch can only seek parameters that best explain the video using that canonical object; these need not equal the true material properties. The synthetic data pipeline uses TRELLIS to construct Gaussian assets from four orthogonal views. For the model branch, the text explicitly specifies a frozen pretrained 3DGS predictor, so the four-view asset-generation procedure should not be confused with a test-time input requirement.

2. Video-based Physical Estimation: predict shared parameters from motion instead of fitting every point independently

InternViT extracts per-frame visual features, a ResNet backbone with self-attention aggregates spatiotemporal information, and an MLP decoder outputs physical attributes. The experiments use InternViT-300M. Although the problem formulation allows an individual mass for each point and individual stiffness and damping for each spring, the implemented model predicts four shared quantities, with the corresponding values reused across points and springs:

\[ \hat{\mathbf{p}}=(\hat m,\hat k,\hat d,\hat f),\qquad \hat m_i=\hat m,\quad \hat k_{ij}=\hat k,\quad \hat d_{ij}=\hat d. \]

Mass here is the shared mass-point parameter, not automatically the total object mass. Motion features allow identical geometry to receive different parameters, while sharing sharply reduces output dimensionality and the difficulty of inferring a high-dimensional material field from 2D observations. The tradeoff is an inability to describe strongly heterogeneous mass or elasticity within one object: physical attribute prediction here is not per-voxel material identification.

3. Sparse Dynamics Binding: simulate a small set of interior anchors and let dense Gaussians follow

The system samples sparse anchors throughout the object volume rather than only on the surface, supporting more stable volumetric deformation. K-nearest neighbors on the initial shape define spring connections and rest lengths, and connectivity remains fixed afterward. Each step updates the state using nonlinear spring forces, damping, and gravity. Spring forces depend on deviations from rest length, while damping suppresses relative motion along a connection. Semi-implicit Euler integration updates velocity before position, followed by ground-collision boundary handling.

Individual Gaussians need not participate in expensive dynamics calculations. Each Gaussian is bound to nearby anchors, and inverse-distance weights determined from initial distances interpolate its center, giving closer anchors more influence. This separates the dense representation needed for rendering from the sparse state needed for simulation, while allowing image errors to propagate through Gaussian centers to anchors and physical parameters. The cached force and integration equations are visibly corrupted by text extraction; this note describes mechanisms supported by the prose without inventing exact equations.

4. Self-Forcing Simulation Supervision: roll out predicted states while truncating gradients across steps

After parameters enter the differentiable simulator, each step proceeds from the model's own previous state rather than resetting to ground-truth or proxy states. The authors call this strategy Self Forcing. Gaussians deform with the anchors and a differentiable renderer produces video frames. Training signals flow from image error through rendering, Gaussian centers, anchor updates, and physical parameters, teaching the predictor to reproduce motion without physical parameter labels.

A fully autoregressive forward trajectory does not imply an unbroken backward graph across time. Before computing each next state, the paper detaches the input state and uses truncated back-propagation to avoid exploding or vanishing gradients over long trajectories. The current step still depends on the shared physical parameters and can supervise them. The stability mechanism therefore exposes the forward process to its own accumulated errors without differentiating through the entire history; it does not replace simulation outputs with true trajectories. The paper provides no quantitative ablation isolating this strategy's benefit.

A Worked Example

Consider the Hamburger object illustrated in the paper, with 20 input frames showing falling and collision. The first frame supplies canonical Gaussian geometry. Interior anchors and fixed nearest-neighbor springs establish its physical structure. The video branch observes compression and rebound after contact and predicts shared mass, stiffness, damping, and friction. The simulator moves this structure, and bound Gaussians follow to form a renderable dynamic object.

During training, the observed first 20 frames constrain rendered motion. Future-prediction evaluation continues simulation for the next 10 frames and compares against withheld ground truth. Assigning different physical attributes to identical geometry should produce different parameter estimates and future trajectories from the two observed motions, rather than identical outputs because appearance is shared. The paper tests this with object pairs carrying two attribute configurations, but does not establish equally accurate recovery of every parameter.

Loss & Training

The objective sums squared L2 reconstruction errors between observed and rendered images over frames. Ground-truth physical parameters are used for synthetic data generation and error evaluation, not in this self-supervised objective. Self-supervision also does not remove dependence on pretrained models or synthetic assets. Because equation extraction is incomplete in the cache, this note does not present repaired expressions as the authors' exact formulas.

The synthetic pipeline semantically filters Objaverse-XL candidates using Qwen3-8B and creates simulation-ready assets. Sampling seeds are derived from object-identifier hashes to reproduce the same anchor configuration for each object. Physical sampling ranges are mass [0.2, 6.0], stiffness [10, 1200.0], damping [0.1, 5.0], and friction [0.0, 1.0]. The text does not specify a consistent set of physical units, so these should not be assigned SI units without evidence.

The pipeline section reports 500 selected objects, whereas the experimental setup uses 496, split into 450 training and 46 test objects, without explaining the difference. Each training object receives 10 physical samples and each test object receives 2. Videos contain 30 frames at 512 × 512 resolution. Training uses 8 RTX 4090 GPUs, batch size 8, and 100K iterations. The reported inference time below 1 second excludes these offline training costs.

Key Experimental Results

Main Results

Original Table 1 evaluates 46 unseen objects. ReconPhys receives a single-view 20-frame sequence, whereas baselines perform per-scene optimization on four-view 20-frame inputs following their original protocols. The authors state that ReconPhys metrics use the held-out single view, while baseline metrics use their optimization views. This is therefore not a controlled comparison with identical input views and evaluation conditions.

Method	Reconstruction PSNR ↑	Reconstruction SSIM ↑	Reconstruction CD ↓	Future PSNR ↑	Future LPIPS ↓	Future CD ↓	Reported time
4DGS	30.33	0.983	0.593	Unsupported	Unsupported	Unsupported	>1 h
Spring-Gaus	22.26	0.874	0.466	13.27	0.2856	0.349	>1 h
ReconPhys	33.84	0.953	0.001	21.64	0.0876	0.004	<1 s

PSNR and SSIM measure image similarity, while LPIPS measures perceptual difference. CD denotes Chamfer Distance and measures geometric discrepancy between 3D point sets. Future PSNR improves by 8.37 dB over Spring-Gaus, but reconstruction SSIM remains below 4DGS's 0.983, so ReconPhys is not best on every metric. The time column also compares two different computational workflows: feedforward inference and per-scene optimization.

Original Table 2 evaluates the real-world Spring-Gaus dataset, as summarized below. It does not report real-world geometry errors or true material parameter errors.

Method	Reconstruction PSNR ↑	Reconstruction SSIM ↑	Reconstruction LPIPS ↓	Future PSNR ↑	Future SSIM ↑	Future LPIPS ↓
Spring-Gaus	22.12	0.865	0.148	13.44	0.736	0.278
ReconPhys	28.53	0.882	0.112	19.45	0.857	0.126

Ablation Study

The available full text provides no component ablation removing the physical branch, Self Forcing, gradient truncation, or binding. The following is the physical attribute recovery analysis from original Table 4, not a fabricated ablation. Metrics are mean absolute errors (MAE) between predicted and true parameters. Columns have different units and should not be summed directly.

Method	Stiffness MAE ↓	Damping MAE ↓	Mass MAE ↓	Friction MAE ↓
Spring-Gaus	827.67	2.546	2.276	1.082
ReconPhys	297.3	1.151	1.337	1.508

Key Findings

• Future-image quality improves on both synthetic and real data, with real-world future PSNR increasing from 13.44 to 19.45. This supports some cross-domain transfer, not effectiveness for arbitrary cameras and materials.
• Stiffness, damping, and mass MAE decrease, but friction MAE increases from 1.082 to 1.508. The friction errors also exceed the width of the stated synthetic sampling interval [0.0, 1.0]; the paper does not explain output bounds or error scaling, so the discrepancy should remain explicit rather than being silently corrected.
• Original Table 3 compares two physical configurations for identical geometry, supporting discrimination between different motion states. This is not equivalent to uniquely identifying each physical parameter.

Highlights & Insights

• Moving per-scene optimization into offline training of a shared predictor makes simulation-ready asset creation a feedforward task. Cross-object learning of physical estimation, rather than simply adding a renderer, changes the deployment workflow.
• Freezing appearance and deriving Gaussian centers from physical states restrict the model's ability to bypass dynamics through visual flexibility. This constraint suits settings with a reliable reconstructor but scarce physical labels.
• Binding sparse anchors to dense Gaussians decouples simulation resolution from visual resolution. The interface is reusable when differentiable physics and high-quality rendering must cooperate in asset modeling.

Limitations & Future Work

• The authors explicitly note that fixed KNN connectivity cannot represent tearing, fracture, or permanent self-contact splitting. The spring-mass prior also limits phenomena outside the training distribution, such as plastic flow.
• Shared parameters, rather than spatial material fields, limit heterogeneous-object modeling. Higher-dimensional parameterizations would still face insufficient monocular evidence.
• Good photometric fits do not guarantee uniquely correct parameters, especially when quantities such as mass and stiffness jointly influence motion. This note identifies an inference risk; the paper provides neither a uniqueness proof nor a dedicated ablation.
• Real-world results remain sensitive to canonical geometry, camera conditions, and motion patterns. The paper demonstrates keyboard-controlled simulation of gripper or hand interactions, not a real robot policy evaluated by task success rate.

Related Work & Insights

• vs Spring-Gaus: ReconPhys retains spring-mass/Gaussian binding but replaces multi-view per-scene optimization with feedforward estimation from monocular video. Deployment speed and cross-object parameter prediction improve, while friction error prevents a claim of uniformly better physical recovery.
• vs 4DGS: The latter primarily fits observed dynamic appearance, whereas ReconPhys explicitly parameterizes dynamics that can continue evolving. This does not negate 4DGS's higher reconstruction SSIM.
• vs PhysGaussian / Vid2Sim: PhysGaussian emphasizes physically driven motion generation. As discussed in this paper, Vid2Sim uses a linear-blend-skinning-based reduced-order simulator and scene-specific refinement. ReconPhys focuses on monocular spring-parameter prediction without test-time optimization.

Rating

• Novelty: 4/5. Feedforward video-based physics, frozen Gaussian reconstruction, and differentiable simulation form a deployment-oriented approach, while the basic physical representation largely builds on prior work.
• Experimental Thoroughness: 3/5. Unseen-object, real-world, and parameter-recovery analyses are included, but component ablations and stricter matched-condition comparisons are missing.
• Writing Quality: 3/5. The pipeline is understandable, but some broad claims exceed the tabulated evidence, and the 500/496 object discrepancy and friction-error scale remain unexplained.
• Value: 4/5. A practical direction for quickly creating interactive non-rigid assets, though not yet a general tool for measuring real material properties.