Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-View Videos

Conference: ECCV2026
Paper: ECCV Paper
Area: 3D Vision
Keywords: 4D human reconstruction, uncalibrated sparse views, streaming reconstruction, dense motion, novel-time synthesis

TL;DR

Forge4D decomposes reconstruction of dynamic humans and interacting objects into scale-consistent streaming 3D Gaussian prediction and bidirectional dense motion matching, then produces intermediate times through occlusion-aware fusion; it achieves 29.0378 dB PSNR on joint novel-time and novel-view evaluation on DNA-Rendering without requiring input camera calibration parameters.

Background & Motivation

Free-viewpoint human video involves two distinct requirements: observing a person from an uncaptured direction and generating a plausible pose at an uncaptured time. Clothing, handheld objects, and body parts can move differently, so reconstructing a sequence of static models does not automatically provide a continuous 4D representation. Per-scene optimization of dynamic NeRFs or Gaussians commonly requires calibrated dense cameras and lengthy optimization, limiting sparse capture and interactive use. Geometry models such as VGGT enable feed-forward reconstruction from uncalibrated images, but a point cloud is not yet a renderable asset with high-quality appearance. Methods such as AnySplat further predict 3D Gaussians, yet remain centered on individual timestamps without directly supplying intermediate-time motion.

Running these static models independently on successive frames introduces a less visible problem: the same person can be reconstructed at different scales across time. The apparent displacement then combines real movement with changes in coordinate scale, causing subsequent motion learning to mistake reconstruction errors for object motion. Putting all temporal and multi-view image tokens into global attention can exchange information, but increases memory consumption and interaction latency as the sequence grows. An alternative based on parametric human templates such as SMPL-X constrains body motion but struggles to cover unconstrained clothing deformation and complex human-object interactions. The required representation must therefore be renderable and template-free while keeping geometry comparable across time.

The authors first make adjacent 3D Gaussian reconstructions reliable and scale-consistent, then learn dense correspondences between the two sets. This converts missing ground-truth 3D motion into observable constraints: whether moving the Gaussians reproduces the neighboring frame and whether their projected motion agrees with image optical flow. An intermediate frame is not an independently generated image, but a combination of endpoint 3D representations moved to the requested time, which still allows the viewing camera to change. Core Idea: express 4D reconstruction as a temporally aligned 3D Gaussian stream plus explicit bidirectional motion matching, then fuse the endpoint representations according to occlusion instead of directly regressing an entangled whole of geometry and motion.

Method

Overall Architecture

Inputs are multiple temporally corresponding uncalibrated RGB videos, with 4 input views in the main experiments; uncalibrated means that input camera parameters are unnecessary when using the model, not that training avoids ground-truth camera information. Outputs comprise pixel-aligned 3D Gaussians at input timestamps, bidirectional 3D motion between adjacent timestamps, and a fusion function for synthesizing intermediate times. Each Gaussian has a position, opacity, color, rotation, and scale, without relying on a predefined human mesh or skeletal binding. The network builds on pretrained VGGT and uses DPT prediction heads to convert geometric features into attributes renderable through Gaussian Splatting.

Processing follows Scale-Aligned Reconstruction, Streaming State Memory, Bidirectional Motion Matching, and Occlusion-Aware Fusion. The first two establish temporally comparable keyframe geometry, the third specifies how geometric points move, and the last determines which moved Gaussians should merge or remain separate. Training progresses through static reconstruction, streaming alignment, and motion with fusion; at inference time, the state updates chronologically and interpolation operates within already observed adjacent-keyframe intervals.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Uncalibrated multi-view videos"] --> Metric["Scale-Aligned Reconstruction"]
    Metric --> State["Streaming State Memory"]
    State --> Motion["Bidirectional Motion Matching"]
    State -->|Endpoint Gaussians| Fusion["Occlusion-Aware Fusion"]
    Motion -->|Motion and correspondences| Fusion
    Fusion --> Output["Intermediate-time Gaussians<br/>Novel-view rendering"]
    Cameras["Training: ground-truth cameras<br/>and multi-view images"] -.-> Metric
    Adjacent["Training: neighboring renders<br/>and pseudo optical flow"] -.-> Motion
    Middle["Training: intermediate-time images"] -.-> Fusion

Key Designs

1. Scale-Aligned Reconstruction: make geometry compatible with the scale of supervision cameras

VGGT predicts geometry in an internal scale, whereas ground-truth camera translations use real-world scale. Rendering the former geometry with the latter cameras can produce incorrect images even when relative shape is accurate, because camera and point-cloud scales disagree. The authors therefore address the scale ambiguity in the supervision coordinate system instead of merely adding an image loss to the Gaussian head. Provided camera rotations and translation directions are reasonable, the translation-length ratios across input cameras should represent a common scale factor. The model averages these ratios into a metric gauge, aligns novel-view cameras with the internal geometric scale, and then applies image supervision.

The camera loss constrains rotations, normalized translation directions, relative scale consistency across cameras, and the scale prediction head. The scale head learns this alignment factor so that the model can estimate real-world scale from images during use without reading ground-truth input cameras again. This is scale estimation supported by training priors, not a geometric guarantee that absolute lengths can always be recovered from arbitrary uncalibrated images. The Gaussian attribute heads jointly learn from input-view and held-out-view rendering supervision, discouraging reconstructions that only appear correct from the supplied views. Removing scale alignment reduces static PSNR to 13.2884 in Table 7, showing that this step directly affects optimization stability.

2. Streaming State Memory: transmit temporal information through recurrent state rather than stacking all video tokens

Independent per-frame reconstruction cannot ensure consistent scales across time, while full-sequence attention accumulates many image tokens. Forge4D introduces learnable state tokens that compress information from previously observed frames into a recurrently updated state. When the current frame reads history, its features act as the Query and state tokens act as Key and Value, supplying temporal context. When updating the state, these roles reverse: state tokens act as the Query, read current-frame features, and pass the updated state to the next frame. This is more than assigning each frame a time label; historical geometric information continuously influences current reconstruction.

The metric gauge also extends from multi-camera consistency at one timestamp to consistency across cameras and time. Scale predictions within a sequence are consequently subject to a shared scale constraint, discouraging unexplained growth or shrinkage of the body between frames. State memory organizes the computation for transmitting information, while temporal scale supervision specifies the constraint to maintain; their roles differ. This structure avoids explicitly sending all historical images into a single global attention operation, but does not establish drift-free operation for arbitrary video lengths. The state-token ablation in Table 7 supports temporal alignment, although the paper does not provide a complete long-duration drift curve here.

3. Bidirectional Motion Matching: move Gaussians to neighboring frames instead of merely making intermediate images look plausible

When a new frame arrives, the motion module reads backbone intermediate features from the current and previous frames and establishes correspondences through cross-frame attention. It contains as many attention blocks as the backbone, aggregates their outputs, and uses a motion DPT head to predict pixel-aligned 3D motion. It predicts both backward motion from current Gaussians to the previous frame and forward motion from previous Gaussians to the current frame. Each endpoint thus has its own geometry and outgoing motion, without assuming that the Gaussian sets already share identical indices.

Without ground-truth dense 3D motion, an intermediate-image loss alone may allow other Gaussian attributes to compensate for implausible motion. The retargeting loss imposes a tighter constraint: change only current Gaussian positions, retain color, opacity, rotation, and scale, and render using cameras at the neighboring time. The position change follows the retargeting relation on page 9:

\[ \mathbf{P}_i^{t\to t-1}=\mathbf{P}_i^t+\mathbf{M}_{i,1}^t. \]

Here, motion denotes displacement between adjacent endpoints, and the aim is for moved Gaussians to reproduce the appearance at the other time. The supervision hierarchy matters: Equation (6) on page 9 compares renders of retargeted Gaussians against renders of reconstructed neighboring-frame Gaussians, rather than supplying ground-truth 3D point correspondences. Those neighboring reconstructions are themselves supervised by real multi-view images and can therefore provide visual constraints for motion learning. The introduction describes ground-truth image supervision more broadly; this note preserves the specific distinction in the method section.

The authors additionally use SEA-RAFT to produce pseudo 2D optical flow and compare it with projected predicted 3D motion. Forward-backward flow cycle consistency downweights occluded regions, avoiding strong supervision from incorrect flow for invisible points. This complements ambiguities in retargeting renders, but pseudo optical flow is not ground-truth 3D motion. Positions at intermediate timestamps follow a constant-velocity assumption: the farther the target is from an endpoint, the larger the applied fraction of its endpoint displacement. Arbitrary time therefore means queryable interpolation between adjacent known timestamps, not prediction of an unobserved future.

4. Occlusion-Aware Fusion: merge reliably corresponding duplicate Gaussians while retaining one-sided evidence

After both endpoint Gaussian sets move to the target time, concatenation leaves duplicate surfaces, while direct averaging can damage occlusion boundaries. Forge4D uses consistency between 2D and 3D motion to determine whether endpoint points correspond reliably. It moves a Gaussian to the neighboring frame using 3D motion, retrieves a candidate Gaussian there through projected optical flow, and compares their 3D positions. If the distance exceeds a threshold, the paper treats the inconsistency as occlusion-related and retains the endpoint Gaussian instead of forcing a merge. Correspondences with smaller distances enter a two-layer MLP that fuses their attributes and reduces duplicate endpoint representations.

The final intermediate representation combines retained occluded Gaussians with fused non-occluded Gaussians and is rendered from the target camera. The fusion MLP learns through a photometric loss against real intermediate-time images rather than acting as a fixed averaging operation after motion learning. This distinction explains why fusion influences luminance jitter and temporal stability as well as Gaussian count. However, occlusion decisions depend on predicted motion and geometric consistency; they are model estimates rather than externally supplied visibility labels.

A Worked Example

Suppose 4 views of an action provide images at times 0 and 1, and the goal is to observe the person and a handheld object from another camera at time 0.5. The model first reconstructs both endpoints with scale alignment, while state tokens carry information from time 0 to time 1 to make their geometry more comparable. The motion module then predicts Gaussian displacements from 0 to 1 and from 1 to 0; under constant velocity, each endpoint moves by half of its corresponding endpoint displacement. If an arm surface corresponds reliably at both endpoints, the fusion MLP merges the moved duplicate Gaussians. If object occlusion makes some points reliable from only one endpoint, the occlusion branch retains them instead of erasing them during fusion. Rendering the resulting time-0.5 Gaussians from a different target camera produces a novel view at that intermediate time. This illustrates the mechanism rather than introducing an experiment; it also shows why time 1 must arrive before bidirectional interpolation of the interval is possible.

Loss & Training

Stage one combines camera loss, input-view photometric loss, and held-out novel-view photometric loss to establish static reconstruction. Photometric supervision includes L2, SSIM, and LPIPS terms; stage two adds temporal camera and scale constraints for streaming alignment. Stage three combines motion matching with novel-time fusion supervision through the loss relationships explicitly stated in the text:

\[ \mathcal{L}_{4D}=\mathcal{L}_{\mathrm{matching}}+\mathcal{L}_{\mathrm{fusion}}, \qquad \mathcal{L}_{\mathrm{matching}}=\mathcal{L}_{\mathrm{flow}}+\mathcal{L}_{\mathrm{retarget}}. \]

The backbone uses pretrained VGGT weights, while the scale head, position and color offset heads, and related attention and motion modules are zero-initialized. Training uses 2,078 human-object interaction video sequences from DNA-Rendering, with all sequences of 10 distinct identities held out for in-domain testing. The available full-text cache excludes the referenced supplementary material, preventing verification of the complete optimization hyperparameters and synthetic-data construction details. Parts of Equations (2), (3), and (5) through (8) have damaged characters or layout, so this note does not reconstruct exact complete formulas for camera losses, the optical-flow mask, or the fusion operator.

Key Experimental Results

Main Results

The following excerpts come from Tables 1 and 2 on page 10; input timestamps evaluate novel views, while novel timestamps jointly evaluate interpolation and novel views. The main setup uses 4 input views with camera angles around 45 degrees, normally evaluated at 518×518; GPS-Gaussian and L4GM use 512×512. Higher PSNR in dB and higher SSIM are better; lower LPIPS is better. Camera-parameter requirements are retained to make differences in input conditions explicit.

Source and Time	Dataset	Method	Input Camera Parameters	PSNR	SSIM	LPIPS
Table 1, input time	DNA-Rendering	GPS-Gaussian	Required	24.2963	0.9247	0.0867
Table 1, input time	DNA-Rendering	AnySplat	Not required	26.1157	0.9430	0.1513
Table 1, input time	DNA-Rendering	Forge4D	Not required	29.8167	0.9606	0.0542
Table 1, input time	Genebody	AnySplat	Not required	25.8010	0.9287	0.1355
Table 1, input time	Genebody	Forge4D	Not required	28.0819	0.9523	0.0548
Table 2, novel time	DNA-Rendering	L4GM	Required	18.0325	0.9152	0.1367
Table 2, novel time	DNA-Rendering	Forge4D	Not required	29.0378	0.9566	0.0535
Table 2, novel time	Genebody	L4GM	Required	14.8572	0.9144	0.1727
Table 2, novel time	Genebody	Forge4D	Not required	27.4247	0.9459	0.0601

Ablation Study

The following excerpts use clearly identifiable rows from Table 7 on page 14, on DNA-Rendering; static and dynamic groups evaluate different tasks and must not share an ablation baseline. The optical-flow and Gaussian-fusion rows are merged by the cached layout, so their numerical assignments are not guessed here.

Evaluation Task	Config	PSNR	SSIM	LPIPS
Static novel view	Full model	29.8167	0.9606	0.0542
Static novel view	Without scale alignment	13.2884	0.1194	0.2184
Novel time and novel view	Full model	29.0378	0.9566	0.0535
Novel time and novel view	Without state tokens	28.5555	0.9513	0.0592
Novel time and novel view	Without retargeting loss	28.4124	0.9530	0.0573

Key Findings

• Novel-time synthesis retains strong cross-dataset results: Forge4D achieves 27.4247 dB on Genebody versus 14.8572 dB for L4GM. This concerns the paper's sparse-view setup, not a universal ranking across input configurations.
• Scale alignment is a prerequisite for stable training; within the dynamic group, removing either retargeting loss or state tokens reduces PSNR, supporting the roles of correspondence learning and temporal alignment.
• Page 14 reports TL-STD of 0.9446 without fusion and 0.5379 with fusion, making the former 75.6% higher. TL-STD denotes temporal luminance standard deviation, with larger values indicating more brightness jitter, but the cached main text does not provide its full calculation protocol.
• Table 6 on page 14 reports 224.27 ms and 4.45 FPS for the full model on H200, versus 225.10 ms and 44.42 FPS with 10 interpolation steps. The higher output frame rate mainly comes from interpolation and is not 44.42 FPS reconstruction of new inputs.
• Numerical boundaries: Table 6 separately lists 1.46 ms for 10 interpolation steps, which does not fully align with its total-latency rows or the page-15 description. Page 11 claims an advantage over AnySplat of up to +2.28 dB, whereas the DNA-Rendering difference in Table 1 is 3.7010 dB. This note retains table values rather than harmonizing the source.

Highlights & Insights

• Scale is not a peripheral evaluation-time correction; it affects both novel-view supervision and motion learning. Enforcing its consistency first helps prevent geometric coordinate drift from being interpreted as human motion.
• Retargeting supervision turns the renderer into a check on motion matching. It requires endpoint assets to move correctly rather than merely producing plausible intermediate images.
• Occluded regions should not automatically enter averaging-based fusion. Retaining one-sided evidence may transfer to other temporal point-cloud or Gaussian asset fusion tasks, but correspondence tests must account for their specific geometric errors.

Limitations & Future Work

• The authors explicitly report degradation for large motion and longer frame intervals because correspondences diminish and constant velocity becomes unreliable. Higher-order motion modeling is a plausible direction, but its benefits are not demonstrated here.
• The main inputs cover the frontal human appearance using 4 views. These results do not establish equal stability for monocular inputs, arbitrary camera arrangements, or completely unseen backsides.
• Quantitative metric-scale and dense-motion evaluation uses synthetic MetaHuman4D rather than complete real-world motion ground truth. Table 4 on page 12 gives a scale point distance of 0.0264 m, while the page-13 text states 0.02 m; this precision discrepancy is retained.
• Bidirectional interpolation needs the later endpoint to arrive, and input processing reaches only 4.45 FPS on H200. Low-latency streaming and high-frame-rate interpolated output are different system requirements.
• The cache lacks supplementary material and a verifiable code link, so baseline camera optimization, motion ground-truth construction, and all loss details cannot be fully reproduced from it.

Related Work & Insights

• VGGT / AnySplat: the former supplies geometric priors and the latter represents static Gaussian reconstruction from uncalibrated inputs. Forge4D additionally addresses temporal scale consistency and novel-time asset synthesis; see pages 4 through 7.
• L4GM / SpaceTimeGS: both support novel-time synthesis and therefore appear in Table 2. Forge4D emphasizes learning human-specific reconstruction and motion priors under uncalibrated sparse multi-view inputs, not merely comparing representation types.
• POMATO / SEA-RAFT: the former is a dense-motion baseline in Table 5 on page 12, while the latter supplies pseudo optical flow for training. Combining geometric correspondences with renderable appearance constraints is the transferable design beyond reliance on 2D motion alone.
• SMPL-X-based template methods: these obtain animation constraints from human structure, whereas Forge4D uses pixel-aligned Gaussians to accommodate clothing and interacting objects. In exchange, it does not naturally inherit template-based skeletal semantics and motion-control interfaces.

Rating

These are subjective reader assessments on a 5-point scale, not experimental metrics reported by the paper. - Novelty: 4/5. Scale alignment, streaming reconstruction, and motion matching form a coherent system whose value lies in how its constraints interact. - Experimental Thoroughness: 4/5. In-domain, out-of-domain, real-capture, and ablation evaluations are included, but motion and metric-scale ground truth mainly comes from synthetic data. - Writing Quality: 3/5. The staged method is clear, but numerical disagreements between text and tables and damaged cached equations increase verification effort. - Value: 4/5. The method offers useful guidance for human-object 4D reconstruction without input calibration, although input latency and large-motion limitations remain substantial.