Syn4D: A Multiview Synthetic 4D Dataset

Conference: ECCV2026
Paper: ECCV Paper
Project: Syn4D
Area: 3D Vision
Keywords: synthetic dataset, multiview video, dynamic point maps, dense tracking, geometry-aware generation

TL;DR

Syn4D combines high-quality dynamic scene rendering with compact mesh-based tracking annotations, providing approximately 4.7K multiview clips and 1.4M frames and improving dense tracking APD on Soviet from 58.35 to 74.46 without changing the 4RC architecture.

Background & Motivation

Feed-forward 3D reconstruction increasingly depends on large-scale geometric supervision, but dynamic video requires more than knowing how far away a pixel lies: a model must identify where the same physical point moves at other times. Real capture often provides only sparse, noisy trajectories. Optical flow or scene flow describes motion between adjacent frames, but does not by itself supply complete physical correspondences across arbitrary time intervals.

Existing synthetic datasets leave different gaps. Kubric supports dense annotations but primarily contains falling rigid objects; PointOdyssey provides relatively sparse long-term trajectories; the BEDLAM family includes complex human animation without equally diverse non-human dynamic objects. Even when high-quality environments and animated assets are available, explicitly storing every pixel's location at every time in every camera reference frame creates a prohibitive storage problem.

Syn4D therefore addresses both content and representation: it constructs diverse dynamic scenes and makes tracking labels queryable through a compact geometric representation. Core Idea: bind each pixel to a fixed triangle and barycentric coordinates on an animated mesh, then recover dense trajectories across time and cameras from the mesh vertex trajectories.

Method

Overall Architecture

Data generation starts with environments, animated objects, and human assets. Unreal Engine 5 renders synchronized multicamera videos and exports camera parameters, depth, instance segmentation, and human body parameters. Pixels are then associated with animated meshes to produce a compact representation of dynamic point maps (DPMs), alongside global and local video captions.

A DPM is not simply a per-frame depth map. A query selects a pixel in a source image, a target time, and a reference camera coordinate system, and returns the same physical point's 3D location at that time in that coordinate system. Neither the time nor the camera must match the source image, enabling reconstruction, long-term tracking, and cross-view supervision within one representation.

The paper also extends ReCamMaster using paired appearance and geometry: given a source video, source point maps, and a target camera trajectory, the model jointly generates target video and point maps. This is a demonstration of the dataset's utility, not the dataset itself.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Assets["Environment, animation,<br/>and human assets"] --> Build["Scene and Camera Construction"]
    Build --> Tracks["Compact Dynamic Point Maps"]
    Build --> Captions["Hierarchical Video Captions"]
    Tracks --> Data["Multiview Appearance<br/>and Geometry Data"]
    Captions --> Data
    Data --> Generator["Geometry-Aware<br/>Multiview Generation"]
    Data --> Tasks["4RC and Human Estimator Training"]

Key Designs

1. Scene and Camera Construction: vary both dynamic content and observation patterns

The authors purchase 30 Fab environments and manually choose suitable root positions for dynamic content. Each scene receives 1 to 3 objects drawn from 1,674 filtered Objaverse animated assets, plus one human from 585 human assets. Filtering renders animations from a bird's-eye view and uses the IoU of adjacent segmentation masks to reject excessive motion or abnormal deformation. A ground-occupancy map reduces collisions during placement. Lumen handles nearby movable lights, while baked lightmaps supply static illumination and environmental detail.

Each clip has 8 cameras with horizontal fields of view spanning 39.6 to 90 degrees; focal-length ranges vary between indoor and outdoor scenes. Static, tracking, dolly, and orbit motions are augmented with Perlin-noise camera shake. Beyond independently sampled trajectories, the dataset includes paired orbits sharing an initial frame and 4 orbit trajectories starting from 4 orthogonal static views. These configurations expose models to different source-target relationships instead of a fixed stereo baseline. Orbit trajectories still dominate, which limits the camera-control coverage discussed later.

2. Compact Dynamic Point Maps: store surface correspondence and decode trajectories on demand

A source pixel identifies a location on a mesh triangle. With fixed mesh topology, storing the triangle index and three barycentric weights, together with complete vertex trajectories, avoids repeatedly recording the same surface point's motion from every image that observes it. The following restates Eq. (1) using nonnegative barycentric weights that sum to 1:

\[ Q_i(t)(u)=\alpha_1 q_{f_1}(t)+\alpha_2 q_{f_2}(t)+\alpha_3 q_{f_3}(t),\qquad \alpha_1+\alpha_2+\alpha_3=1. \]

Here, \(i\) identifies the source image, \(u\) the source pixel, and \(f_1,f_2,f_3\) the triangle's vertex indices; \(q_{f_j}(t)\) is a vertex position at time \(t\) in a common reference frame. Decoding combines the three vertex trajectories with the stored weights, applies a known rigid transform to the target camera frame, and optionally projects into the target image. A computable projection does not imply visibility: occlusion and points outside the image remain distinct from geometric location.

Unreal has no rendering pass that directly exports the required triangle indices and barycentric weights. The authors therefore unproject pixels using depth and camera parameters, then find the mesh face and barycentric coordinates that best explain each 3D point. Pixel-to-mesh binding is handled during preprocessing, leaving efficient decoding to the training data loader. Static backgrounds do not require repeated dynamic trajectories, and additional compression can exploit the small fraction of dynamic pixels and lower-bit packing.

Even after using a common reference frame, explicitly storing point maps for all target times costs \(O(HWT^2C)\); mesh binding reduces this to \(O(HWTC+VT)\), where \(V\) is the number of vertices. The paper's illustrative configuration uses \(H=W=512\), \(T=300\), \(C=8\), and 4 bytes per scalar: explicit storage requires 2.1 TiB, whereas the compact representation with 100K vertices requires 9.3 GiB. This is a representation-cost example, not evidence that every released clip uses this resolution or has exactly the same duration.

3. Hierarchical Video Captions: supply semantic context for full clips and local windows

A video contains approximately 300 frames. One global caption can miss brief actions, whereas short-window captions alone lose overall scene context. Tarsier2-7B samples 16 frames across the sequence for a global caption; the video is also divided into 81-frame windows, with 32 sampled frames per window for local captions. Users can match caption granularity to training-window length instead of describing a local action with text about the entire sequence.

Captions are dataset metadata; geometric ground truth comes from rendering and meshes, not from the vision-language model. The main text does not separately ablate caption granularity or establish that captions account for the generator's gains. Hierarchical captions should therefore not be presented as an independently validated source of performance improvement.

4. Geometry-Aware Multiview Generation: jointly model appearance and point maps in one diffusion model

ReCamMaster originally generates a target video conditioned on a source video and target camera trajectory. Syn4D supplies paired point maps for each view, allowing the authors to adapt the video encoder to point maps and concatenate appearance and geometry latents along the spatial width dimension rather than adding a separate large prediction network:

\[ Z=[L_{\mathrm{vid}};L_{\mathrm{DPM}}]\in\mathbb{R}^{H\times(2W)\times T\times d}. \]

Here, \(H,W,T\) are latent dimensions and \(d\) is the latent channel dimension. The pretrained diffusion architecture is retained, and the paper states that spatial concatenation introduces no additional model parameters; doubling spatial width does not, however, imply unchanged computation. The model receives source video and point maps and jointly generates target-view video and point maps, enabling paired supervision of appearance and geometry under camera conditioning.

The main text uses point maps at each frame's corresponding time here, rather than explicitly producing every source pixel's full set of DPM queries across time. The dataset's ability to represent long-term trajectories should not be confused with this baseline directly generating an equally complete trajectory tensor.

Loss & Training

The dataset has no single learning objective. The 4RC experiment retains the original architecture and training protocol while adding Syn4D to the original training data; the geometry-aware generator is fine-tuned separately on Kubric and Syn4D. The cached main text does not specify the generator's full loss formulation, learning rate, or complete compute budget, so these details cannot be supplied reliably.

For human estimation, SMPL-X annotations are converted to SMPL through the official optimization-based refitting procedure before continuing from the final MA-HMR checkpoint. Both the Syn4D-augmented model and the control using only AGORA, BEDLAM, and DTO-Humans continue for 6 epochs, with learning rate \(10^{-5}\) and total batch size 128. Segmentation masks filter out frames containing heavily occluded humans.

Key Experimental Results

Main Results

The following extracts Table 3. 4RC (Syn4D) uses the original training data plus Syn4D, not training from scratch on Syn4D alone. APD averages the percentage of points within distance thresholds and is higher-is-better; EPE is trajectory end-point error and is lower-is-better. The cached main text does not enumerate the APD threshold set.

Test Set / Type	4RC APD	+Syn4D APD	4RC EPE	+Syn4D EPE
ADT / Sparse	87.82	89.44	0.1480	0.1258
PStudio / Sparse	87.32	88.10	0.1304	0.1276
Soviet / Dense	58.35	74.46	3.8458	1.8934
Warehouse / Dense	79.07	88.79	0.3302	0.1915

ADT and PStudio come from TAPVid-3D. Soviet and Warehouse are rendered Syn4D test sets with 50 sequences each; a 24-frame clip is sampled per sequence, and dense trajectories start from its first frame. Soviet APD improves by 16.11 percentage points, but this synthetic-domain dense gain should not be equated with an identical improvement on real video.

The next table reproduces the training-data comparison from Table 2, evaluated on 280 video pairs with scenes and dynamic objects unseen during training. Lower FVD and higher CLIP-V are better. The P variants operate on RGB representations of point maps, not on pixelwise geometric errors.

Generator Training Data	CLIP-V	FVD	ATE	RPE-T	RPE-R	CLIP-V-P	FVD-P
Kubric	0.643	631	0.064	0.023	0.328	0.757	229
Syn4D	0.740	452	0.070	0.021	0.272	0.816	139

Back on Track estimates camera trajectories from generated videos, followed by Umeyama alignment to the conditioned trajectories. Appearance and point-map distribution metrics improve, but ATE increases from 0.064 to 0.070; the results do not support a claim of uniformly better camera accuracy.

Ablation Study

The paper does not individually remove scene content, camera patterns, or barycentric encoding. Its clearest controlled experiment is the equal-epoch continued-training comparison in Table 6. The following retains all three Hi4D metrics, measured in mm and lower-is-better.

Config	MPJPE	PA-MPJPE	PVE
MA-HMR	58.8	43.9	73.6
MA-HMR+Cont	58.7	44.1	73.2
MA-HMR+Syn4D	57.7	43.0	72.2

MPJPE measures joint-position error, PA-MPJPE applies Procrustes alignment first, and PVE measures mesh vertex error. Evaluation uses the Hi4D and 3DPW test sets and the CHI3D training set; none of these evaluation datasets is used to train or fine-tune the models.

Key Findings

• Table 3 shows larger gains for dense than sparse tracking, consistent with the goal of complete dynamic correspondence supervision. Different test domains prevent attribution solely to annotation density.
• In Table 4, Sintel camera ATE decreases from 0.144 to 0.076, while TUM-dynamics ATE increases from 0.010 to 0.012, illustrating uneven transfer across tasks and datasets.
• In Table 5, scale-only aligned Sintel depth Rel decreases from 0.311 to 0.211, but Bonn threshold accuracy under the same alignment decreases from 97.4 to 97.3. Not every metric improves.
• In Table 6, continued training alone makes little difference, whereas adding Syn4D improves all three human metrics on all three evaluation datasets. This is stronger evidence for data utility than continued training without a control.

Highlights & Insights

• The central contribution is a queryable annotation representation. Fixed pixel-to-surface bindings reuse vertex motion and address the gap between rendering dense labels and being able to store them.
• One dataset covers geometry, motion, cameras, and human parameters, allowing supervision to be tested across tasks. The unchanged 4RC architecture helps distinguish data gains from network changes.
• Spatial concatenation of video and point-map latents reuses generative priors with a small parameter-level change. It motivates joint appearance and geometry generation without establishing strict physical consistency.

Limitations & Future Work

• The authors explicitly call for more diverse camera patterns, supported by the generation experiment's ATE regression. Broader distributions of paired trajectories are a concrete next step.
• Barycentric binding assumes fixed mesh topology and stable surface correspondence, so it does not directly cover topology-changing fluids or fragmentation. This is an implication of the representation, not a reported failure experiment.
• The 30 environments and filtered animations cannot span all real dynamics. Removing fast motion, abnormal deformation, and heavily occluded human frames may further reduce coverage of extreme cases.
• CLIP-V-P and FVD-P are distribution-level proxies and do not directly verify a physical point's 3D error or long-term trajectory consistency. More direct geometric evaluation is needed.
• Separate ablations of data scale, asset diversity, and camera patterns are missing, making their individual contributions difficult to quantify. The cache also lacks a complete generation-training budget and released-data resolution breakdown.

Related Work & Insights

• vs Kubric: Both provide dense synthetic supervision. Syn4D adds non-rigid animation, complex environments, and an efficient tracking representation rather than merely increasing the number of falling objects.
• vs PointOdyssey / Dynamic Replica: Long-term but sparse annotations restrict arbitrary pixel-level queries. Syn4D extends supervision to dense multiview surface correspondence.
• vs ReCamMaster: The original method focuses on camera-controlled appearance generation; this paper uses paired point maps to extend it to joint geometry generation. Their relationship is a baseline and a data-enabled extension.
• Data-engineering lesson: When a supervision tensor repeats many correspondences, identify a reusable physical or geometric representation and reconstruct task-specific labels during loading. Here that reusable representation is the animated mesh, not a compressed list of independent trajectories.

Rating

• Novelty: 4/5. Dense multiview dynamic annotations and mesh bindings form a clear dataset contribution.
• Experimental Thoroughness: 4/5. Multiple downstream tasks and controlled human-model continuation are strengths, but fine-grained data ablations are missing.
• Writing Quality: 4/5. DPM definitions and complexity arguments are clear, while some training configurations remain incomplete in the cached main text.
• Value: 5/5. Reusable supervision for 4D reconstruction and tracking matters beyond improving one model's scores.