Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flow

Conference: ECCV2026
Paper: ECCV Official Page
Project: https://shenhanqian.github.io/flow4r
Area: 3D Vision
Keywords: scene flow, dynamic scene reconstruction, 3D point tracking, camera ego-motion, pose weights

TL;DR

Flow4R uses a shared two-view Transformer to predict local geometry and cross-frame point positions, then separates camera and object motion through a learned-weight geometric solver, unifying 4D reconstruction and tracking without global post-optimization and achieving 71.1 all-point tracking APD and 81.00 reconstruction APD on Point Odyssey.

Background & Motivation

Recovering a dynamic 3D world from video requires more than estimating depth for each frame: a system must establish where a point is, how the camera moves, and how the object itself moves. Feedforward reconstruction methods such as DUSt3R recover pointmaps effectively, but their shared-reference-frame formulation originates in static scenes. MonST3R extends this approach to dynamic video while retaining post-optimization to organize multi-frame geometry. Other tracking methods add prediction heads for different times or coordinate systems, tying the architecture to particular output tasks.

A more fundamental issue is that being stationary depends on the reference object. Watching an escalator from the ground and watching the ground from the escalator lead to different interpretations of camera and object motion. Specifying a static background first and then separately regressing camera and object motion is therefore not the most flexible modeling order. Meanwhile, real dynamic 3D flow annotations are much scarcer than static depth data, making a system that depends entirely on complete dynamic supervision difficult to train.

The paper first learns observable relative geometry and motion between two frames, then lets reliable pixels determine the reference frame, deriving camera poses, world-coordinate trajectories, and optical flow from one prediction set. Core Idea: connect local geometry and cross-frame correspondences through scene flow, use learned pose weights to determine its rigid-motion interpretation, and train this unified representation jointly on static and dynamic data.

Method

Overall Architecture

The input consists of two RGB images that may differ in both viewpoint and time; two parameter-sharing paths produce 3D properties aligned with their respective pixels. Processing proceeds through Symmetric Property Prediction and Pose-Weight Solving, followed by Anchor-Based Scale Alignment for videos to place independent pair predictions in a consistent reference frame.

The conceptual properties are local point positions \(P\), scene flow \(F\), pose weights \(W\), and confidence \(C\). The implemented network predicts cross-view, cross-time positions \(P_{vt}\) instead of directly outputting \(F\): this changes the regression variable, not the underlying motion task. Each image receives its own property set, naturally supporting bidirectional tracking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image pair or anchored pairs"] --> B["Symmetric Property<br/>Prediction"]
    B --> C["Pose-Weight Solving"]
    C --> D["Anchor-Based<br/>Scale Alignment"]
    D --> E["Consistent-frame reconstruction<br/>and 3D trajectories"]
    B --> F["Projection to optical flow"]
    G["Static and dynamic annotations"] -.->|Training only: geometry and motion supervision| B
    G -.->|Training only: pose supervision| C

Dashed edges represent training supervision, not additional depth, ground-truth pose, or motion inputs required at inference time. A single image pair does not require sequence-level scale alignment. Optical flow can also be obtained directly by projecting the properties, without first removing camera motion.

Key Designs

1. Symmetric Property Prediction: regress motion endpoints directly

The two images pass through shared ViT encoders, decoders that exchange cross-view information, and prediction heads. Unlike architectures with separate heads for different timestamps, both paths share their encoders, decoders, and heads. Swapping the inputs preserves the same semantics, eliminating the need to manually construct symmetrized image pairs as in DUSt3R. The high-resolution training stage uses a DPT head to recover dense, pixel-aligned outputs.

Here, \(P\) locates a source-image pixel in the source camera's local space, while \(P_{vt}\) locates the same physical point at the target time in the target camera's space. The viewpoint and timestamp both change, retaining the combined effects of observer and object motion instead of excluding foreground motion in advance. The source explicitly states the following relationship:

\[ F = P_{vt} - P. \]

Regressing \(F\) and adding \(P\) is representationally equivalent to directly regressing \(P_{vt}\), but the two choices need not be equally easy to learn. Table 3 finds that the latter performs better in all six evaluation columns. The authors suggest that directly supervising motion endpoints better matches evaluation on absolute point positions and reduces error accumulation when constructing positions from intermediate quantities. This is an empirical explanation, not a proof that equivalent parameterizations must have a particular optimization advantage.

The claim of reference-frame independence needs a qualified reading: predicting relative motion does not require choosing which scene object is static beforehand. It does not mean that numerical 3D coordinates remain unchanged under arbitrary coordinate transformations. Both \(P\) and \(P_{vt}\) still follow explicit camera-coordinate conventions.

2. Pose-Weight Solving: let reliable correspondences determine camera motion

Total relative motion contains object motion, so fitting a camera pose with equal weights for all correspondences is inappropriate. Flow4R predicts normalized per-pixel weights \(W\) and uses weighted least squares to fit a rigid transformation from \(P\) to \(P_{vt}\). This transformation provides the camera-motion interpretation. Point positions after changing viewpoint but not time are denoted by \(P_v\), from which rigid flow is obtained; the residual describes non-rigid motion. To recover a trajectory in the source reference frame, the estimated inverse pose maps \(P_{vt}\) back into source coordinates.

Consequently, \(W\) is not an ordinary foreground segmentation map. Static foreground objects may support localization, whereas moving, distant, reflective, textureless, or occluded pixels may be unreliable for pose estimation. The separate output \(C\) represents confidence in 3D predictions and weights geometry and motion regression; it should not be confused with \(W\), which indicates usefulness for camera-pose estimation. The paper constrains pixel weights in \(W\) to sum to 1 and confidence \(C\) to exceed 1.

The crucial question is how to train weights without ideal pixel-level labels. A differentiable pose solver estimates a transformation, and the transformed pointmap is compared with the result specified by ground-truth pose and geometry. Gradients through \(P\) and \(P_{vt}\) are stopped for this loss, leaving only \(W\) to be optimized. The model learns which points yield a correct camera-motion estimate rather than copying a semantic foreground mask. Training still uses ground-truth depth and poses: self-supervision here means that direct labels for \(W\) are unnecessary, not that the entire pipeline is annotation-free.

At inference time, \(W\) can be adjusted or overridden to change the reference region used to interpret motion, without training another pose-regression head. Camera intrinsics also have a geometric route: focal length is estimated from reprojection consistency between the local pointmap and pixel positions, assuming identical focal lengths along both axes. Projecting source points and motion endpoints and taking their image-plane difference yields optical flow. The formulation does not require a separate neural head for every downstream task.

3. Anchor-Based Scale Alignment: calibrate independent pairs through a shared frame

For video, the first frame is fixed as an anchor, forming pairs such as \((I_0,I_1)\), \((I_0,I_2)\), and \((I_0,I_3)\) instead of relying solely on stepwise propagation between neighboring frames. Every prediction includes the anchor's local pointmap, providing a shared object whose predicted scale can be compared across pairs rather than guessing scale between point clouds without a common reference.

The paper defines \(s_n\) as the mean Euclidean norm of the anchor pointmap predicted for pair \(n\). Taking the first pair as the scale reference, subsequent pointmaps and scene flows are multiplied by \(s_1/s_n\). This establishes scale consistency across pairs, but it does not imply recovery of absolute metric scale without external alignment. It is also not global bundle adjustment: all poses and points are not jointly optimized over the video, so sequence quality still depends on pairwise predictions and anchor visibility.

A Worked Example

Consider a camera moving around a jumping robot in a 4-frame video, with frame 0 as the anchor. The model processes 3 anchored image pairs. For a robot-surface pixel in frame 0, it predicts both the current location and the location at each target time in the target camera's space. The resulting total motion therefore contains both camera movement and the robot's jump.

Pose weights allow reliable walls and floor regions to dominate rigid fitting while reducing contributions from the jumping robot. Mapping the robot's endpoints back into anchor coordinates through the inverse pose produces its trajectory with camera ego-motion removed; background points should remain relatively stable. Finally, the common anchor pointmap in the 3 predictions aligns their scales so that the trajectories can be displayed together. This example explains the data flow and is not an additional quantitative experiment reported in the paper.

Loss & Training

Training begins by normalizing ground-truth and predicted pointmaps by their respective mean Euclidean norms, with identical supervision for both symmetric paths. The point-position loss uses local pointmaps obtained by back-projecting ground-truth depth with camera intrinsics, combining confidence-weighted regression with a logarithmic confidence term. A 3D motion loss supervises motion endpoints wherever scene-flow or 3D trajectory annotations are available; these two losses share \(C\).

When only optical flow or 2D tracks are available, motion endpoints are projected into the image plane for 2D supervision, without 3D confidence weighting. This accommodates dense optical flow, often restricted to adjacent frames, as well as sparse tracks that span longer intervals. Their supervision is not equivalent: neither should automatically be treated as complete 3D motion ground truth.

Static data provides abundant, inexpensive rigid-motion supervision: ground-truth depth and relative camera poses specify point positions in the target view. On dynamic data, this rigid target is appropriate only for static regions, so the loss uses stop-gradient \(W\) multiplied by the number of image pixels to suppress moving objects. On static data, the weight is simply 1. Blocking this loss from updating \(W\) prevents the weights from degenerating merely to avoid rigid-regression errors, complementing the pose loss that updates only \(W\).

Operators are missing from the cached pose least-squares equation, some losses, and the total-loss equation, so their exact formulas and coefficient assignments are not reconstructed here. The mechanisms above follow the intact explanatory text surrounding those equations. The source clearly specifies point-position, 3D motion, 2D motion, pose-weight, and rigid-motion supervision.

The model is initialized from CroCo rather than directly fine-tuned from DUSt3R or MonST3R. Stage one uses resolution 224 and a linear head for 100 epochs, sampling 900K pairs per epoch. Stage two uses resolution 512, random aspect ratios, and a DPT head for 100 epochs, sampling 84K pairs per epoch. Video pairs are sampled within a 50-frame neighborhood of the same scene, so training should not be described as joint processing of arbitrarily long videos.

Optimization uses Adam, with warmup over 10 and 20 epochs in the respective stages, reaching a learning rate of 1e-4 before cosine decay to 1e-6. Gradient norms are clipped to 10. Total batch sizes are 256 and 64, using 8 NVIDIA A100/H100 GPUs, and full training takes about 4 days. Training mixes real and synthetic, static and dynamic datasets, including Virtual KITTI 2 for scene flow, Spring for optical flow, and PointOdyssey for 3D tracks.

Key Experimental Results

Main Results

WorldTrack evaluates trajectories in world coordinates, jointly testing geometry, pose, and motion rather than only 2D correspondence. After aligning predictions with ground truth, APD3D measures the fraction of points with errors below 0.1, 0.3, 0.5, and 1.0 m, aggregated across thresholds and the first 64 frames; higher is better. Reconstruction also reports endpoint error, EPE, for which lower is better. Table 1 uses global median alignment and Table 2 uses global median scaling, so neither should be read as an unaligned absolute-scale evaluation.

The following selects three methods from the source's Table 1. ADT denotes Aria Digital Twin, DR Dynamic Replica, PO Point Odyssey, and PS Panoptic Studio. APD is shown on the source's percentage scale, and parameter counts are in billions.

Method	All ADT	All DR	All PO	All PS	Dynamic ADT	Dynamic DR	Dynamic PO	Parameters
MonST3R	74.4	58.1	33.5	51.3	67.9	51.9	39.4	0.7
St4RTrack	76.0	73.7	68.0	69.7	75.3	68.1	68.7	0.7
Flow4R	78.6	78.5	71.1	64.3	70.9	77.2	72.9	0.4

Reconstruction results from the source's Table 2 follow. GA denotes global-alignment post-optimization; the other two rows are feedforward methods.

Method	PO APD ↑	PO EPE ↓	TUM-Dynamics APD ↑	TUM-Dynamics EPE ↓
MonST3R+GA	72.31	0.263	63.87	0.343
St4RTrack	78.73	0.205	83.42	0.185
Flow4R	81.00	0.182	79.87	0.202

Relative to St4RTrack, Flow4R improves PO reconstruction APD by 2.27 percentage points and reduces EPE by 0.023, but its TUM-Dynamics APD is 3.55 percentage points lower. The supported conclusion is competitiveness across several datasets, not universal superiority in dynamic reconstruction.

Ablation Study

The source's Table 3 trains for only half the full number of epochs to compare motion representations. Every metric is all-point APD3D, so differences from fully trained main results cannot be attributed to a single module. The first column is the direct prediction variable and the second is the supervision target; the second row constructs endpoints from predicted flow before supervising them.

Direct Prediction	Supervision Target	Tracking ADT	Tracking DR	Tracking PO	Tracking PS	Reconstruction PO	Reconstruction TUM
\(F\)	\(\bar F\)	78.03	73.26	60.23	55.80	69.36	79.78
\(F\)	\(\bar P_{vt}\)	77.72	76.41	61.21	63.69	66.29	80.05
\(P_{vt}\)	\(\bar P_{vt}\)	78.50	78.48	67.93	67.17	77.20	80.34

With endpoint supervision held fixed, changing the direct prediction from \(F\) to \(P_{vt}\) raises PO tracking from 61.21 to 67.93 and PO reconstruction from 66.29 to 77.20. This explains the implemented endpoint regression more concretely than the general claim of unifying tasks through scene flow.

Key Findings

• The full model achieves 77.2 on DR dynamic-point tracking versus St4RTrack's 68.1, a gain of 9.1 percentage points. On ADT dynamic points, however, the comparison is 70.9 versus 75.3, showing that gains are scene-dependent.
• Table 4 measures Flow4R at 26.8 pairs/s and 3152 MB on an RTX PRO 6000, versus 27.9 pairs/s and 6711 MB for St4RTrack. Memory usage falls by more than 50%, while throughput is slightly lower; this is not an across-the-board speed improvement.
• Weight-map visualizations suppress moving people, robots, and some textureless regions, supporting the localization-reliability interpretation. The main-text ablation chiefly tests motion representation and does not separately quantify every loss and weight-learning component.

Highlights & Insights

• The unification comes from geometric relationships rather than accumulating task heads. The same point positions and motion endpoints produce optical flow through projection and poses through rigid fitting, making the outputs explicitly related.
• Training pixel reliability through a differentiable solver better matches the task than treating semantic foreground as a motion mask. The transferable idea is to learn weights from downstream geometric correctness instead of manually enumerating every type of unreliable pixel.
• Preserving both images' local geometry provides not only bidirectional outputs but also a shared reference for scale calibration across pairs. The choice of intermediate predictions directly affects whether sequence inference needs additional optimization.

Limitations & Future Work

• The authors identify the scarcity of high-quality real 3D scene-flow data as a continuing constraint. Static data and synthetic motion supervision cannot eliminate every domain gap in real dynamic scenes.
• The current model remains two-view: anchored pairs and scale calibration are not joint multi-view modeling. The authors suggest multi-view attention mechanisms similar to VGGT to improve temporal consistency and reconstruction quality.
• Real-time online tracking under strict resource limits remains an acknowledged open problem. The reported pairs/s is image-pair throughput on a specific GPU, not the frame rate of a complete online system.
• Regarding evaluation scope, the first 64 frames and global-alignment protocol do not establish arbitrary-length tracking, absolute-scale recovery, or long-term occlusion recovery. Testing failure rates when the anchor loses overlap and exploring adaptive anchor switching are further directions proposed by this note.

Related Work & Insights

• vs DUSt3R / MASt3R: Flow4R retains the two-view Transformer route to geometry learning but explicitly organizes local geometry and cross-time correspondence while sharing both decoders and prediction heads. Its objective extends beyond static multi-view pointmaps.
• vs MonST3R: MonST3R organizes dynamic scenes with local temporal connections and post-optimization; Flow4R uses anchored connections, pose solving, and scale alignment. Removing global post-optimization does not automatically preserve all global consistency constraints.
• vs St4RTrack / POMATO: Flow4R emphasizes symmetric bidirectional properties and local pointmaps for both views, while adopting anchored sequence processing from St4RTrack. The paper evaluates POMATO's pairwise model because its sequential model tracks in the opposite direction to the standard anchor-to-other-frame setting used by the other methods.
• vs VGGT: Joint multi-view attention enables stronger cross-frame interaction. Flow4R does not implement this extension yet; it provides a compact geometric representation that could be combined with it.

Rating

• Novelty: 4/5. The contribution centers on symmetric representation, learned geometric solving, and joint static-dynamic supervision rather than a new Transformer backbone.
• Experimental Thoroughness: 4/5. Evaluation covers tracking, reconstruction, efficiency, and motion parameterization across multiple datasets, but separate ablations of training components remain limited.
• Writing Quality: 4/5. The geometric explanation and visualizations are clear; readers must distinguish conceptual flow prediction from implemented endpoint outputs and qualify the meaning of reference-frame independence.
• Value: 4/5. The method offers a reusable route toward memory-efficient unified 4D perception, while real long-sequence and online systems require further validation.