Skip to content

Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/ljjTYJR/Multiple-view-Dynamic-Reconstruction
Area: 3D Vision
Keywords: dynamic scene reconstruction, multi-camera visual SLAM, camera pose estimation, dense depth estimation, spatio-temporal connection graph

TL;DR

This paper proposes a two-stage optimization framework for the setting in which several freely moving cameras film the same dynamic scene: a feed-forward reconstruction model (VGGT) provides wide-baseline initialization and a shared scale anchor, a spatio-temporal connection graph unifies intra-camera temporal, same-timestamp inter-camera, and cross-camera historical constraints into a joint multi-camera bundle adjustment, and a final stage uses wide-baseline dense optical flow for frame-level scale alignment followed by alternating pose and per-pixel depth refinement. It outperforms COLMAP/GLOMAP and VGGT/Fast3R/FastVGGT/CUT3R in pose accuracy, scene consistency, and GPU memory, and releases the motion-capture-grounded MultiCamRobolab dataset.

Background & Motivation

Multi-camera capture is becoming everyday practice: multi-camera rigs on robot platforms, multi-angle sports broadcasting, and several phones or action cameras recording the same event. Downstream tasks such as AR, dynamic Gaussian Splatting, and multi-view video analysis all require 3D reconstructions that are consistent across cameras, yet today's dynamic scene reconstruction methods are almost all restricted to monocular input — robust dynamic monocular SLAM such as MegaSAM exploits only temporal connections inside one camera and throws away cross-camera observations entirely. The other line, multi-camera SLAM, either assumes cameras rigidly mounted on a pre-calibrated rig with known extrinsics, or performs multi-agent/multi-session collaborative mapping in which each agent builds a local static sub-map that is fused later; neither handles dynamic content. The closest prior work can reconstruct a mesh of dynamic objects in view, but it requires fixed cameras and prior extrinsic calibration.

Once the setting is relaxed to freely moving cameras plus dynamic content, three technical difficulties appear immediately. First, scale ambiguity: monocular depth is inherently scale-ambiguous, and without shared observations each camera's reconstruction drifts to its own scale. Second, limited overlap: unlike a rigid rig with fixed overlapping fields of view, free-moving cameras may share little or no overlap, which makes the conventional "pick initialization frames by overlap" recipe unreliable. Third, dynamic content: moving objects violate the static-world assumption underlying classical multi-view geometry, so the correspondences themselves are untrustworthy. Added to this is a practical constraint from the existing toolchain — feed-forward reconstruction models blow up in memory on long sequences (VGGT cannot process all frames even on an A100-40GB), while chunk-wise processing is worse than global prediction.

The paper's response is to split the problem: rather than letting one model do poses and depths at once, it decouples camera tracking from dense depth refinement into two stages. Stage one gets a global scale anchor and coarse poses from a feed-forward model, then extends the sliding-window BA of monocular SLAM into multi-camera joint optimization through a spatio-temporal connection graph. Stage two abandons sparse features, builds cross-camera constraints from wide-baseline dense optical flow, first aligns frame-level affines, and then alternates between per-pixel depth and camera pose optimization. Core idea: wire the observations of several free cameras into a single optimization problem via a connection graph with three edge types — temporal, same-instant spatial, and cross-camera historical — and replace "detect dynamic objects then delete them" with learnable soft confidence weighting.

Method

Overall Architecture

The input is \(N\) time-synchronized monocular video streams \(\{(\mathbf{I}_i^t,\mathbf{K}_i)\}\) (\(i=1,\dots,N\), \(t=1,\dots,T\)); only each camera's own intrinsic \(\mathbf{K}_i\) is known, with no extrinsics or relative positions between cameras. The output is a camera pose \(\mathbf{T}_i^t\in SE(3)\) and a dense depth map \(\mathbf{D}_i^t\) for every frame.

The method is a two-stage optimization pipeline. Stage one performs multi-camera tracking: the first few frames of each stream are fed to the feed-forward reconstruction model VGGT to obtain rough but globally consistent initial poses and depths, and UniDepth's monocular depths are aligned to that scale with a single global scale/offset pair to serve as a depth prior in later optimization. A spatio-temporal connection graph is then built incrementally (intra-camera temporal edges, inter-camera spatial edges at the same timestamp, and cross-camera edges between the current keyframe and historical keyframes of other cameras); new frames are initialized under a constant-velocity assumption, and poses plus per-frame disparities are jointly optimized inside a sliding window. Stage two performs multi-view dense consistency refinement: the connection graph is augmented with extra long-range temporal edges at offsets \(\{+2,+4,+8\}\) inside each camera, all edges are re-matched with the wide-baseline flow model UFM, and the optimization first freezes poses and solves only per-frame scale/shift and per-flow confidence (Phase 1), then freezes those affine parameters and alternately optimizes per-pixel depth and camera pose with trajectory smoothness regularization (Phase 2), finally emitting consistent multi-camera depths and refined poses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multiple time-synchronized RGB streams"] --> B["Wide-baseline initialization<br/>VGGT coarse poses + monocular depth alignment"]
    B --> C["Spatio-temporal connection graph<br/>temporal / spatial / spatio-temporal edges"]
    C --> D["Confidence-weighted joint optimization<br/>reprojection + depth prior regularization"]
    D --> E["Two-phase dense refinement<br/>frame-level affine → alternating pose and depth"]
    E --> F["Dense depth maps + camera poses"]

Key Designs

1. Wide-baseline initialization: a feed-forward reconstruction model establishes a shared scale and a reliable starting point for free cameras

Single-camera tracking can be initialized by "picking two frames with enough overlap", which video streams guarantee by temporal continuity; but under a multi-camera setup, images from different viewpoints may share no common field of view at all, so overlap-based initialization simply fails. The paper instead calls the robust feed-forward scene reconstruction model VGGT: the first frames of every stream are passed in together, yielding initial camera pose estimates and per-frame depths \(\mathbf{D}_{\text{VGGT}}\). These predictions are coarse, but they are consistent across cameras, which makes them usable as the starting point for the whole system.

Poses alone are not enough — the tracking BA also needs dense depth priors to constrain textureless regions, and monocular depth models are only defined up to an affine transform. The authors therefore predict per-frame monocular depth \(\mathbf{D}_{\text{mono}}\) with UniDepth and fit a global scale \(s\) and offset \(o\) by least squares, aligning all monocular depth to the VGGT scale:

\[(s,o)=\arg\min_{s,o}\sum_{k}\left\|s\mathbf{D}^k_{\text{mono}}+o-\mathbf{D}^k_{\text{VGGT}}\right\|^{2}\]

where \(k\) indexes the selected initialization frames. This single \((s,o)\) pair is then applied consistently to every monocular depth prediction from every camera (the reference depth used in tracking is \(\mathbf{D}^s_i=s\mathbf{D}^{\text{mono}}_i+o\)), and that is where the "shared scale anchor" comes from: cross-camera scale consistency is not established by overlapping observations between cameras, but by making all cameras share one affine parameter set derived from the VGGT scale. The design is also fairly insensitive to initialization noise — an extra experiment adds 3° rotation noise plus varying translation noise, and the ATE only rises from 0.013 to 0.016 at 0.05 m before deteriorating to 0.044 at 0.10 m, showing that the later two stages can genuinely recover from this starting point.

2. Spatio-temporal connection graph: extending inter-frame constraints from single-camera temporal links to cross-camera spatio-temporal ones

The backbone of tracking is a frame connection graph \(\Omega=\Omega^{\text{temp}}\cup\Omega^{\text{spat}}\cup\Omega^{\text{st}}\) that determines which image pairs receive reprojection constraints. \(\Omega^{\text{temp}}\) follows monocular SLAM practice and keeps a temporal window holding only the most recent keyframes for each camera, since consecutive frames of one camera always overlap sufficiently. \(\Omega^{\text{spat}}\) is responsible for cross-camera scale consistency: at timestamp \(t\), the pixel grid of camera \(i\) is projected onto camera \(j\)'s image plane using the current state, and a spatial connection is created if more than 75% of the projected points fall inside the image boundary. \(\Omega^{\text{st}}\) mines longer history: the current keyframe evaluates overlap against the inactive (historical) keyframes of all other cameras and connects whenever the overlap suffices. To keep the graph from growing without bound on long sequences, a connection-balance strategy caps the number of edges in the tracking active window, distributes inter-camera connections evenly when they exist, and removes the oldest edges when new ones would exceed the cap.

Spatial edges remain valid in dynamic scenes because different views at the same instant observe the same momentarily frozen scene — even with moving objects, the geometry at that instant is consistent across cameras; only over tens of frames does the motion invalidate the correspondence, which is why cross-temporal edges need confidence weighting as a safety net. The ablation confirms the value of this graph directly: removing it (the method degenerates into reconstructing each stream independently and aligning them with the feed-forward model's poses) degrades ATE by 0.158 on RoboDogoverlap and 0.141 on DynamicHuman, but only by 0.006 on RoboDognon-overlap where the cameras hardly share any field of view — exactly the signature one expects if its mechanism is "turn cross-camera overlap into constraints", since with no overlap there is nothing to exploit.

3. Confidence-weighted joint optimization: unreliable correspondences down-weight themselves instead of being cut away by motion masks

Dynamic objects violate the static assumption, and the apparently direct fix is to segment dynamic regions and remove them from the optimization. This paper does not do that; instead every correspondence carries its own reliability weight. In the tracking stage the reprojection error is weighted by the diagonal matrix \(\Sigma_{ij}=\text{diag}(w_{ij})\) (reusing the learned matching weights of the correspondence model, which are naturally low on dynamic and occluded pixels), and the sliding window minimizes

\[\mathcal{L}(\mathbf{T},\mathbf{d})=\sum_{(i,j)\in\Omega}\left\|\mathbf{u}^{\text{reproj}}_{ij}-\mathbf{u}^{\text{flow}}_{ij}\right\|^{2}_{\Sigma_{ij}}+\lambda\left\|\mathbf{d}_i-\mathbf{D}^{s}_i\right\|^{2}\]

The first term requires geometric reprojection to agree with the optical-flow correspondence, and the second is a depth-prior regularizer pulling the estimated disparity toward the aligned monocular depth, preventing pure geometric optimization from drifting in textureless regions. In the refinement stage the confidence becomes an explicit optimizable variable \(c_i\in(0,1]\) that weights the flow residual, together with a \(\log(1/c_i)\) term — without it, the optimizer would drive \(c_i\) to zero and simply switch the whole residual off, so the log term exists precisely to prevent that collapse.

The authors argue for this choice through the failure of the baselines: adding dynamic masks to COLMAP/GLOMAP does not always help, and on the DynamicHuman scene both methods fail outright after masking (recorded as failure in the tables, unable to register all images); only on RoboDogoverlap does masking improve COLMAP's ATE from 0.134 to 0.045. They offer three explanations: under synchronized cameras a dynamic object is momentarily consistent across viewpoints and thus supplies valid inter-camera correspondences; at 30 FPS the object moves little within a short time window so features stay locally consistent; and in textureless indoor environments dynamic objects are often exactly where keypoints come from. Hard deletion throws those observations away, whereas soft weighting keeps them and merely lowers their influence.

4. Two-phase dense refinement: align frame-level scale first, then alternate between depth and pose

The single global \((s,o)\) fitted at initialization cannot cover two things: per-frame scale drift of monocular depth over a long video, and the monocular model's per-pixel error. The refinement stage therefore reorganizes the constraints. The connection graph is first augmented into \(\Omega^{\text{refine}}\) (adding long-range temporal edges at offsets \(\{+2,+4,+8\}\) frames for each frame), and all edge pairs are re-matched with the wide-baseline dense flow model UFM — the flow used during tracking is low-resolution and not good enough under large baselines and large displacements. The objective sums a flow residual term and a disparity consistency term:

\[\mathcal{L}_{\text{reproj}}=w_f\mathcal{L}_{\text{flow}}+w_d\mathcal{L}_{\text{disp}}\]

\(\mathcal{L}_{\text{flow}}\) penalizes disagreement between optical-flow correspondences and geometric reprojections, with each flow weighted by its confidence \(c_i\) and normalized by the valid-flow mask \(m_{ij}\); \(\mathcal{L}_{\text{disp}}\) penalizes inconsistency between the depth \(D^{\text{flow}}_j\) warped from frame \(i\) by optical flow and the depth \(D_j\) estimated at frame \(j\), which suppresses depth flickering. (The typesetting of these two equations was destroyed when the cached text was scraped; they are reconstructed here from the prose description — ⚠️ refer to the original paper for the exact norm and weighting.) Crucially, the scale used here is a per-frame affine pair \(s_i,\beta_i\), independent of the global \(s,o\) from initialization: reprojection is computed with \(s_i\mathbf{D}_i+\beta_i\), so each stage keeps its own scale freedom.

The optimization then alternates two steps. Phase 1 freezes all poses and optimizes only the per-frame \((s_i,\beta_i)\) and the per-flow confidences \(c_i\), first pulling all frames onto a consistent scale and letting unreliable correspondences down-weight themselves. Phase 2 freezes the affine parameters and directly optimizes per-pixel depth values \(D_i\) while refining camera poses. The authors deliberately alternate pose and depth rather than optimizing them jointly — following MegaSAM's experience that releasing both at once is unstable. The pose side parameterizes the increment by the exponential map (\(\Delta_i=\exp(\delta_i)\in SE(3)\) with \(\delta_i\in\mathbb{R}^6\) the Lie-algebra rotation and translation) and adds the regularizer \(\mathcal{L}_{\text{pose}}=\mathcal{L}_{\text{prior}}+\mathcal{L}_{\text{smooth}}\) on top of the reprojection loss: the first penalizes deviation from the tracking-stage pose estimate, the second enforces temporal smoothness along each camera trajectory (the Frobenius norm of \(R_i^{t\top}R_i^{t+1}\) close to the identity, and the translation term \(t_i^t-t_i^{t+1}\) close to zero). The overall objective is \(\mathcal{L}=\mathcal{L}_{\text{reproj}}+\mathcal{L}_{\text{pose}}\). Ablations show both phases are necessary: with Phase 1 only, RoboArm still lags by +0.044 in Abs.Rel and +0.087 in Md; with neither phase, DynamicHuman's Md lags by +0.063 — frame-level alignment cannot fix per-pixel error.

Loss & Training

This is a pure test-time optimization framework with no training whatsoever; VGGT, UniDepth, MegaSAM, and UFM are all invoked as off-the-shelf models. The tracking stage minimizes the weighted reprojection error plus a depth-prior regularizer, with the earliest poses in the window fixed to remove gauge freedom and new frames initialized by the constant-velocity model \(\mathbf{T}_i^t\leftarrow\mathbf{T}_i^{t-1}(\mathbf{T}_i^{t-2})^{-1}\). The refinement stage minimizes the sum of the flow residual and disparity consistency, plus the prior and trajectory-smoothness terms on the pose side, solved by the "affine first, then alternate pose and depth" two-phase schedule. The hyper-parameters (number of initialization frames, maximum edges in the active window, \(\lambda\), \(w_f\), \(w_d\), smoothness weights) are not given numerically in the cached body text — ⚠️ refer to the original paper and its supplementary material.

Key Experimental Results

Main Results

Pose evaluation treats the trajectories of all cameras as a single trajectory and reports Absolute Translation Error (ATE), Relative Translation Error (RTE), and Relative Rotation Error (RRE) after aligning with the initial ground-truth pose. Depth is measured by Abs.Rel and \(\delta<1.25\) (fraction of predicted depths within a 1.25 factor of the true depth), and scene consistency by the median Euclidean distance Md between corresponding 3D points of the predicted and ground-truth reconstructions (after deriving a global scale from trajectory alignment and rescaling the depths).

Table 1 shows pose results on the synthetic MultiCamVideo dataset and on the three-camera subset of the real dataset:

Method MultiCamVideo ATE↓ RTE↓ RRE↓ MultiCamRobolab-3cam ATE↓ RTE↓ RRE↓
COLMAP 0.073 0.004 0.110 0.343 0.013 0.376
VGGT OOM OOM OOM – – –
VGGT† (interval=8) 0.027 0.016 0.177 – – –
Fast3R 0.144 0.056 1.948 0.376 0.154 1.825
FastVGGT 0.023 0.008 0.080 0.032 0.017 0.363
CUT3R 0.175 0.011 0.164 0.474 0.035 0.467
Ours 0.005 0.001 0.011 0.020 0.011 0.326

Table 2 shows ATE and peak GPU memory for the four MultiCamRobolab scenes (× means reconstruction failure, i.e. not all images were registered; VGGT OOMs everywhere without subsampling):

Method RoboDogoverlap RoboDognon-overlap RoboArm DynamicHuman Peak GPU (GB)
COLMAP 0.134 × 0.008 0.133 –
COLMAP + dynamic masks 0.045 × 0.008 × –
GLOMAP × × 0.006 0.020 –
VGGT† (interval=8) 0.024 0.019 0.006 0.032 20.45
Fast3R 0.148 0.701 0.063 0.180 39.20
FastVGGT 0.021 0.020 0.006 0.020 22.08
CUT3R 0.277 0.377 0.055 0.196 22.43
Ours 0.011 0.020 0.005 0.013 20.04

Table 3 reports depth quality and scene consistency (Abs.Rel↓ / \(\delta_{1.25}\)↑ / Md↓; VGGT OOMs on all frames and is omitted):

Method RoboDogoverlap RoboDognon-overlap RoboArm DynamicHuman
VGGT† (interval=8) 0.068 / 0.939 / 0.376 0.060 / 0.947 / 0.327 0.171 / 0.772 / 0.547 0.194 / 0.900 / 0.461
Fast3R 0.144 / 0.788 / 0.580 0.157 / 0.749 / 1.501 0.191 / 0.714 / 0.763 0.214 / 0.735 / 1.799
FastVGGT 0.026 / 0.978 / 0.128 0.018 / 0.988 / 0.095 0.060 / 0.906 / 0.292 0.030 / 0.987 / 0.185
CUT3R 0.048 / 0.968 / 0.715 0.040 / 0.987 / 0.581 0.158 / 0.745 / 0.708 0.060 / 0.963 / 0.503
Ours 0.011 / 0.989 / 0.091 0.018 / 0.991 / 0.092 0.059 / 0.947 / 0.289 0.030 / 0.971 / 0.112

Ablation Study

Table 4 ablates the two core components; the numbers are deviations relative to the full method (positive = worse; the negative RRE entries mean that configuration was marginally better in rotation):

Config RoboDogoverlap ATE RoboDognon-overlap ATE RoboArm ATE DynamicHuman ATE
w/o wide-baseline init. +0.284 +0.496 +0.084 +0.130
w/o spatio-temporal graph +0.158 +0.006 +0.012 +0.141
Full method 0.011 0.020 0.005 0.013

Table 5 isolates the two refinement phases (again deviations from the full method; ✓ = enabled):

Config RoboDogoverlap Abs.Rel / Md RoboDognon-overlap RoboArm DynamicHuman
× Phase1; × Phase2 +0.020 / +0.066 +0.014 / +0.060 +0.068 / +0.226 +0.028 / +0.063
✓ Phase1; × Phase2 +0.012 / +0.029 +0.005 / +0.012 +0.044 / +0.087 +0.003 / +0.012
✓ Phase1; ✓ Phase2 (full) 0.011 / 0.091 0.018 / 0.092 0.059 / 0.289 0.030 / 0.112

Key Findings

  • Wide-baseline initialization is the most critical component: without it, ATE on the non-overlapping RoboDognon-overlap scene deviates by +0.496, essentially destroying that scene — confirming that when free cameras share no overlap, an external reconstruction model must supply the prior. The spatio-temporal graph contributes most on overlapping scenes (RoboDogoverlap +0.158, DynamicHuman +0.141) and only +0.006 where there is no overlap.
  • Dynamic objects do not have to be removed: adding dynamic masks to COLMAP/GLOMAP makes reconstruction fail outright on DynamicHuman, and only pays off on RoboDogoverlap (COLMAP ATE 0.134→0.045). Momentary cross-view correspondence from dynamic objects under synchronized cameras and the keypoints they supply in textureless scenes are observations a hard mask destroys.
  • The ranking of feed-forward models is informative: FastVGGT is the best of all feed-forward baselines (processing all frames together beats CUT3R's test-time recurrent optimization), whereas VGGT itself OOMs even on an A100-40GB and must be subsampled at interval=8 in evaluation; the subsampled VGGT† lands at ATE 0.024–0.032 on MultiCamRobolab, showing that trading frames for memory costs accuracy.
  • Good per-frame depth does not imply scene consistency: on the non-overlapping scene CUT3R's single-frame depth has Abs.Rel 0.040 (better than VGGT†'s 0.060), yet its scene consistency Md is 0.581, more than six times the proposed method's — its depth is accurate per frame but not self-consistent across frames or cameras.
  • The method does not win every cell: on RoboDognon-overlap it ties FastVGGT in ATE (0.020 each) and only leads in RTE (0.003 vs 0.013) and RRE (0.163 vs 0.266); on DynamicHuman's \(\delta_{1.25}\) it is slightly below FastVGGT (0.971 vs 0.987), though its Md on the same scene is clearly better (0.112 vs 0.185). The defensible claim is "best overall with the lowest memory", not a clean sweep.
  • The accuracy-plus-memory combination is the real advantage: alongside the best pose accuracy, peak memory is 20.04 GB, below FastVGGT (22.08 GB), CUT3R (22.43 GB), and Fast3R (39.20 GB), because the feed-forward model is called only once on a short sequence at initialization while the long sequence goes to the sliding-window BA.

Highlights & Insights

  • Dynamic objects are reframed from contamination into exploitable observations: synchronized cameras see one geometrically consistent instant, so a moving object's cross-view correspondences are valid at that moment. This shift in viewpoint leads directly to soft confidence weighting instead of hard masking, and the data where COLMAP/GLOMAP degrade after masking nails the argument down.
  • Graph edges are decided by a geometric criterion rather than an appearance one: a cross-camera connection requires "more than 75% of projected pixels inside the image", needing neither feature-match counts nor retrieval similarity, so the graph can still be built in textureless laboratory scenes.
  • Two-level scale parameterization: one global \((s,o)\) anchors all cameras to a shared scale at initialization, while per-frame \(s_i,\beta_i\) absorb frame-level drift during refinement — the conflicting needs for a unified scale and for tolerating per-frame drift are assigned to two different stages instead of being forced onto one parameter set.
  • A reusable division of labor between feed-forward models and optimization-based BA: let the feed-forward model do only what it is good at and only needs short sequences for — providing the scale anchor and coarse poses — and hand long-sequence accuracy and scalability to a sliding-window optimizer. This split transfers to any long-sequence geometry task with a strong prior model but memory or sequence-length limits (large-scale SfM, long-video depth, SLAM initialization).
  • \(\log(1/c_i)\) against confidence collapse: whenever per-sample confidence is a free variable in a robust loss, the optimizer pushes it to zero to switch the residual off; the log penalty is the general-purpose fix and carries over to any confidence-weighted optimization.
  • Collection and evaluation are separated in the dataset: RGB-D cameras and a motion-capture system are used to record, but the method consumes only RGB and the depth and pose ground truth serve evaluation alone. This "rich at collection, strict at input" recipe is worth reusing for other geometry benchmarks.

Limitations & Future Work

  • Time-synchronized input is assumed: the dataset aligns RGB and motion-capture timestamps with a time server on the Qualisys PC, and the method itself assumes the streams are already synchronized; clock drift, different frame rates, and dropped frames are not discussed.
  • Hyper-parameters and efficiency are missing: the number of initialization frames, the maximum edge count of the active window, \(\lambda\), \(w_f\), \(w_d\), and the smoothness weights are all absent from the body text, and the paper reports peak memory but neither runtime nor frame rate — for a test-time optimization framework that runs two optimization stages per sequence, practical usability cannot be judged from the current numbers.
  • Scene scale is limited: the real evaluation covers only 2–3 Azure Kinect cameras in a laboratory, 150–300 frames at 30 FPS, with a robot dog, a 6-DOF manipulator, and a walking human as the dynamic content; large outdoor scenes, fast motion, and long stretches with no overlap are untested, and precisely on the non-overlapping scene the advantage narrows noticeably.
  • The framework is sensitive to failures of its external models: a wrong global scale from VGGT, a systematically biased UniDepth prediction, or UFM failing at extreme baselines propagates straight to the final result, and there is no cross-camera loop closure or relocalization to correct accumulated drift.
  • Improvement directions: (1) add cross-camera loop closure/relocalization to constrain long-sequence drift; (2) combine the learnable confidence with semantic or motion cues into a hybrid "soft mask plus prior guidance" weighting; (3) let the alternating pose/depth optimization adapt its iteration count and step size instead of using a fixed schedule; (4) report runtime and convergence analysis to demonstrate usability in a real capture pipeline.
  • vs MegaSAM: the correspondence estimation and sliding-window optimization here are built on MegaSAM, but MegaSAM links only temporal frames inside one camera and never exploits cross-camera consistency; this paper extends it into a three-edge-type connection graph and uses VGGT for multi-camera initialization. The cost is more external models; the payoff is cross-camera scale consistency and robust tracking in dynamic scenes.
  • vs VGGT / Fast3R / FastVGGT: these feed-forward models produce multi-view poses and geometry in a single forward pass, skipping optimization entirely, but memory explodes with frame count (VGGT OOMs on an A100-40GB and must be subsampled to interval=8) and long sequences degrade visibly. This paper calls them only on a short initialization sequence and gives the full frames to optimization, yielding better poses (MultiCamVideo ATE 0.005 vs VGGT† 0.027) at lower memory.
  • vs CUT3R: CUT3R uses a stateful recurrent model to cut memory and support video input, but it recurses frame by frame without explicit global consistency optimization, giving non-smooth trajectories and poor scene consistency (Md 0.581 vs 0.092 here), especially with dynamic objects. Saving memory and achieving scene consistency are hard to have both without global optimization.
  • vs COLMAP / GLOMAP: classical SfM plus BA is accurate in static scenes (GLOMAP reaches ATE 0.006 on RoboArm, on par with 0.005 here), but dynamic objects make COLMAP produce noisy poses and make GLOMAP fail on two of four scenes, and both output only sparse point clouds with no dense depth. The dense optical flow plus confidence weighting here exists to stay stable in dynamic scenes.
  • vs fixed multi-camera array methods (e.g. R3D3-style multi-camera dynamic reconstruction): they require rigidly mounted cameras with pre-calibrated extrinsics, so multi-view constraints are geometrically fixed relationships; this paper allows freely moving cameras with unknown extrinsics and delegates that extra degree of freedom to the spatio-temporal graph and feed-forward initialization — broader applicability at the cost of a harder optimization.

Rating

  • Novelty: ⭐⭐⭐⭐ First framework to unify dense reconstruction and pose estimation for "multiple freely moving cameras plus a dynamic scene"; the spatio-temporal connection graph and wide-baseline initialization are genuine mechanism designs, though VGGT/UniDepth/MegaSAM/UFM are off-the-shelf, making this "key modifications plus a clever composition".
  • Experimental Thoroughness: ⭐⭐⭐⭐ Synthetic and self-collected real datasets, three metric families (pose, depth, scene consistency), both classical SfM and feed-forward baselines, three ablations plus a noise-robustness study, and memory reporting; missing runtime and hyper-parameter details, and the real scenes are of one narrow scale.
  • Writing Quality: ⭐⭐⭐⭐ Clear problem definition, with the three difficulties (scale ambiguity / limited overlap / dynamic content) mapping one-to-one onto the subsequent designs, and ablation conclusions consistent with the claims; some of the results discussion is slightly convoluted.
  • Value: ⭐⭐⭐⭐ Gives the nascent multi-camera dynamic reconstruction direction a reusable "feed-forward initialization plus multi-camera optimization-based BA" recipe, open-sources the motion-capture-grounded MultiCamRobolab dataset, and measures the right dimensions (pose, depth, scene consistency, memory).