VideoSfM: Exploiting Temporal Structure for Video-Based Structure-from-Motion

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/cvg/vidmap
Area: 3D Vision
Keywords: Structure-from-Motion, Visual SLAM, Video Temporal Structure, Dense Feature Matching, Monocular Metric Depth Prior

TL;DR

VidMap (VideoSfM) combines the sequential tracking constraints of SLAM with the non-causal global optimization of offline SfM, achieving drift-free metric 3D reconstruction on long, unconstrained, and uncalibrated videos through provenance-aware data association and metric depth regularization.

Background & Motivation

Recovering accurate metric camera poses and calibration parameters from arbitrary, unconstrained video streams captured by handheld phones or robotic platforms is a foundational building block for embodied navigation and large-scale 3D scene representation. However, existing paradigms suffer from acute architectural trade-offs: visual SLAM systems operate in a causal and strictly incremental manner, committing irreversibly to pose estimates using only historical frames. Consequently, SLAM is brittle to initialization errors, prone to catastrophic tracking loss under degenerate or rapid motions, and typically demands pre-calibrated camera intrinsics while compounding drift irrevocably over time.

In contrast, offline Structure-from-Motion (SfM) defers geometric decisions until the full observation view-graph is established, enabling optimal initialization, joint camera self-calibration, and global bundle adjustment. Nonetheless, classical SfM treats continuous video sequences as an unordered set of independent images. By ignoring temporal continuity, it discards the strong local tracking guarantees of adjacent frames and becomes profoundly vulnerable to visual aliasing caused by repetitive patterns and environmental symmetries; moreover, its inability to adapt to motion dynamics leads to severe computational inefficiency when processing raw video. While feedforward deep models offer robustness, their limited context windows incur accumulated chunk-alignment drift, lacking geometric precision on long horizons.

The core tension lies in exploiting temporal continuity for reliable, dense correspondence tracking without succumbing to the irreversible drift of causal SLAM, while retaining global loop closure and joint optimization without letting spurious non-sequential matches corrupt the trajectory. VidMap addresses this challenge by treating temporal ordering as a first-class citizen inside an offline global SfM formulation, augmented by metric monocular depth priors to resolve gauge and scale ambiguity. Core idea: VidMap explicitly separates sequential tracking edges from loop-closure edges using provenance-aware data association, suppresses long-track drift via multi-flow covariance propagation, and couples global positioning and bundle adjustment with per-image depth scale optimization.

Method

Overall Architecture

The VidMap architecture is structured into two principal non-causal phases: Video-Aware Extraction and Provenance-Aware Global Mapping. The system takes as input an unconstrained continuous video sequence with unknown camera intrinsics, and outputs metric keyframe camera poses \(T_i \in \mathrm{SE}(3)\), calibrated intrinsics \(K_i\), and a sparse 3D point cloud \(X\).

During the extraction phase, the system first detects motion-adaptive keyframes via low-resolution dense matching and extracts monocular metric depth priors alongside focal length candidates. It then establishes long feature tracks across keyframes using dense warp chaining, dynamically correcting multi-step drift using a minimal-covariance path selection strategy. Meanwhile, global image retrieval extracts loop-closure candidates, which are linked to sequential tracks as separate soft observations tagged with their origin provenance. In the global mapping phase, VidMap computes shared focal lengths, performs provenance-aware rotation averaging with distinct robust losses, solves global positioning (GP) augmented with monocular depth and per-image scale factors, and finally refines all parameters jointly via robust bundle adjustment (BA).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Unconstrained Video Sequence"] --> B["Motion-Adaptive Keyframe Detection<br/>Thresholding coarse dense warp displacement"]
    B --> C["Monocular Metric Depth & Focal Prior Extraction<br/>GeoCalib focal estimation + metric depth maps"]
    C --> D["Dense Track Chaining & Multi-Flow Drift Correction<br/>Dense warp propagation with minimal covariance paths"]
    D --> E["Provenance-Separated Loop Closure & Track Setup<br/>MegaLoc candidate retrieval + soft-linked LC observations"]
    E --> F["Provenance-Aware Rotation Averaging<br/>Huber loss for sequential edges + Cauchy loss for LC edges"]
    F --> G["Global Positioning with Metric Depth Priors<br/>Joint optimization of centers, 3D points & scale factors"]
    G --> H["Joint Global Bundle Adjustment<br/>Full refinement of poses, points, intrinsics & scales"]
    H --> I["Output Metric Camera Trajectory & 3D Point Cloud"]

Key Designs

1. Motion-Adaptive Keyframe Detection: Bypassing Fixed-Interval Sampling

To handle arbitrary video dynamics ranging from stationary pauses to fast maneuvers, VidMap dispenses with rigid temporal subsampling in favor of motion-triggered keyframing. Using a coarse-stage dense matcher (e.g., RoMA coarse stage), the pipeline computes dense warp fields and certainty maps between consecutive frames. Sparse keypoints sampled on the current keyframe are projected along the warp field. A new keyframe is instantiated as soon as the proportion of keypoints that are either lost (due to occlusion or out-of-frame motion) or displaced beyond a geometric threshold exceeds a specified ratio. This mechanism ensures uniform spatial baseline distribution while eliminating degenerate near-zero motion pairs.

2. Multi-Flow Drift Correction: Minimal-Covariance Temporal Chaining

While dense matchers provide sub-pixel accuracy even in low-texture regions, consecutive pairwise chaining inevitably accumulates spatial drift. For every newly identified keyframe \(K_i\), VidMap computes dense correspondences not only to its immediate predecessor \(K_{i-1}\), but across a sliding temporal window of up to \(W=9\) preceding keyframes. Each dense match provides an estimated 2D pixel displacement alongside a localization covariance \(\Sigma_{(i-n)\to i} \in \mathbb{R}^{2\times 2}\). The cumulative covariance along the sequential chain is \(\Sigma_i^{\text{seq}} = \Sigma_{i-1}^{\text{seq}} + \Sigma_{(i-1)\to i}\), whereas a direct link from any preceding anchor \(K_{i-n}\) has total covariance \(\Sigma_{(i-n)i} = \Sigma_{i-n}^{\text{seq}} + \Sigma_{(i-n)\to i}\). VidMap identifies the anchor with the minimum trace \(n^* = \arg\min_n \mathrm{tr}(\Sigma_{(i-n)i})\) and accepts the direct hop only if its Mahalanobis distance to the sequential estimate under \(\Sigma_{(i-n^*)i} + \Sigma_i^{\text{seq}}\) falls below a statistical threshold; otherwise, the sequential estimate is maintained. This reliably prunes erroneous long-range hops while curbing drift over extended sequences.

3. Provenance-Separated Track Establishment: Guarding Against Visual Aliasing

Standard global SfM pipelines merge correspondences across all image pairs via transitive closure, producing tracks that collapse when spurious loop-closure matches connect visually symmetric or repetitive scene elements. VidMap strictly restricts unified track construction to sequential tracking edges. When loop-closure (LC) matches are identified via MegaLoc retrieval and verified by two-view geometry, they are never merged transitively; rather, each LC observation is linked softly to the sequential track and tagged with provenance \(l_{ik} = \text{lc}\), whereas sequential observations carry \(l_{ik} = \text{seq}\). In the subsequent rotation averaging step, sequential edges are assigned a Huber loss to preserve continuous trajectory integrity, while LC edges receive a Cauchy loss that heavily downweights outliers: $$ \min_{{R_i}} \sum_{(i, j) \in \mathcal{E}} \rho_{l_{ij}}\left( |\log(R_j^\top \tilde{R}_{ij} R_i)| \right) $$ This separation ensures that visual symmetries do not distort the underlying global camera orientations.

4. Global Positioning with Monocular Depth & Per-Image Scale Optimization: Taming Scale Drift and Degeneracy

Pure multi-view bearing constraints become ill-conditioned under forward camera motions, low parallax, or texture-free surfaces. VidMap incorporates monocular depth maps \(m_{ik}\) and their uncertainties \(\sigma_{ik}\) into global positioning (GP) and bundle adjustment (BA), introducing an explicit, optimizable scale factor \(s_i\) for each keyframe. In the GP stage, holding orientations \(\{R_i\}\) fixed, camera centers \(\{c_i\}\), 3D point positions \(\{X_k\}\), and scale parameters \(\{s_i\}\) are solved jointly: $$ E_{\text{GP}} = \sum_{(i,k)} \left[ \rho_{l_{ik}}^{\text{bearing}}\left( e_{ik}^{\text{GP}} \right) + \rho^{\text{depth}}\left( r_{ik}^{\text{depth}} \right) \right] + \sum_{i} \rho^{\text{scale}}\left( r_i^{\text{scale}} \right) $$ The bearing residual \(e_{ik}^{\text{GP}} = v_{ik} - d_{ik}(X_k - c_i)\) is weighted by the 3D bearing covariance \(\Sigma_{v_{ik}} = J_i \Sigma_{ik} J_i^\top\) derived from pixel matching covariance. The depth residual enforces agreement with the scaled monocular prediction: $$ r_{ik}^{\text{depth}} = \begin{cases} \log\left(\dfrac{z_{ik}}{s_i \cdot m_{ik}}\right), & z_{ik} > 0 \[6pt] z_{ik} - s_i \cdot m_{ik}, & z_{ik} \le 0 \end{cases} $$ where \(z_{ik} = \mathbf{e}_3^\top R_i (X_k - c_i)\), and \(r_i^{\text{scale}} = \log s_i\) regularizes the scale factor. In the final global BA, poses, 3D structure, intrinsics, and depth scales are co-refined, preventing metric scale drift across hundreds of meters.

Loss & Training

VidMap operates as an offline optimization framework that builds upon pre-trained vision models (RoMA v2, Depth Anything 3, GeoCalib, and MegaLoc) without end-to-end retraining. During iterative non-linear least squares, the pipeline implements an annealing strategy inspired by graduated non-convexity (GNC): Cauchy loss scale parameters are progressively tightened as iterations advance in both GP and BA. Furthermore, pairwise depth-geometry consistency checks verify monocular predictions against triangulated geometry, switching any inconsistent depth observations to heavy Cauchy penalties to protect global convergence.

Key Experimental Results

Main Results

Evaluations were conducted on the challenging indoor AR benchmark LaMAR (featuring long, non-expert handheld sequences with symmetries and low-texture corridors) and the CroCoDL dataset (capturing phone and robot traversals in unstructured disaster buildings). Translation accuracy is quantified via Windowed AUC (W-AUC, %, with error threshold adapted to 5% of window size; higher is better).

Table 1: Pose estimation accuracy on the LaMAR dataset (Window-AUC, %)

Method	Category	Calibration	10 m	25 m	50 m	100 m	full
LoGeR	Deep Model	Uncalibrated	65.9	64.5	60.3	60.1	59.8
VGGT-SLAM2	Deep SLAM	Uncalibrated	74.0	69.2	60.3	49.4	56.9
Lingbot-Map	Deep Model	Uncalibrated	74.2	75.0	70.1	64.7	63.0
DA3-Long	Deep Model	Uncalibrated	86.7	86.7	84.6	78.1	76.5
GLOMAP-LG	Global SfM	Uncalibrated	76.4	69.4	60.1	53.6	62.3
GLOMAP-RoMA	Global SfM	Uncalibrated	71.0	66.0	58.4	46.7	59.4
ViPE	Deep SLAM	Uncalibrated	86.6	83.8	80.2	78.1	76.6
VidMap (Ours)	Hybrid Global SfM	Uncalibrated	91.9	92.3	89.9	89.3	88.5
DPV-SLAM	Learned SLAM	Calibrated	53.1	39.0	25.1	14.9	38.2
DROID-W	Learned SLAM	Calibrated	88.0	84.7	80.2	79.5	76.2
MP-SfM	Incr. SfM	Calibrated	88.6	83.7	74.5	60.2	60.6
GLOMAP-LG	Global SfM	Calibrated	77.3	70.4	61.6	48.4	63.0
ViPE	Deep SLAM	Calibrated	87.3	84.9	80.4	79.5	78.8
VidMap (Ours)	Hybrid Global SfM	Calibrated	93.2	93.0	90.5	89.9	89.7

Table 2: Pose estimation accuracy on the CroCoDL dataset (Window-AUC, %)

Method	Calibration	Phone 10m	Phone 50m	Phone full	Robot 10m	Robot 50m	Robot full
DA3-Long	Uncalibrated	86.2	88.5	88.9	79.5	82.0	78.3
GLOMAP-LG	Uncalibrated	77.8	60.0	76.4	10.2	10.3	19.2
ViPE	Uncalibrated	66.7	68.6	71.4	59.5	66.2	71.0
VidMap (Ours)	Uncalibrated	94.0	95.3	95.5	91.4	82.5	80.3
DPV-SLAM	Calibrated	19.8	14.9	31.8	66.1	73.0	77.0
DROID-W	Calibrated	71.8	67.9	72.7	71.9	71.0	67.8
GLOMAP-LG	Calibrated	73.9	62.0	72.2	42.7	34.1	42.4
ViPE	Calibrated	70.0	71.0	74.3	62.1	68.9	72.2
VidMap (Ours)	Calibrated	94.7	96.3	96.6	91.5	80.6	78.2

Ablation Study

The ablation benchmarks on LaMAR and ETH3D-SLAM dissect the specific contributions of depth priors and temporal provenance modeling.

Table 3: Ablation study on LaMAR and ETH3D-SLAM datasets

Configuration	LaMAR 10m	LaMAR 50m	LaMAR 100m	LaMAR full	ETH3D 5cm	ETH3D 10cm	ETH3D 1m
VidMap (Full Model)	91.9	89.9	89.3	88.5	51.4	68.3	94.0
Depth Priors Ablation
No depth in GP	67.6	24.9	13.6	37.7	50.5	66.5	91.9
No depth in BA	91.5	88.0	86.5	85.5	44.5	60.6	91.9
No scale optimization	69.3	25.6	13.9	39.7	50.2	65.9	92.4
No metric scale	90.2	81.9	71.7	74.1	52.2	68.8	94.0
Temporal Structure Ablation
No loop closure edges	90.8	87.6	86.6	85.5	32.9	77.5	85.9
No provenance losses	88.4	80.2	77.0	79.9	50.6	87.1	91.5

Key Findings

• Monocular depth in GP is essential for long-range stability: Eliminating depth priors during global positioning (No depth in GP) causes the 100m W-AUC on LaMAR to plummet from 89.3% down to 13.6% (full sequence drops from 88.5% to 37.7%). In long forward motions with minimal baseline, bearing-only optimization encounters geometric singularity; depth priors prevent trajectory collapse.
• Provenance-aware loss isolates visual symmetry corruption: When treating loop-closure edges with the same loss as sequential edges (No provenance losses), the 100m W-AUC degrades by 12.3% (dropping from 89.3% to 77.0%) on LaMAR, proving that downweighting loop-closure candidates with Cauchy robust kernels prevents false positive associations from tearing the map apart.
• Self-calibration matches ground truth performance: On LaMAR, uncalibrated VidMap attains an 88.5% full-sequence W-AUC, virtually matching the 89.7% scored under ground-truth intrinsics, validating the robustness of joint GeoCalib and view-graph self-calibration.

Highlights & Insights

• Provenance-Aware Data Association: Rather than collapsing image correspondences into monolithic tracks via transitive closure (which allows a single false positive match in a repetitive hallway to ruin global geometry), VidMap keeps sequential tracks as structural backbones and treats loop closures as soft, provenance-tagged links, safeguarding trajectory stability.
• Soft Monocular Regularization with Per-Image Scales: Rather than rigidly locking monocular depth or assuming perfect cross-frame scale consistency, VidMap co-optimizes per-image depth scale parameters \(s_i\) alongside camera poses and 3D points in both GP and BA, marrying metric priors with multi-view constraints gracefully.
• Acausal Exploitation of Video Sequences: VidMap demonstrates that video geometry estimation need not be trapped in causal, frame-by-frame filtering; running multi-pass, non-causal global optimization over temporally chained tracks delivers unprecedented accuracy and zero-shot generalization across robotic and handheld platforms.

Limitations & Future Work

• Residual Drift in Ultra-Long Trajectories: While VidMap sets a new state of the art for vision-only reconstruction, its absolute metric accuracy on multi-kilometer sequences does not yet match visual-inertial systems (VIO/VINS) backed by hardware IMU measurements.
• Static Scene Assumption: The current pipeline assumes predominantly static scenes. High proportions of dynamic moving obstacles (such as walking crowds or passing vehicles) can degrade dense optical flow chaining and distort monocular depth maps.
• Computational Footprint: Multi-view dense matching and non-linear least squares optimization over thousands of keyframes remain compute-intensive, suggesting a need for hierarchical submapping to enable real-time cloud-streaming pipelines.

Related Work & Insights

• vs GLOMAP: GLOMAP is the leading modern global SfM framework for unordered image collections. VidMap builds on its global formulation but resolves its Achilles' heel on video: replacing transitive track establishment with provenance-separated tracking and introducing monocular depth priors to eliminate scale drift and visual aliasing.
• vs DROID-SLAM / DPV-SLAM: While learned causal SLAM algorithms optimize dense flow fields across local temporal windows, early pose drift in narrow-baseline segments becomes locked into the map irreversibly. VidMap defers geometric commitments until the entire sequence is processed, providing vastly superior global consistency.
• vs MP-SfM: MP-SfM incorporates monocular depth into incremental SfM but fixes scale parameters after reconstruction and incurs high computational cost from repeated resectioning. VidMap optimizes scale factors dynamically across global positioning and bundle adjustment, yielding faster and more coherent reconstructions.

Rating

• Novelty: ⭐⭐⭐⭐☆ Elegantly bridges the divide between causal SLAM and global SfM by marrying temporal track provenance with metric depth-scale optimization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across LaMAR, CroCoDL, ETH3D, and EuRoC with over ten baselines, rigorous ablations, and runtime scalability profiling.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate narrative, crystal-clear motivation, and rigorous mathematical formulations.
• Value: ⭐⭐⭐⭐⭐ Provides an open-source, highly robust metric reconstruction baseline for uncalibrated in-the-wild video data, offering profound utility for 3D computer vision and robotics.