Skip to content

GRF-Recon: Global Ray-Field Optimization for Long-Sequence Feed-forward Reconstruction

Conference: ECCV 2026
Paper: ECCV Official
Full Text Cache: ../paper_cache/ECCV2026/eccv-3776.txt
Area: 3D Vision
Keywords: Long-sequence Reconstruction / Ray-field Optimization / Feed-forward Mapping / Global Consistency / Camera Tracking

TL;DR

GRF-Recon introduces re-normalized LoRA geometric prior injection and a hybrid-weight sparse ray-field factor graph optimization to enable low-drift, globally consistent kilometer-scale feed-forward 3D reconstruction from uncalibrated monocular videos within a strict 24 GB GPU budget.

Background & Motivation

Recovering precise camera poses and dense 3D scene representations from uncalibrated monocular RGB video streams represents a fundamental yet elusive goal in 3D vision and autonomous driving. Because monocular vision intrinsically cannot observe absolute metric scale, accumulated drift over extended trajectories is particularly devastating: small local estimation imperfections compound into massive global scale distortions, non-rigid warping, and trajectory divergence. Traditional visual SLAM and Structure-from-Motion (SfM) pipelines (e.g., ORB-SLAM3, COLMAP, and DROID-SLAM) depend heavily on handcrafted or learned local feature extraction, cross-view feature association, and iterative backend bundle adjustment. These methods remain brittle in low-texture or high-glare environments and accumulate irreversible scale drift over kilometer-scale trajectories.

Recently, Transformer-based feed-forward foundation models such as DUSt3R, MASt3R, VGGT, and Depth Anything 3 (DA3) have introduced a paradigm shift by directly predicting 3D point maps and camera poses from raw images. However, their attention mechanisms exhibit quadratic memory growth with sequence length, triggering out-of-memory (OOM) failures when scaling beyond short sequences of several dozen frames. Existing chunk-and-align strategies (such as VGGT-Long and DA3-Streaming) attempt to bypass memory bottlenecks by chunking sequences and stitching them using rigid similarity transformations over overlapping keyframes. Yet, because intra-chunk poses rely exclusively on unconstrained feed-forward predictions, these methods lack explicit cross-frame geometric coupling, leading to severe pose discontinuities, scale drift, and point-cloud ghosting at intersections. Furthermore, foundation models fine-tuned purely on general datasets often suffer from geometric over-smoothing on slender structures and high-frequency boundaries.

The core tension lies in the trade-off between the expressive generalization of feed-forward foundation models and the strict physical consistency enforced by multi-view geometry. Core idea: construct an end-to-end framework that couples re-normalized LoRA geometric distillation with a hybrid-weight sparse ray-field factor graph optimization, leveraging dual-confidence ray sampling and joint loop-closure constraints to achieve low-drift, globally consistent feed-forward reconstruction across kilometer-scale uncalibrated sequences.

Method

Overall Architecture

The GRF-Recon framework uses Depth Anything 3 (DA3) as its perceptual backbone and is structured into four sequential stages: 1. Local Depth Refinement via Re-normalized LoRA: Freezing the multi-view matching decoder while inserting low-rank adapters into the DINOv2 encoder attention layers, distilling monocular geometric priors from an advanced estimator to restore sharp depth discontinuities and thin structures; 2. Chunk-based Initial Trajectory Alignment: Partitioning long videos into temporal chunks and aligning overlapping frames via decoupled \(\text{Sim}(3)\) estimation (SVD for rotation and closed-form least-squares for scale and translation) to establish a coarse global initial trajectory; 3. Hybrid-Weight Sparse Ray-Field Construction & Matching: Predicting 6D ray parameters alongside depth and ray confidences, adaptively sampling sparse rays through joint confidence and depth gradient metrics, and establishing cross-frame ray correspondences via perpendicular ray distances, gradient similarity, and photometric errors verified by bidirectional symmetry; 4. Topological Loop Closure & Global Factor Graph Optimization: Detecting long-range loop candidates via global visual descriptor cosine similarity and solving a sparse factor graph over camera poses using the Huber robust loss and Levenberg-Marquardt algorithm.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Uncalibrated Monocular RGB Video<br/>Long-sequence input"] --> B["Re-normalized LoRA Geometric Distillation<br/>Freeze decoder + adapt DINOv2 encoder"]
    B --> C["Chunk-based Coarse Alignment<br/>Solve Sim(3) on overlapping frames"]
    C --> D["Hybrid-Weight Sparse Ray-Field Construction<br/>Dual-confidence sampling + multi-metric matching"]
    D --> E["Loop Detection & Global Ray-Field Graph Optimization<br/>Cosine similarity retrieval + LM solver"]
    E --> F["Globally Consistent Dense Point Cloud<br/>& Closed-loop Camera Trajectory"]

Key Designs

1. Re-normalized LoRA Geometric Distillation: Restoring Fine Structures Without Degrading Matching Generalization

Large-scale feed-forward networks often suffer from edge over-smoothing and blurred boundaries around thin objects such as utility poles, foliage, and vehicle silhouettes. Direct full fine-tuning on domain-specific datasets induces catastrophic forgetting of the broad cross-view correspondence capabilities acquired during pretraining. To circumvent this, the cross-view matching decoder is kept frozen, and low-rank adaptation (\(r=8\)) is inserted into the Query-Key-Value projection matrices of the DINOv2 encoder. Because naive distillation introduces feature norm drift that upsets the frozen decoder's expected input distribution, the effective weights \(W'\) are constrained using Frobenius norm re-normalization: $\(W' = \frac{W_0 + BA}{\|BA\|_F} \|W_0\|_F\)$ This adaptation adds less than 1% trainable parameters while preserving the baseline activation scale. Supervision utilizes a teacher-student dual-branch distillation objective \(\mathcal{L} = \mathcal{L}_{\text{depth}} + \lambda \mathcal{L}_{\text{point}}\) (\(\lambda=0.4\)): the depth branch employs pseudo-labels from an advanced monocular estimator (MoGe) under a log-uncertainty-weighted \(\ell_2\) loss, while the point branch enforces ground-truth LiDAR 3D regression, restoring high-frequency local boundaries without overfitting.

2. Hybrid-Weight Sparse Ray-Field Construction: Establishing Explicit Cross-View Geometric Coupling

Monocular tracking cannot directly measure 3D spatial distances, and traditional homography or 2D optical flow constraints struggle to resolve scale drift along the camera optical axis. GRF-Recon leverages the DA3 ray decoding head to output a 7-channel ray vector per pixel: the first 6 channels define the 3D ray \(r_i = [d_i^\top, o_i^\top]^\top \in \mathbb{R}^6\) (\(d_i\) as unit direction and \(o_i\) as origin), while the 7th provides the prediction confidence \(c_i^{\text{ray}}\). Local 3D points are reconstructed via \(P_i = o_i + z_i d_i\). To restrict matching complexity, sparse rays are sampled using a composite score combining dual confidence fusion and depth gradient magnitude: $\(c_i^{\text{joint}} = \beta \cdot \hat{c}_i^{\text{depth}} + (1 - \beta) \cdot \hat{c}_i^{\text{ray}}\)$ $\(s_i = \gamma_{\text{conf}} \cdot c_i^{\text{joint}} + \gamma_{\text{grad}} \cdot \|\nabla D(i)\|_2\)$ with \(\beta=0.5, \gamma_{\text{conf}}=0.8\), and \(\gamma_{\text{grad}}=0.2\), followed by Non-Maximum Suppression (NMS). Candidate rays in neighboring frames are queried around an initial perspective projection anchor, and scored via a multi-dimensional matching metric: $\(\mathcal{S}(p_i, p_j) = w_r e_{\text{ray}} + w_g e_{\text{grad}} + w_p e_{\text{photo}}\)$ where \(e_{\text{ray}}\) measures the perpendicular Euclidean distance from target ray \(r_j\) to source 3D point \(P_i\), \(e_{\text{grad}}\) represents normalized gradient cosine dissimilarity, and \(e_{\text{photo}}\) denotes patch-level photometric SAD (\(w_r=0.6, w_g=0.2, w_p=0.2\)). Bidirectional symmetry verification (\(\tau_{\text{sym}} < 5.0\) pixels) discards spurious associations, enforcing strong physical cross-view rigidity under uncalibrated settings.

3. Loop Detection & Global Factor Graph Optimization: Explicitly Mitigating Accumulated Drift

Pure forward chunk concatenation or streaming estimators inevitably drift over multi-kilometer trajectories. The system dynamically selects keyframes using DINOv2 global feature cosine similarity: a new keyframe is registered whenever visual similarity drops below \(\tau_{\text{kf}}=0.70\), naturally adapting sampling density to rotational vs. translational motion (averaging 3–5 frames between keyframes on KITTI). When similarity with a past keyframe exceeds \(\tau_{\text{sim}}=0.85\) across a temporal window \(\Delta t > 100\) frames, a loop closure candidate is triggered, and high-confidence relative pose constraints \(\hat{T}_{ij}\) are extracted. All camera poses \(\{T_k \in \text{SE}(3)\}\) are then jointly optimized within a unified factor graph: $\(E(\{T_k\}) = \sum_{i,j,p} \rho\left(e_{\text{ray}}(T_j^{-1} T_i P_i, r_j)\right) + \lambda_{\text{pose}} \sum_{i,j} \rho_{\text{pose}}(T_i, \hat{T}_{ij})\)$ where \(\rho(\cdot)\) denotes the robust Huber loss and \(\lambda_{\text{pose}}=10.0\) balances pose-graph priors against point-to-ray Euclidean residuals. Benefiting from accurate metric initialization from coarse chunk alignment and the extreme sparsity of the factor graph, Levenberg-Marquardt optimization converges in approximately 10 seconds for sequences of several thousand frames, effectively closing trajectory loops and eliminating long-range warping.

Loss & Training

The perception front-end is fine-tuned on a single NVIDIA RTX 3090 GPU (24 GB) with automatic mixed precision (AMP), using a batch size of 16 across 15 epochs (\(\approx 14\) hours). The total parameter increase remains strictly under 3%. The depth supervision objective follows: $\(\mathcal{L}_{\text{depth}} = \frac{1}{|\Omega|} \sum_{i \in \Omega} \left( \frac{\|z_i - z_i^{\text{gt}}\|_2^2}{2 \sigma_i^2} + \frac{1}{2} \log \sigma_i^2 \right)\)$ During test-time inference, images are scaled to a width of 504 pixels. The backend runs a sparse C++ factor graph solver using a chunk size of 60 frames with 10 frames of overlap.

Key Experimental Results

Main Results

GRF-Recon was comprehensively benchmarked on the KITTI Odometry dataset (11 challenging sequences spanning up to 5067 meters) and the Waymo Open Dataset. All evaluated methods operated under a strictly monocular setup without prior intrinsic calibration or depth supervision. Trajectories were aligned to LiDAR ground truth via \(\text{Sim}(3)\) prior to computing Absolute Trajectory Error (ATE RMSE [m] ↓).

Table 2 in the paper reports camera tracking accuracy on KITTI (Average excludes the high-speed highway Sequence 01):

Method Intrinsic Calibration Avg. ATE (m) ↓ Seq. 00 (3724m) Seq. 05 (2206m) Seq. 07 (650m) Seq. 08 (3223m) Seq. 09 (1705m)
ORB-SLAM3 Calibrated 7.93 5.12 4.60 1.55 25.30 8.50
DROID-SLAM Calibrated 74.88 95.10 128.50 15.58 151.60 67.33
DPV-SLAM++ Calibrated 24.25 7.67 5.34 2.01 124.46 63.64
VGGT-Long Uncalibrated 19.78 11.16 9.35 5.23 56.15 42.24
Pi-Long Uncalibrated 19.01 10.82 6.13 3.24 36.84 20.27
DA3-Streaming Uncalibrated 12.72 9.34 5.87 4.83 30.82 10.93
GRF-Recon (Ours) Uncalibrated 7.18 4.35 3.25 1.64 26.40 7.25

Noticeably, vanilla VGGT and DA3 suffered Out-of-Memory (OOM) failures on all extended sequences, while MASt3R-SLAM encountered Tracking Lost (TL) across every run due to abrupt viewpoint changes. GRF-Recon not only outperformed all uncalibrated feed-forward competitors by a wide margin (7.18m vs. 12.72m for DA3-Streaming) but also surpassed the calibrated gold-standard ORB-SLAM3 (7.93m).

On the monocular depth estimation subtask (Table 1), fine-tuning DA3 with Re-normalized LoRA reduced relative error (Rel) on KITTI from 5.14 to 4.50 and raised threshold accuracy \(\delta_1\) from 92.4% to 97.1%, closely approaching the teacher model MoGe (Rel 4.39, \(\delta_1\) 97.2%).

Ablation Study

Table 6 in the paper dissects the performance impact of each individual module across KITTI and Waymo sequences (ATE RMSE [m] ↓):

Configuration Description KITTI 00 (4542 frames) KITTI 07 (1101 frames) Waymo 346 (199 frames) Waymo 371 (199 frames)
Baseline (Full Model) Default (C/O = 60/10) 4.35 1.64 2.52 2.15
w/o LoRA Remove LoRA encoder adaptation 6.80 (+2.45) 2.65 (+1.01) 3.10 (+0.58) 2.80 (+0.65)
w/o Sparse KF Disable adaptive keyframing 5.45 (+1.10) 2.15 (+0.51) 3.15 (+0.63) 2.74 (+0.59)
Ray Comp. (0,0,0) Completely remove ray matching 7.35 (+3.00) 4.85 (+3.21) 6.60 (+4.08) 4.25 (+2.10)
w/o Loop Remove loop detection & priors 8.12 (+3.77) 5.85 (+4.21) 6.85 (+4.33) 6.10 (+3.95)
w/o Opt. Disable backend optimization 12.35 (+8.00) 6.65 (+5.01) 7.40 (+4.88) 8.20 (+6.05)

Key Findings

  • Ray-field optimization is critical for eliminating drift: Discarding ray matching caused tracking errors on KITTI 07 to leap from 1.64m to 4.85m; disabling the entire backend optimization caused KITTI 00 error to explode to 12.35m, proving that raw feed-forward pose concatenation cannot sustain scale stability.
  • Distilled geometric priors anchor ray correspondence: Re-normalized LoRA provides crisp structural boundaries; omitting it degraded tracking accuracy by 56% on KITTI 00 (jumping to 6.80m) due to correspondence ambiguities on smoothed surfaces.
  • Asymmetric compute efficiency: The multi-threaded pipeline effectively conceals chunk alignment (0.15s/chunk) and ray matching (0.16s/keyframe) behind feed-forward inference (\(\approx 8\)s/chunk). The sparse backend optimizer converged in only 10.23 seconds on the 4,542-frame KITTI 00 sequence, enabling full reconstruction of 3.7 km in under 12 minutes.

Highlights & Insights

  • Re-normalized low-rank adaptation: Constraining weight updates via Frobenius norm preservation prevents feature activation distribution shifts between the adapted encoder and the frozen cross-view decoder, achieving sub-1% parameter overhead with substantial boundary sharpening.
  • Ray-distance metric over pixel re-projection: Directly computing Euclidean perpendicular distances between 6D spatial rays and 3D points replaces traditional 2D optical flow or homography residuals, translating uncalibrated monocular depth ambiguities into rigid 3D spatial penalties.
  • Plug-and-play 3D Gaussian Splatting initialization: Dense point clouds reconstructed by GRF-Recon provide a robust, geometry-aware geometric initialization for 3DGS, completely replacing costly offline COLMAP SfM pipelines and converging within 1,000 iterations for novel view synthesis on UAV datasets.

Limitations & Future Work

  • Vulnerability to dynamic foregrounds: The framework assumes a static rigid world; dynamic vehicles or pedestrians in congested traffic introduce erroneous ray constraints that require semantic masking or robust dynamic outlier rejection.
  • Drift in loop-free open-ended exploration: In pure unidirectional trajectory sequences lacking loop revisitation, the factor graph relies solely on inter-chunk and adjacent keyframe ray links, leaving residual unobservable scale drift over tens of kilometers.
  • Real-time deployment barriers: Although pipeline parallelism conceals matching overhead, per-frame ViT inference latency remains too high for real-time onboard robotic platforms (target: \(\ge 30\) FPS).
  • vs. VGGT-Long / DA3-Streaming: Prior long-sequence extensions operate purely on chunk-and-align pose stitching without intra-chunk or cross-chunk geometric re-optimization; GRF-Recon incorporates sparse ray-field constraints to tightly couple 3D geometry across chunk boundaries.
  • vs. DROID-SLAM / ORB-SLAM3: Traditional calibrated SLAM degrades in low-texture scenes or aggressive rotational maneuvers; GRF-Recon operates completely calibration-free while outperforming ORB-SLAM3 in trajectory accuracy on long sequences.
  • vs. MASt3R-SLAM: MASt3R-SLAM suffers from catastrophic tracking loss when frame baselines expand rapidly; GRF-Recon leverages adaptive visual keyframing and chunk-level fallback to preserve continuous kilometer-scale tracking.

Rating

  • Novelty: ⭐⭐⭐⭐☆ [A thoughtful synthesis of re-normalized LoRA distillation and sparse ray-field factor graph optimization for scalable monocular mapping.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive benchmarking across long-sequence KITTI and Waymo benchmarks, accompanied by rigorous depth, tracking, runtime, and ablation evaluations.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, concise mathematical formulation, and informative pipeline diagrams.]
  • Value: ⭐⭐⭐⭐⭐ [Solves a pressing bottleneck in feed-forward 3D foundation models, providing an efficient bridge toward real-world large-scale uncalibrated autonomous navigation and 3DGS mapping.]