Skip to content

RayMap3R: Inference-Time RayMap for Dynamic 3D Reconstruction

Conference: ECCV 2026
Paper: ECCV 2026
Code: https://raymap3r.github.io/
Area: 3D Vision
Keywords: Streaming 3D Reconstruction, Dynamic Scenes, RayMap, Inference-Time Dynamic Identification, Camera Pose Estimation

TL;DR

RayMap3R observes that a streaming reconstruction model trained with RayMap exhibits a static-scene bias when queried with RayMap features alone, and turns that bias into a training-free dynamic detector: the per-pixel depth discrepancy between a main branch and a RayMap-only branch gates memory-state updates, while reset metric alignment and state-aware smoothing keep long trajectories stable, pushing pose, depth, and reconstruction accuracy on dynamic scenes to the front of streaming methods with no training and constant memory.

Background & Motivation

Streaming feed-forward 3D reconstruction moves joint estimation of scene geometry and camera poses from offline batch processing to the online setting. Offline paradigms such as DUSt3R/MASt3R/VGGT first collect all images and then run dense pairwise matching with global attention; accuracy is high, but the cost explodes with sequence length β€” by VGGT-SLAM's accounting, 200 frames already need close to 48 GB, which rules out real-time use. Streaming methods including CUT3R, TTT3R, Point3R, StreamVGGT, and Spann3R replace this with a memory mechanism: an implicit state cache (or explicit point anchors, or a spatial memory bank) interacts with each incoming frame, keeping memory constant and inference fast over thousands of frames. But their memory takes in everything indiscriminately β€” training data carries almost no dynamic annotations, so the model is never taught which content should not be written to memory.

The trouble is that a streaming model only has limited historical context when processing a frame. Once a moving object enters the picture, the reconstructed dynamic geometry is accepted as a trustworthy observation and written into the state; subsequent frames then decode pose and depth from that contaminated state, and the error accumulates frame by frame into camera drift and structural distortion (the trajectory drift of CUT3R and TTT3R in Fig. 1 of the paper comes from exactly this). Existing dynamic reconstruction schemes generally bolt on external modules: flow-based methods depend on an off-the-shelf flow estimator plus hand-tuned thresholds and generalize poorly, segmentation-based methods are limited by their training categories, and tracking-based methods require known camera intrinsics and iterative optimization β€” all of them add overhead and domain dependence that clash with the real-time, constant-memory goal of streaming.

This paper's angle comes from a neglected property: RayMap, a per-pixel camera-ray parameter map that encodes ray origin and unit direction, carries camera geometry only and no appearance information. So when the model is fed RayMap features alone, it cannot "see" objects in the current frame and must retrieve scene content from memory. Static structures are observed repeatedly across history and can be recalled reliably, whereas dynamic objects appear only transiently and are more readily suppressed β€” which gives RayMap-only predictions a systematic bias toward the static background. The authors verify this on 108 sequences from MPI Sintel, DAVIS 2017, and TUM RGB-D: the per-pixel depth discrepancy between the main and RayMap branches correlates positively with the ground-truth dynamic ratio (Spearman ρ=0.77, p<10⁻²²), making it a stable and usable dynamic cue. Core idea: treat the RayMap branch's inability to see dynamic objects as a free prior, use the depth discrepancy between the two branches to compute staticness weights at inference time, and gate memory-state updates with them β€” making streaming reconstruction robust to dynamic scenes without any training or retraining.

Method

Overall Architecture

The method takes a stream of images and outputs camera poses, depth maps, and point clouds. The base is a streaming reconstruction model with implicit memory (the paper's ablation explicitly uses CUT3R as the base), over which RayMap3R adds three inference-time components, all without training or back-propagation: β‘  dual-branch RayMap dynamic identification β€” at each timestep a main branch (image + RayMap features) and a RayMap branch (RayMap features rebuilt from the main branch's predicted pose only) decode from the same frozen state, and the depth discrepancy between them yields per-state-token staticness weights that gate the state update; β‘‘ reset metric alignment β€” streaming models periodically reset memory to fight forgetting, and a reset makes the same repeated frame produce inconsistent scale before and after, so a Sim(3) estimated from the repeated frame's point clouds reconnects the new segment to the old scale; β‘’ state-aware smoothing β€” using "trajectory acceleration Γ— internal state change" as an uncertainty signal to adaptively and exponentially filter inter-frame displacements, suppressing pose jitter online.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image stream"] --> B["Main branch: image + RayMap<br/>decodes pose / depth / confidence"]
    B --> C["Dual-branch RayMap dynamic identification<br/>RayMap-only query gives depth gap<br/>β†’ staticness weights Ξ± gate state update"]
    C -->|at periodic reset| D["Reset metric alignment<br/>Sim(3) from repeated-frame point clouds"]
    D --> E["State-aware smoothing<br/>acceleration Γ— state change<br/>adaptively filters frame displacements"]
    E --> F["Pose / depth / point cloud"]

Key Designs

1. Dual-branch RayMap dynamic identification: turning the static bias of RayMap-only queries into a training-free dynamic gate

This is the core of the paper and the component that contributes most. The same frozen state \(s_{t-1}\) is decoded twice: the main branch consumes image tokens \(f_t\) together with RayMap tokens \(r_t\), while the RayMap branch consumes RayMap tokens only β€” and crucially, that RayMap is not the one bundled with the input frame; it is rebuilt from the main branch's own predicted pose \(\hat{T}_t\) and then encoded into \(r'_t\). The two branches therefore differ along exactly one axis, "whether appearance information is present" (same state, same viewpoint), so the discrepancy between their predictions can be attributed cleanly to dynamic content:

\[\hat{y}_t^{\text{main}} = \mathrm{Dec}(f_t + r_t,\ s_{t-1}), \qquad \hat{y}_t^{\text{raymap}} = \mathrm{Dec}(r'_t,\ s_{t-1})\]

Each prediction carries a depth map and a confidence map. Per pixel, the relative depth difference between the main and RayMap branches, \(\delta_i = |z_i^{\text{main}} - z_i^{\text{raymap}}| / |z_i^{\text{main}}|\), grows with the likelihood that the pixel is dynamic β€” because the main branch can see it while the RayMap branch can only recall the static background at that location from memory. State updates, however, act on state tokens rather than pixels, so \(\delta\) is aggregated in two hops: first a confidence-weighted average within each patch (weights are the main branch's confidence scores) pools pixel scores into image-token-level scores; then those are projected onto each state token as a weighted average using the decoder's cross-attention weights \(A_{jk}\) between state token \(j\) and image token \(k\). The per-state-token discrepancy \(\delta^{\text{state}}\) is finally converted into staticness weights:

\[\alpha_t = \sigma\!\left(\gamma \cdot \frac{\operatorname{median}(\delta^{\text{state}}) - \delta^{\text{state}}}{\operatorname{IQR}(\delta^{\text{state}})}\right)\]

Median and interquartile range are used instead of mean and variance because dynamic regions usually occupy only a small part of the frame and are textbook outliers; robust statistics keep a few large deviations from skewing the overall scale. The scalar \(\gamma\) controls the sensitivity of dynamic-static separation, and the sigmoid squeezes the result into \((0,1)\). As a result, tokens whose \(\delta\) falls below the median receive \(\alpha\) close to 1 and are updated as usual, while dynamic tokens with markedly higher \(\delta\) receive \(\alpha\) close to 0 and are kept out of memory. The weights are additionally smoothed over time with an exponential moving average for temporal stability, and the final state update is written as

\[s_t = s_{t-1} + \alpha_t \odot \Delta s_t\]

where \(\Delta s_t\) is the state increment produced by the decoder. In addition, the pixel-level staticness map is used to build a weighted global feature that biases pose retrieval toward static regions during memory updates. The whole dual-branch procedure runs frame by frame after a short warmup (main branch only) and involves no back-propagation.

This design contrasts directly with TTT3R's soft gating: TTT3R's gate comes from an implicit cross-attention signal and cannot hold back large dynamic regions; here the gate comes from an explicit depth comparison between two input modalities querying the same state, which is a more direct criterion and pulls ATE on Sintel from 0.210 to 0.166 (Tab. 2).

2. Reset metric alignment: reconnecting the two segments around a reset to a single scale

The memory of a streaming model is gradually washed out by new observations as the sequence advances, so a common remedy is a periodic reset β€” clearing the state and reinitializing it from a "repeated frame" to preserve temporal continuity. The authors point out a previously overlooked problem with this reset: it alters the memory state, so the same repeated frame gets decoded into inconsistent camera parameters and geometry before and after the reset. This shows up as a scale mismatch between segments and propagates as systematic drift into all subsequent frames, accumulating error over long sequences.

The fix exploits a physical property of the repeated frame: it observes the same physical scene on both sides of the reset, so the difference between the two reconstructions is the metric misalignment introduced by the reset β€” a measurable self-supervised signal. Concretely, a Sim(3) transformation is estimated from the repeated frame's point clouds before and after the reset, covering both the scale mismatch and the pose offset, and then applied to all subsequent frames of the new segment to restore metric consistency and curb error accumulation. Compared with global optimization over the whole trajectory, this pays a single tiny estimation cost at each reset boundary and remains fully compatible with online inference.

3. State-aware smoothing: using internal state change as an uncertainty proxy to filter trajectories online

Streaming methods estimate poses frame by frame, so noise accumulates over time; post-hoc optimization such as MASt3R-SfM, DROID-SLAM, or DPVO can smooth a trajectory offline but is incompatible with online processing. The problem to solve is therefore "how to judge whether a frame's pose is trustworthy without looking into the future." The authors use two complementary signals. The first is the state change signal \(sc_t\), the mean \(\ell_2\) norm of \(\Delta s_t\) over all state tokens, computed before the gated update so that it reflects the full change the model proposes β€” a proxy for how uncertain the model is at that moment. The second is the trajectory acceleration \(a_t = \lVert d_t - d_{t-1} \rVert_2\), where \(d_t = \tau_t - \tau_{t-1}\) is the inter-frame displacement, a proxy for how irregular the motion is.

The key is to multiply the two rather than use either alone: acceleration by itself would mistake steady fast motion for noise and over-smooth it, while the state change signal by itself would also fire on a genuine new observation. Only when both are large is the motion abrupt and the model unsure of itself. The product is mapped through an inverse relation into a smoothing coefficient and used to exponentially filter the inter-frame displacement:

\[\beta_t = \frac{1}{1 + \lambda |a_t \times sc_t|}, \qquad \hat{d}_t = \beta_t d_t + (1-\beta_t)\hat{d}_{t-1}\]

When the product is large, \(\beta_t \to 0\) and the filter falls back on the accumulated estimate \(\hat{d}_{t-1}\), attenuating jitter; when the product is small, \(\beta_t \to 1\) and the raw displacement is kept. The position is updated recursively as \(\hat{\tau}_t = \hat{\tau}_{t-1} + \hat{d}_t\), and expanding the recursion gives the closed-form trajectory

\[\hat{\tau}_t = \tau_0 + \sum_{m=1}^{t}\sum_{k=1}^{m} \beta_k \prod_{l=k+1}^{m}(1-\beta_l)\, d_k\]

where the product term \(\beta_k \prod_{l=k+1}^{m}(1-\beta_l)\) governs the effective contribution of the displacement at frame \(k\) to the current estimate, forming an adaptive decay: weights concentrate on recent frames during stable intervals and the filtering horizon widens during high-uncertainty intervals. The scheme is inherently causal, needs no explicit history storage, and costs almost nothing. Note that the displacement \(d_t\) here is the translation component \(\tau_t \in \mathbb{R}^3\) only, i.e. the filter acts on position but not on rotation (⚠️ the original paper does not discuss rotational jitter).

A Worked Example

Take a handheld sequence containing a walking pedestrian (the numbers below illustrate the mechanism and are not measured values from the paper). In some frame a person walks by on the right: the main branch sees them and predicts roughly 2.0 m of depth in that region (foreground), whereas the RayMap branch, with no appearance information, recalls the static background there from memory and gives roughly 3.1 m. The relative depth difference over those pixels is about 0.5, far above the roughly 0.02 on the static wall. Pixel scores are pooled by the main branch's confidence within each patch into image-token scores, then spread onto the corresponding state tokens via the decoder's cross-attention weights. Because the pedestrian occupies only a small part of the frame, those tokens' \(\delta^{\text{state}}\) are outlier highs; after median/IQR standardization and the sigmoid they are pressed toward 0, so \(\alpha \approx 0\) and "a person walked past" is essentially never written into memory. The wall's state tokens in the same frame sit below the median with \(\alpha \approx 1\) and are updated normally. After the next reset, the point clouds reconstructed from the repeated frame on both sides of the reset yield a Sim(3) that corrects the new segment's scale and pose offset in one shot; and if handheld shake produces an anomalous displacement in a frame, \(a_t \times sc_t\) grows, \(\beta_t\) shrinks, and that displacement is filtered out instead of contaminating the trajectory.

Loss & Training

RayMap3R is a training-free framework: there is no new loss function, no fine-tuning of the base model, and no external network β€” all three components act at inference time, and the dual-branch forward pass itself runs without gradients. The only knobs are inference-time ones: \(\gamma\) (sensitivity of the staticness-weight separation), \(\lambda\) (sensitivity of the smoothing map), the exponential moving average coefficient for the staticness weights, and the number of warmup frames before the dual branch starts; the paper does not report concrete values or a sensitivity analysis for these (⚠️ refer to the original paper). The ablation uses CUT3R as the base, and the full model is that base plus dual-branch identification (R), reset metric alignment (M), and state-aware smoothing (S).

Key Experimental Results

Main Results

Evaluation covers three tasks β€” video depth estimation, camera pose estimation, and 3D reconstruction β€” on the dynamic datasets Sintel, TUM-Dynamics, KITTI, and Bonn and the static datasets ScanNet and 7-Scenes.

Camera pose (Tab. 2, with Sim(3) alignment; ATE and translational RPE, lower is better):

Method Online Sintel ATE↓ Sintel RPE_t↓ TUM-dyn ATE↓ TUM-dyn RPE_t↓ ScanNet ATE↓
StreamVGGT βœ“ 0.251 0.149 0.061 0.033 0.161
Point3R βœ“ 0.351 0.128 0.075 0.029 0.106
Spann3R βœ“ 0.329 0.110 0.056 0.021 0.096
CUT3R βœ“ 0.208 0.072 0.031 0.009 0.098
TTT3R βœ“ 0.210 0.091 0.019 0.008 0.065
Ours βœ“ 0.166 0.056 0.018 0.005 0.064

Video depth (Tab. 1, per-sequence scale alignment; Abs Rel↓ / Ξ΄<1.25↑):

Method KITTI Abs Rel↓ KITTI Ξ΄<1.25↑ Bonn Abs Rel↓ Bonn Ξ΄<1.25↑ Sintel Abs Rel↓ Sintel Ξ΄<1.25↑
Point3R 0.135 84.0 0.061 96.2 0.451 48.7
StreamVGGT 0.173 72.1 0.063 97.2 0.323 65.7
CUT3R 0.118 88.1 0.078 93.7 0.421 47.9
TTT3R 0.114 90.4 0.068 95.4 0.409 48.8
Ours 0.098 92.8 0.057 97.4 0.401 50.9

3D reconstruction and efficiency (Tab. 3 on 7-Scenes, 200 frames per scene; Tab. 4 on a single RTX A6000 48GB):

Method Acc↓ (mean) Comp↓ (mean) NC↑ (mean) Chamfer↓ (mean) 50 views mem / FPS 1000 views mem / FPS
Spann3R 0.298 0.205 0.650 0.503 β€” β€”
CUT3R 0.043 0.031 0.621 0.027 6.4 GB / 19.7 FPS 6.5 GB / 19.7 FPS
TTT3R 0.027 0.023 0.582 0.025 7.6 GB / 19.6 FPS 7.7 GB / 19.6 FPS
Point3R β€” β€” β€” β€” 30.0 GB / 5.0 FPS OOM
VGGT β€” β€” β€” β€” 20.0 GB / 21.0 FPS OOM
MonST3R β€” β€” β€” β€” 32.0 GB / 0.31 FPS OOM
Ours 0.023 0.022 0.629 0.024 9.2 GB / 13.8 FPS 9.4 GB / 13.8 FPS

Ablation Study

Component-by-component ablation of the three designs on the CUT3R base (Tab. 5; the original table does not state the dataset or protocol used, ⚠️ refer to the original paper):

Config Pose ATE↓ Depth Abs Rel↓ Recon. Chamfer↓ Note
Base (CUT3R) 0.114 0.208 0.322 no component
+R 0.113 0.186 0.245 dual-branch identification dominates the depth/reconstruction gain, with almost no effect on trajectory
+R+M 0.110 0.184 0.213 metric alignment mainly fixes Chamfer by removing cross-segment scale mismatch
+R+S 0.084 0.185 0.196 smoothing mainly fixes the trajectory, with a large ATE drop
Full (R+M+S) 0.081 0.183 0.170 the three are complementary, best on every metric

Quantitative evaluation of the dynamic map itself (Tab. 6, 108 sequences and 6631 frames in total; disc is the mean discrepancy ratio between dynamic and static regions, AUC casts the signal as a binary classifier with 0.5 as chance, and ρ is the Spearman correlation with the ground-truth dynamic ratio):

Dataset Seqs Frames Mean disc↑ Mean AUC↑ Mean IoU↑ ρ↑
MPI Sintel 19 746 1.61 0.464 0.298 0.900
DAVIS 2017 81 5205 1.68 0.538 0.189 0.713
TUM RGB-D 8 680 1.88 0.560 0.206 0.643
All 108 6631 1.69 0.532 0.203 0.771

Key Findings

  • The three components have distinct, complementary roles rather than overlapping ones: dual-branch identification (R) cuts depth Abs Rel from 0.208 to 0.186 and Chamfer from 0.322 to 0.245, making it the largest contributor to depth and reconstruction, yet it barely moves ATE (0.114 β†’ 0.113) because it operates at the state level and filters geometry rather than trajectory; metric alignment (M) pushes Chamfer further from 0.245 to 0.213; state-aware smoothing (S) targets the trajectory specifically, and adding it alone drives ATE from 0.110 down to 0.084, with the full model at 0.081.
  • Clear gains on dynamic scenes, no degradation on static ones: on Sintel, where dynamic objects occupy a large share of the frame, ATE drops from the next-best 0.210 to 0.166; the method also leads on TUM-dynamics. On static ScanNet it ties the best streaming method (0.064 vs TTT3R's 0.065), showing that the staticness weights do not harm normal updates when nothing is moving.
  • There is one honest exception on depth: under per-sequence scale alignment on Sintel, StreamVGGT achieves lower Abs Rel than Ours (0.323 vs 0.401), but StreamVGGT's memory grows with sequence length and it is not a constant-memory streaming solution; under the metric-scale protocol Ours leads on Bonn and KITTI and matches the strongest streaming method on Sintel (0.954 vs TTT3R's 0.977) with higher Ξ΄<1.25 (24.0 vs 23.2).
  • Speed and memory are the real cost of this family of methods: the dual branch costs roughly 30% of the frame rate (13.8 FPS vs about 19.7 FPS for CUT3R/TTT3R) and raises memory from about 6.5 GB to about 9.4 GB, but it stays flat at 9.4 GB with 1000 frames. Offline methods VGGT and MonST3R already need 20–32 GB at 50 views, and Point3R, though online, runs out of memory past 900 views because of its explicit point memory.
  • The dynamic map is a soft gating cue, not a segmenter: disc exceeds 1 on all three datasets (1.61–1.88), confirming that the depth discrepancy really is systematically larger in dynamic regions; AUC exceeds 0.5 on real-world datasets (DAVIS 0.538, TUM 0.560) and is thus discriminative. Sintel's AUC is below 0.5 (0.464) while its ρ is the highest (0.900) β€” heterogeneous motion patterns make threshold-based binary classification harder, yet the sequence-level ranking signal remains strong. An IoU near 0.20 is expected, since a continuous signal map is thresholded and compared against binary ground truth; it is meant as a hint rather than a segmentation output.
  • Closing in on offline methods: on 7-Scenes, Ours beats the offline dynamic method MonST3R-GA on Acc and Chamfer (mean Acc 0.023 vs 0.248, mean Chamfer 0.024 vs 0.514), showing that selective state updates can partly compensate for the absence of global optimization in the streaming setting; offline feed-forward methods such as VGGT and DUSt3R remain ahead, but they are not applicable to long sequences in the first place.

Highlights & Insights

  • Treating the model's own defect as free supervision is the paper's true "aha" moment. RayMap tokens carry no appearance, so the model sees no dynamic object when fed RayMap alone β€” from a conventional standpoint that is a defect. The authors invert it into a stable dynamic detector: since the model can only recall static structure, a large two-branch discrepancy means something is moving there. Dynamic identification therefore needs no labels, no training, and no external model.
  • Building the RayMap query from the model's own predicted pose confines the two-branch difference strictly to the input modality β€” same frozen state, same viewpoint, with the presence of appearance as the only variable. That is how one writes a controlled experiment into an architecture, and it lets the depth discrepancy be attributed cleanly to dynamic content rather than to viewpoint or state differences.
  • The two-hop aggregation from pixels to state tokens (pixel β†’ image token via confidence weighting β†’ state token via cross-attention weights) solves a very practical mismatch: the evidence is computed on pixels, but the thing to be gated is a state token. This idea of "borrowing the decoder's attention to move evidence onto the units that need gating" transfers directly to any memory-based streaming model with cross-attention decoding.
  • Using internal state change as an uncertainty proxy avoids training a separate confidence network, and multiplying it with trajectory acceleration separates "moving fast at constant speed" from "the model is unsure," two situations that each inflate a single signal on its own, so stable fast motion is not over-smoothed.
  • The repeated frame is a natural calibration target: the reset mechanism already requires replaying a frame, so the authors reuse it to estimate a Sim(3) that corrects both scale and pose offset, paying the estimation cost only at reset boundaries.
  • Median/IQR robust standardization for turning a continuous discrepancy into gating weights is worth reusing: dynamic regions are naturally a minority in the frame (outliers), so mean/variance gets dragged by a few large deviations whereas quantiles adaptively decide "how large counts as large."

Limitations & Future Work

  • Limitations admitted by the authors: the method relies on the implicit dynamic awareness the model acquires during training, and this bias is inherited from CUT3R's backbone, so its reliability depends on whether that model was trained on dynamic scenes; the scope is therefore restricted to streaming models that have recurrent memory and natively support RayMap-only querying. The authors also note that as RayMap representations spread to offline feed-forward models, studying comparable query mechanisms in other RayMap-based architectures is a natural direction for extending dynamic reconstruction at broader scales.
  • Narrow applicability: the training-free premise requires the base model to decode geometry from RayMap alone. Streaming models without such a RayMap-only query branch (Spann3R, Point3R, StreamVGGT) cannot use this plug-in directly, so it is closer to a drop-in upgrade for the CUT3R/TTT3R family than a general dynamic reconstruction solution.
  • Uneven evaluation coverage: quantitative 3D reconstruction is only measured on 7-Scenes, a static indoor dataset (200 frames per scene); on dynamic datasets (Sintel, TUM-dynamics, DAVIS) there are only pose, depth, and qualitative results. Direct numerical evidence for "reconstruction quality in dynamic scenes" β€” precisely the goal in the title β€” is missing. In addition, the ablation table does not state its dataset or protocol, and the values and sensitivity of \(\gamma\), \(\lambda\), and the warmup length are unreported.
  • Smoothing acts on translation only: state-aware smoothing filters the inter-frame displacement \(d_t\) (the translation component), leaving rotation unsmoothed, so rotational jitter remains; Tab. 2 shows Ours slightly behind DUSt3R on Sintel rotational RPE (0.720 vs 0.579) and roughly level with TTT3R (0.722), which sits in some tension with the "more stable trajectory" narrative.
  • A degenerate case for the IQR normalization: on a frame that is almost entirely static, the interquartile range of \(\delta^{\text{state}}\) becomes very small, and the division in the equation amplifies numerical noise, making \(\alpha\) unstable; the paper does not discuss this case. A natural fix is a floor such as \(\max(\text{IQR}, \epsilon)\) or cross-frame sliding-window statistics to stabilize the scale (⚠️ this point is this note's own analysis, not a conclusion of the paper).
  • Errors may be masked by the dynamic gate: because state updates are suppressed in dynamic regions, a static region misclassified as dynamic will simply stop being updated without ever raising an error. Such silent failures are hard to expose with the current metrics, and a diagnostic such as "long-term coverage of suppressed tokens" would be worth adding.
  • vs CUT3R: CUT3R uses an implicit memory cache with a location dictionary for continuous real-time reconstruction and is this paper's base; but in dynamic scenes its memory is contaminated by moving objects and it drifts visibly (Sintel ATE 0.208, 7-Scenes Chamfer 0.027). RayMap3R adds three inference-time components on the same backbone and lifts it to ATE 0.166 / Chamfer 0.024 without retraining, at the price of a modest memory and frame-rate concession.
  • vs TTT3R: TTT3R also extends CUT3R, using cross-attention as a soft gate to mitigate memory forgetting, and is the most direct competitor. The difference is that its gate comes from an implicit attention signal and cannot hold back large dynamic regions, whereas this paper's criterion is an explicit modality contrast; hence the clearly lower ATE on Sintel and TUM-dynamics (0.166 vs 0.210, 0.018 vs 0.019). On 3D reconstruction the two are already close (mean Chamfer 0.024 vs 0.025), and TTT3R is slightly better on minimum Comp and median Chamfer.
  • vs offline dynamic methods such as MonST3R / MegaSaM: MonST3R handles motion with optical flow and is more accurate in some offline settings, but it needs the full sequence and its memory explodes (32 GB at 50 views, OOM at 1000 views), and it cannot output online. On 7-Scenes, Ours actually surpasses MonST3R-GA on Acc and Chamfer, indicating that "selectively writing to memory" can partly substitute for global optimization in a streaming method.
  • vs flow-, segmentation-, and tracking-based dynamic reconstruction: all three families require external modules β€” flow-based methods depend on an off-the-shelf flow estimator with hand-tuned thresholds and generalize poorly, segmentation-based methods are bounded by their training categories, and tracking-based methods require known intrinsics and iterative optimization. This paper's route is zero external modules and zero training, replacing them with an implicit prior read out of the model's own behavior.
  • vs offline feed-forward models such as VGGT / DUSt3R / MASt3R: they reach the highest accuracy through dense pairwise matching and global attention over the full sequence, but they must first collect every frame and their cost grows rapidly with input count. This paper targets an entirely different constraint β€” online, constant memory, thousands of frames β€” and within that constraint pushes streaming accuracy to a new ceiling.
  • Transferable insight: any multimodal feed-forward model that can decode from a partial modality may harbor a similar modality bias (e.g. depth-only or text-only prompting), and such a bias is itself a source of annotation-free priors; the key is to find a controlled comparison that reads the bias out, and then to transport the discrepancy onto whatever unit needs intervention.

Rating

  • Novelty: ⭐⭐⭐⭐ Repurposing the static bias of RayMap-only queries as a training-free dynamic prior is a distinctive angle backed by a solid observation (statistically validated on 108 sequences); the individual components (Sim(3) alignment, exponential smoothing) are themselves fairly conventional.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Covers three tasks and six datasets, plus a four-metric quantitative analysis of the dynamic map and measured speed/memory; weaker on quantitative reconstruction evaluation in dynamic scenes, and the ablation table omits its dataset while hyper-parameters and warmup get no sensitivity analysis.
  • Writing Quality: ⭐⭐⭐⭐ The chain from observation (static bias) to mechanism (why static content is recalled reliably) to method (how the gate is read out) is clear, and the figures and tables agree with the claims; some formula layout and table annotations are rough.
  • Value: ⭐⭐⭐⭐ Drop-in, training-free, and constant-memory: a low-cost upgrade to the dynamic robustness of existing streaming 3R models, and the "mine priors from a model's own bias" idea is instructive for feed-forward geometry models more broadly.