LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory

Conference: ECCV2026
Paper: ECCV Paper
Project: https://LoGeR-project.github.io/
Area: 3D Vision
Keywords: dense 3D reconstruction, long-sequence modeling, hybrid memory, sliding-window attention, test-time training

TL;DR

LoGeR reconstructs video chunk by chunk, preserving adjacent-chunk detail through sliding-window attention and propagating global geometry through test-time-updated fast weights; its feedforward-aligned LoGeR* variant reduces average KITTI ATE from TTT3R's 72.86 m to 18.65 m, although very-long-sequence evaluation includes periodic state resets.

Background & Motivation

Geometric foundation models such as DUSt3R, VGGT, and π3 directly predict dense pointmaps and camera poses from multiple images, reducing dependence on conventional matching and optimization backends. Their bidirectional multi-view attention exploits complementary observations, but extending it to thousands or tens of thousands of frames incurs quadratic attention costs. Improving inference efficiency, as in FastVGGT, also does not automatically turn training on spatially bounded scenes into reliable city-scale reconstruction.

Splitting a video into small chunks addresses memory pressure but leaves independently estimated coordinate systems and scales to reconcile. Similarity transformations estimated from overlapping frames can propagate local scale errors along a trajectory. Compressing all history into a recurrent state, as in CUT3R-style models, can instead discard the fine detail needed for precise boundary alignment. Recent geometry and distant history therefore impose different memory requirements: the former needs high-fidelity information, while the latter needs a compact coordinate and scale reference.

The paper consequently distinguishes fitting more frames from reconstructing a larger scene, changing both cross-chunk information pathways and the training distribution. Core Idea: use adjacent-chunk sliding-window memory for precise local transitions and global fast-weight memory for compressed long-range geometric references, then train their cooperation with large-scene data and a progressive curriculum.

Method

Overall Architecture

The input is a temporally ordered monocular RGB video. Outputs are dense per-frame pointmaps in local camera coordinates and camera poses in a global coordinate system; transforming the pointmaps by the poses produces world-coordinate point clouds. Processing is causal between chunks but bidirectional within a chunk, so this is chunk-level streaming rather than strictly frame-by-frame causal prediction.

The backbone inherits π3's patchifier, per-frame attention, within-chunk bidirectional attention, and prediction heads. Inside a residual block, processing follows per-frame attention, sparsely inserted adjacent-chunk sliding-window memory, global fast-weight memory, and within-chunk bidirectional attention. This structure repeats across network depth before pointmap and pose decoding. LoGeR* additionally constructs a feedforward rigid alignment from an overlapping frame's poses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB video"] --> B["Chunk-wise geometry reasoning<br/>Chunking and per-frame encoding"]
    B --> C["Adjacent-chunk sliding-window memory"]
    C --> D["Global fast-weight memory"]
    D --> E["Within-chunk bidirectional attention<br/>Pointmap and pose decoding"]
    E --> F["Feedforward pose alignment<br/>LoGeR* branch"]
    E -->|LoGeR| G["Per-frame poses and dense pointmaps"]
    F --> G
    D -->|Write state for the next chunk| D

The diagram summarizes repeated network blocks rather than depicting a network with only one attention layer. The adjacent-chunk feature cache and the fast weights are separate states: the former retains recent explicit features, while the latter writes longer history into a fixed-size parameter state.

Key Designs

1. Chunk-wise geometry reasoning: retain bidirectional multi-view inference within short windows

Converting a bidirectional model directly into a frame-wise recurrence can weaken its use of multiple views for geometry. LoGeR instead partitions consecutive frames into slightly overlapping chunks while retaining multi-view interaction inside each chunk. Each network block first extracts spatial features independently per frame, incorporates cross-chunk memory, and then applies within-chunk bidirectional attention. Current geometry can consequently depend on both local multi-view evidence and information propagated from earlier chunks.

The local pointmap decoder predicts each pixel's 3D position in that frame's camera coordinates, while the pose decoder relates each camera to the global coordinate system. Keeping these outputs distinct matters: correct local shape does not guarantee that all chunks share a coherent world coordinate system. The memory pathways and global supervision address this gap.

Chunking also keeps each local inference closer to the distribution of available short-sequence training data. With fixed resolution, chunk size, and memory capacity, total processing cost grows linearly with sequence length. This does not remove the quadratic cost inside a chunk, nor does it make storage of all output point clouds independent of the number of frames.

2. Adjacent-chunk sliding-window memory: retain precise alignment evidence as explicit features

Geometric errors at chunk boundaries often require fine-grained correspondence evidence that a compressed state cannot reliably preserve. Sliding Window Attention (SWA) accesses the per-frame attention outputs from the previous and current chunks. Current processing can therefore use high-fidelity tokens from its immediate predecessor rather than relying solely on a compressed summary.

This pathway spans only neighboring chunks and is inserted at four network depths, avoiding a return to a full-history KV cache. The paper's description of local memory as lossless means that features are retained without compression, not that geometry estimates are error-free. Older explicit features leave the window, so SWA alone cannot maintain a long-distance scale reference.

3. Global fast-weight memory: read history before writing the current chunk into state

The complementary pathway uses Large-Chunk Test-Time Training (LaCT). Instead of retaining an ever-growing token sequence, it compresses history into a fast-weight module. The fast weights use a SwiGLU network and Muon updates at test time, while the backbone's slow weights remain frozen. Test-time training here refers to updating an internal memory state, not retraining the entire reconstruction network.

Section 3 explains the memory through key-value association: predict a value from its key to store that association in parameters, then retrieve information using a query. Section 4.1 specifies a chunk-level apply-then-update procedure. Existing fast weights first modulate the current chunk's representations with historical geometry; the current chunk then updates the weights for the next chunk. Pre-normalization stabilizes long-horizon state propagation.

Fast weights are suited to compressible information such as coarse geometry, coordinate references, and scale, whereas SWA retains nearby detail. Neither pathway replaces the other. However, fixed-size state does not imply indefinitely reliable memory. In long-sequence evaluation, the paper resets fast weights every five windows and performs feedforward pose alignment at a reset. Results on tens of thousands of frames therefore do not demonstrate one uninterrupted fast-weight state spanning the entire sequence.

The claim of no backend optimization also requires qualification. There is no additional pose-graph or global reconstruction optimization backend, but the fast weights still undergo test-time updates. A feedforward reconstruction pipeline can coexist with this internally updated memory.

4. Feedforward pose alignment: connect coordinate systems through a shared frame

LoGeR* addresses residual coordinate inconsistency by explicitly using a frame shared by neighboring chunks. Let its already aligned pose in the previous chunk be \(\widetilde{T}^{(m-1)}_k\), and its raw prediction in the current chunk be \(\widehat{T}^{(m)}_k\). The alignment in Section 4.2 and Equation (12) is:

\[ A_m=\widetilde{T}^{(m-1)}_k\left(\widehat{T}^{(m)}_k\right)^{-1},\qquad \widetilde{T}^{(m)}_t=A_m\widehat{T}^{(m)}_t. \]

This determines how the current chunk's coordinate system fits into the preceding one and applies the same rigid transformation to every camera pose in the current chunk. It is an \(SE(3)\) alignment and does not estimate an additional scale. Unlike the Pi3-Chunk baseline, which estimates a \(SIM(3)\) transformation between chunks, it still relies on learned geometry and memory for scale consistency.

LoGeR uses this alignment during both training and inference, continuing fine-tuning from the first-stage model. It is consequently not just a test-time post-processing switch applied to an otherwise identical base model. Experimental results must distinguish LoGeR from LoGeR rather than assigning all starred-variant gains to the base model.

A Worked Example

Consider the short-sequence setting from Section 5.2: 64 frames per chunk with an overlap of 3 frames. The first chunk covers frames 1 through 64, and the second covers frames 62 through 125. These frame numbers illustrate window movement; they are not a new experiment.

When the second chunk enters the network, SWA reads retained features from the first chunk to support boundary correspondence. TTT first reads the fast weights written by the first chunk, then writes information from the second. Within-chunk bidirectional attention further resolves local shape using all views in the second chunk, and the prediction heads output per-frame pointmaps and poses.

For LoGeR*, a shared frame then provides the pose relationship used to place the second chunk in the first chunk's coordinate system. As processing continues, SWA retains only adjacent explicit context, while the compressed TTT state propagates onward. Periodic resets in very-long-sequence evaluation are a separate setting and should not be conflated with this short-sequence chunk size.

Loss & Training

Section 4.2 combines local pointmap, relative pose, and global pointmap supervision. Local pointmaps are aligned using one scale per sequence, and reconstruction errors are normalized by depth. Relative poses supervise rotation and scale-aligned translation, with a Huber penalty on translation. Global pointmap supervision transforms local predictions into world coordinates using predicted poses, reducing the freedom for individually plausible chunks to form an inconsistent reconstruction.

The combined objective appears in Equation (11):

\[ \mathcal{L}=\mathcal{L}_{\mathrm{local}}+\mathcal{L}_{\mathrm{pose}}+\lambda_{\mathrm{global}}\mathcal{L}_{\mathrm{global}}. \]

Some component equations are damaged in the text extraction, so missing terms and numerical loss weights are not reconstructed here. Geometry supervision defines the main training task; the self-supervised key-value association in TTT is an internal memory update mechanism. The paper therefore belongs to 3D Vision rather than self-supervised representation learning.

The curriculum starts with 48 frames split into 4 chunks, increases to 12 chunks while retaining 48 frames, and finally expands the context to 128 frames and progressively to 20 chunks. Increasing the number of boundaries makes reliance on local sliding-window context less sufficient and encourages use of global state.

The data mixture emphasizes large-scene sources including TartanAir, TartanAirV2, Waymo, Virtual KITTI 2, and OmniWorld-Game, alongside indoor and other real and synthetic data. Training uses AdamW for 40k steps with batch size 32. The authors report approximately two days on 32 H100 GPUs followed by approximately two days on 32 H200 GPUs; evaluation uses an A100 40GB. The maximum training context is only 128 frames, not a complete trajectory with tens of thousands of frames.

Key Experimental Results

Main Results

The following results are selected from Table 2: average Absolute Trajectory Error (ATE, meters, lower is better) across 11 KITTI sequences. Section 5.1 specifies Umeyama alignment of predicted and ground-truth trajectories. ATE measures trajectory position discrepancy after alignment, not unaligned absolute localization accuracy.

Method	Type	Average ATE (m)	Interpretation
DPV-SLAM++	Optimization-based	25.75	Includes an optimization backend
VGGT-Long	Optimization-based	27.64	Geometric foundation model with optimization
InfiniteVGGT	Feedforward	206.78	Long-stream baseline
CUT3R	Feedforward	91.62	Recurrent-state baseline
TTT3R	Feedforward	72.86	Previous TTT baseline
Pi3-Chunk	Feedforward	52.07	Independent chunks stitched by similarity transforms
LoGeR	Feedforward	25.44	Hybrid-memory version
LoGeR*	Feedforward	18.65	Feedforward pose alignment in training and inference

Calculated from Table 2, LoGeR reduces average ATE by approximately 74.4% relative to TTT3R and 26.7% relative to base LoGeR. The paper's prose mentions a 32.5% improvement over VGGT-Long; numerically, this refers to LoGeR, not the base model with 25.44 m ATE.

VBR contains 7 evaluation sequences, spanning 8,815 to 18,846 frames and trajectories of 1.4 to 11.5 km. Figure 4 reports a 55.2% relative improvement on long sequences, but the text cache does not provide a table of individually verifiable curve values, so no absolute ATE values at each horizon are invented here. This evaluation resets fast weights every five windows and performs feedforward alignment when resetting.

Short-sequence evaluation covers 50 to 500 frames for point-cloud reconstruction on 7-Scenes and 50 to 1,000 frames for pose estimation on ScanNetV2 and TUM-Dynamics, using 64-frame chunks and 3-frame overlap throughout. Figure 9 also notes that Pi3-Chunk can achieve slightly better pose metrics on small-scale TUM sequences while LoGeR produces more coherent visible structure. Trajectory error alone consequently does not characterize all aspects of dense reconstruction quality.

Ablation Study

The following values come from Table 3 and report ATE in meters, lower is better. Ablation models use fewer training frames than the final model and are evaluated on a ScanNet subset and TUM; their values should not be compared directly with the final-model KITTI results in Table 2.

Config	ScanNet 500 frames	ScanNet 1,000 frames	TUM 500 frames	TUM 1,000 frames
LoGeR	0.087	0.107	0.033	0.050
Without TTT	0.108	0.162	0.043	0.079
Without SWA	0.115	0.143	0.039	0.053
Without five large-scene datasets	0.102	0.156	0.050	0.072
LoGeR, without curriculum	0.098	0.133	0.049	0.062
LoGeR*	0.070	0.080	0.031	0.036
LoGeR*, without curriculum	0.078	0.093	0.029	0.040

Key Findings

• On ScanNet at 1,000 frames, removing TTT increases ATE from 0.107 to 0.162 m, while removing SWA increases it to 0.143 m. Both pathways matter, with long-range state particularly important in this setting.
• Removing the five large-scene datasets raises the same metric to 0.156 m. A scalable architecture does not substitute for an appropriate training scene distribution.
• Curriculum training generally helps but does not improve every entry: LoGeR* records 0.031 m on TUM at 500 frames, compared with 0.029 m without curriculum. The prose's claim of consistent improvement should not override this exception in the table.

Highlights & Insights

• Memory is divided by required information fidelity, not merely temporal range. Explicit tokens preserve precise adjacent alignment evidence, while compressed state carries long-term scale references, explaining the pathways' complementary roles.
• Chunking preserves the multi-view reasoning ability of a bidirectional geometric backbone as well as saving memory. Frame-wise recurrence and chunk-wise bidirectional processing are not interchangeable just because their asymptotic costs may match.
• Increasing the number of training chunks before increasing total frames is a curriculum aimed at cross-chunk dependence. It changes how many boundaries the model must bridge instead of simply enlarging its input.

Limitations & Future Work

• The authors acknowledge bounded fast-weight capacity and the need for resets on very long streams. Reliable memory span and the total number of frames the full pipeline can process are distinct quantities that should be reported separately.
• Linear complexity at fixed chunk size does not establish real-time operation. The cached main text lacks a complete throughput and latency table, and training uses dozens of high-end GPUs; deployment cost needs separate evaluation.
• Several appendix references appear as question marks, and the available cache has no appendix. Long-sequence chunk size, detailed implementation choices, and full hyperparameters cannot be filled in from short-sequence settings; only explicit main-text settings are used here.
• As a reader assessment, reduced-scale ablations support the mechanism but do not directly quantify each component's contribution on full VBR sequences. Further evaluation could vary reset intervals, dynamic-object prevalence, and the availability of overlapping views.

Related Work & Insights

• vs π3 / Pi3-Chunk: LoGeR inherits π3's within-chunk geometry reasoning but does not rely solely on \(SIM(3)\) stitching through overlapping frames for long-range consistency. Pi3-Chunk is already a strong baseline, so gains from chunking and gains from hybrid memory should be separated.
• vs CUT3R / TTT3R: These methods emphasize frame-wise state propagation, whereas LoGeR retains bidirectional reasoning within chunks and adds uncompressed adjacent context. Context organization is the central difference, not simply the presence of TTT.
• vs FastVGGT / VGGT-Long: The former mainly improves inference efficiency, while the latter incorporates optimization for long sequences. LoGeR learns cross-chunk memory and large-scene geometry, but having no optimization backend does not mean that no parameter state changes at test time.

Rating

• Novelty: 4/5. The memory components are established, but their roles are clearly tailored to geometric precision and scale drift.
• Experimental Thoroughness: 4/5. Long and short sequences plus component, data, and curriculum ablations are covered; complete efficiency and appendix details remain unavailable here.
• Writing Quality: 4/5. The mechanism is clear, but variant attribution and curriculum claims require checking the tables.
• Value: 5/5. The paper provides a concrete, analyzable approach to extending short-window geometric priors to long videos.