Fast Spatial Memory with Scalable Elastic Test-Time Training

Conference: ECCV 2026
OpenReview: https://eccv.ecva.net/virtual/2026/poster/4558
Paper: CVF Open Access
Code: https://fast-spatial-memory.github.io/
Area: 3D Vision
Keywords: test-time training, elastic weight consolidation, 4D reconstruction, novel view synthesis, spatial memory

TL;DR

To resolve catastrophic forgetting and fast-weight drift caused by unconstrained plasticity in Large Chunk Test-Time Training (LaCT) on dynamic sequences, this paper introduces Large Chunk Elastic Test-Time Training (LaCET) with Fisher-weighted elastic consolidation around a streaming-EMA anchor, establishing an efficient and scalable 4D spatial memory model (FSM).

Background & Motivation

Building a spatial memory requires compressing visual observations across viewpoints and time into a unified 4D representation that captures both geometric structure and temporal motion. While recent Large Reconstruction Models (LRMs) and Large View Synthesis Models (LVSMs) achieve impressive feedforward reconstruction by learning extensive multi-view priors, their standard Transformer sequence architectures remain fundamentally constrained by activation memory in a single forward pass. This activation bottleneck becomes particularly prohibitive in 4D dynamic domains, where observation videos are temporally extended yet spatially sparse, causing existing feedforward models to degrade sharply beyond their trained context length.

Test-Time Training (TTT) and its chunk-wise extension, Large Chunk Test-Time Training (LaCT), offer a compelling remedy by compressing ever-growing key-value contexts into dynamic fast weights updated in-forward without explicit backpropagation. However, the fully plastic nature of continuous LaCT updates fails in dynamic 4D regimes: as illumination, camera perspectives, and object motion continuously shift during inference, unconstrained fast-weight adjustments induce severe parameter drift. This leads to inference-time catastrophic forgetting and acute overfitting. Consequently, LaCT is typically forced into a single large chunk covering the entire input, forfeiting chunked streaming capabilities and frequently degenerating into trivial camera-pose interpolation shortcuts rather than acquiring true view-conditioned spatial intelligence.

To overcome this plasticity dilemma, this paper draws inspiration from Elastic Weight Consolidation (EWC) in continual learning: fast weights should adapt to novel observations without forfeiting foundational geometric memory. Core idea: propose Large Chunk Elastic Test-Time Training (LaCET), which integrates an elastic consolidation operator after the chunk-wise fast-weight update to softly pull critical parameters toward an evolving Streaming-EMA anchor guided by online Fisher-style importance estimates, yielding a highly scalable Fast Spatial Memory (FSM) for robust multi-chunk 4D reconstruction.

Method

Overall Architecture

FSM employs an end-to-end feedforward architecture trained solely with photometric supervision. The input comprises \(V\) posed images across arbitrary view-time combinations. Each image is concatenated with its canonical Plücker ray map and normalized timestamp map, partitioned into non-overlapping patches, and projected linearly into \(D\)-dimensional token embeddings. These tokens are processed by stacked LaCET blocks, where fast weights inside SwiGLU layers are updated chunk-by-chunk using input-view key-value statistics and elastically consolidated to suppress drift. For target view generation, FSM supports two decoupled rendering decoders: a lightweight LVSM-style image decoder that predicts RGB patches directly, and an LRM-style 4D Gaussian Splatting (4DGS) decoder that outputs pixel-aligned Gaussian primitives for tile-based rasterization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Spatiotemporal Observations<br/>RGB Images + Plücker Ray Maps + Timestamps"] --> B["Spatiotemporal Patch Tokenization<br/>Concat(I, P, T) linearly projected to D-dim tokens"]
    B --> C["Chunk-wise Key-Value Adaptation<br/>SwiGLU-MLP forward and surrogate pseudo-gradient update"]
    C --> D["Streaming Elastic Consolidation<br/>Soft pullback to Streaming-EMA anchor via online importance"]
    D --> E["Dual-Modality Rendering Decoders<br/>Direct LVSM patch prediction / LRM 4DGS rasterization"]
    E --> F["Predicted Novel View & Time<br/>High-fidelity 4D novel view synthesis"]

Key Designs

1. Chunk-wise Key-Value Adaptation: High-Throughput Fast-Weight Updates Decoupled from Target Views To overcome the poor hardware efficiency of per-token TTT, LaCET amortizes adaptation across chunks of size \(b\). The fast-weight network is parameterized by a bias-free SwiGLU-MLP with weight matrices \(\boldsymbol{\theta} = \{\boldsymbol{\theta}_1, \boldsymbol{\theta}_2, \boldsymbol{\theta}_3\}\). For all tokens \(x_i\) within chunk \(c\), the model evaluates the associative loss between transformed keys and values: $\(\mathcal{L}(f_{\boldsymbol{\theta}}(k_i), v_i) = -\,f_{\boldsymbol{\theta}}(k_i)^\top v_i = -\,[\mathrm{SiLU}(\boldsymbol{\theta}_1 k_i) \circ (\boldsymbol{\theta}_2 k_i)]^\top \boldsymbol{\theta}_3^\top v_i\)$ The chunk surrogate update then computes a collective step for chunk \(c\): $\(\boldsymbol{\theta}'_c = \boldsymbol{\theta}_c - \sum_{i=1}^b \eta_i(x_i) \left. \nabla_{\boldsymbol{\theta}} \mathcal{L}(f_{\boldsymbol{\theta}}(k_i), v_i) \right|_{\boldsymbol{\theta}=\boldsymbol{\theta}_c}\)$ Crucially, only input-view tokens are passed through the key-value projections to compute pseudo-gradients. Target-view query tokens are strictly prohibited from generating fast-weight updates, preventing cross-target information leakage or dynamic evaluation shortcuts and ensuring that novel views can be rendered independently.

2. Online Synaptic Importance Estimation: Lightweight Second-Order Curvature Tracking To determine which parameters carry persistent spatial-temporal memory, LaCET continuously tracks parameter importance online without costly offline Fisher calculations. The importance matrix \(F_c\) is maintained as an exponential moving average (EMA) across chunk index \(c\): $\(F_{c+1} = \alpha F_c + (1 - \alpha)\varphi(\mathbf{S}_c)\)$ where \(\alpha\) is the decay factor and \(\mathbf{S}_c\) represents the chunk update statistic. The architecture supports three estimators: - Memory Aware Synapses (MAS): \(\mathbf{S}_c = \boldsymbol{\theta}'_c - \boldsymbol{\theta}_c\) with \(\varphi(\mathbf{S}_c) = |\mathbf{S}_c|\), capturing the absolute update magnitude; - Elastic Weight Consolidation (EWC): \(\mathbf{S}_c = \boldsymbol{\theta}'_c - \boldsymbol{\theta}_c\) with \(\varphi(\mathbf{S}_c) = \mathbf{S}_c^2\), prioritizing parameters subjected to consistently large squared updates; - Synaptic Intelligence (SI): \(\mathbf{S}_c = (\boldsymbol{\theta}'_c - \boldsymbol{\theta}_c) \odot (\boldsymbol{\theta}'_c - \boldsymbol{\theta}_c^\star)\) with \(\varphi(\mathbf{S}_c) = |\mathbf{S}_c|\), weighting the parameter displacement by its displacement relative to the anchor state. Empirically, the SI estimator delivers superior numerical stability during long-context pretraining, effectively pinpointing weights that sustain static background geometry and global structure.

3. Streaming Elastic Consolidation: An Adaptive Spring Mitigating Inference Drift Following the intermediate fast-weight update \(\boldsymbol{\theta}'_c\), LaCET applies the consolidate operator. This step acts as a quadratic penalty pulling parameters back toward their anchor reference \(\boldsymbol{\theta}_c^\star\): $\(\boldsymbol{\theta}_{c+1} = \boldsymbol{\theta}'_c - \lambda F_c \odot (\boldsymbol{\theta}'_c - \boldsymbol{\theta}_c^\star)\)$ where \(\lambda\) governs the elastic restorative stiffness. While a fixed Global anchor reduces consolidation to an importance-weighted \(L_2\) regularizer and an instantaneous Streaming anchor discards cross-chunk memory, the Streaming-EMA policy updates the anchor via \(\boldsymbol{\theta}^\star \leftarrow \beta \boldsymbol{\theta}^\star + (1 - \beta)\boldsymbol{\theta}\). This functions as a low-pass filter over the fast-weight trajectory. The resulting adaptive spring allows transient, low-importance parameters to flexibly adjust to local motion and illumination, while anchoring high-importance weights to prevent unbounded temporal drift.

4. Dual-Modality Rendering Decoders: Balancing Ultra-Fast Synthesis and Explicit Geometry FSM flexibly pairs its sequence backbone with two complementary rendering paradigms: - LVSM-style direct decoding: Target view queries with empty appearance channels and specified camera/time coordinates are appended to the sequence. A lightweight linear projection layer maps token representations directly to \(3p^2\)-dimensional RGB patches followed by a Sigmoid function. This setup avoids explicit geometric proxy construction and maximizes rendering speed; - LRM-style 4D Gaussian Splatting decoding: For applications requiring physical asset extraction or downstream simulation, the backbone outputs pixel-aligned 4DGS primitives (\(V \times H \times W\)). Each primitive is parameterized by a 20-dimensional vector encompassing 3D position, time center, RGB color, spatial scale, temporal scale, dual-quaternion rotations, and opacity, rendered via tile-based rasterization with deferred backpropagation to minimize GPU memory.

Loss & Training

The network is supervised end-to-end using photometric loss across \(U\) target views, combining mean squared error (\(\ell_2\)) and perceptual LPIPS loss (using VGGNet): $\(\mathcal{L} = \frac{1}{U} \sum_{i'=1}^U \left( \|\widehat{\mathbf{I}}^*_{i'} - \mathbf{I}^*_{i'}\|_2^2 + \mu \cdot \mathrm{LPIPS}(\widehat{\mathbf{I}}^*_{i'}, \mathbf{I}^*_{i'}) \right)\)$ with \(\mu = 0.5\). The pretraining curriculum progressively scales context difficulty, raising input resolution from \(128 \times 128\) to \(256 \times 256\), extending the temporal span from 128 to 256 frames, and dynamically increasing input view counts. Controlled ablations are trained on 8 NVIDIA H100 GPUs with batch size 16 at \(128 \times 128\) resolution for 32K steps (\(\approx 32\text{B}\) tokens).

Key Experimental Results

Main Results

On the large-scale real-world dynamic video benchmark Stereo4D and the dynamic benchmark NVIDIA, FSM was evaluated against optimization-based baselines (SoM, MoSca) and feedforward rendering models (L4GM, 4DGT, MoVieS) using matched baseline resolutions:

Dataset	Model	Resolution	PSNR (dB) ↑	LPIPS ↓	SSIM ↑	Speed / Characteristics
Stereo4D	4DGT	\(504 \times 504\)	24.62	0.102	0.785	Feedforward rendering
Stereo4D	MoVieS	\(504 \times 504\)	27.19	0.114	0.888	Feedforward (~1s / scene)
Stereo4D	FSM-LRM	\(256 \times 256\)	27.29	0.147	0.876	Explicit 4DGS feedforward
Stereo4D	FSM-LVSM	\(256 \times 256\)	32.16	0.043	0.931	End-to-end feedforward SOTA
NVIDIA	SoM (optimization)	\(379 \times 672\)	15.30	0.509	0.317	~10 min per scene
NVIDIA	MoSca (optimization)	\(379 \times 672\)	21.45	0.265	0.712	~45 min per scene
NVIDIA	L4GM	\(256 \times 256\)	10.07	0.587	0.235	Requires multi-view diffusion prior
NVIDIA	4DGT	\(504 \times 504\)	14.13	0.640	0.131	Feedforward rendering
NVIDIA	MoVieS	\(379 \times 672\)	19.16	0.315	0.514	Feedforward rendering
NVIDIA	FSM-LRM	\(256 \times 256\)	20.17	0.337	0.567	Explicit 4DGS feedforward
NVIDIA	FSM-LVSM	\(256 \times 256\)	23.90	0.105	0.747	Outperforms all feedforward baselines

On the static scene benchmark DL3DV (\(256 \times 256\)), FSM-LVSM scores 26.69 dB PSNR, 0.091 LPIPS, and 0.846 SSIM, outperforming both GS-LRM (23.02 dB) and LVSM (23.10 dB), indicating that elastic memory retains high fidelity on static scenes.

Ablation Study

On the Stereo4D test set (12-layer backbone, 32K steps, 32 input frames, 8 target views), the components of LaCET were ablated against vanilla LaCT configurations:

EWC Consolidation	Train #Chunks	Test #Chunks	Anchor Policy	Importance Estimator	Train \(\ell_2\) (\(\times 10^3\)) ↓	Test PSNR ↑	Test LPIPS ↓	Test SSIM ↑	Description
✗ (LaCT)	1	1	-	-	1.80	26.021	0.1179	0.792	Single-chunk full sequence baseline
✗ (LaCT)	4	4	-	-	2.04	26.908	0.0988	0.814	Multi-chunk without elasticity (overfitting)
✓ (LaCET)	4	4	streaming-ema	SI	2.36	29.989	0.0517	0.903	Full model (optimal configuration)
✓ (LaCET)	4	4	streaming-ema	EWC	2.36	29.781	0.0537	0.897	EWC squared-update estimator
✓ (LaCET)	4	4	streaming-ema	MAS	2.28	29.922	0.0519	0.899	MAS magnitude estimator
✓ (LaCET)	4	4	streaming	MAS	1.71	26.960	0.0966	0.817	Instantaneous anchor reset, lacks cross-chunk memory
✓ (LaCET)	4	4	global	MAS	3.00	28.347	0.0653	0.863	Fixed initial anchor, overly rigid
✓ (LaCET)	1	1	global*	MAS	1.73	26.965	0.0960	0.817	Degenerate single-chunk setting
✓ (LaCET)	1	4	streaming-ema	MAS	1.73	21.993	0.3429	0.650	Train/test chunk mismatch collapse

Key Findings

• Elastic consolidation suppresses test-time overfitting: Comparing 4-chunk LaCT without EWC (train loss 2.04, test PSNR 26.908 dB) against LaCET (train loss 2.36, test PSNR 29.989 dB), the test PSNR jumps by over 3 dB despite a slightly higher training loss. This proves that unconstrained plasticity primarily overfits to transient noise and local shortcuts.
• Streaming-EMA anchor is vital for multi-chunk continuity: Resetting the anchor per chunk (Streaming) drops PSNR to 26.960 dB, while freezing it to initialization (Global) yields 28.347 dB. Only Streaming-EMA delivers an effective low-pass filter over the parameter trajectory.
• Mitigating camera-pose interpolation shortcuts: Under continuous temporal views, vanilla LaCT exploits temporal proximity to perform trivial frame interpolation, suffering sharp degradation when evaluated on sparse viewpoints. In contrast, LaCET exhibits robust performance across sparse spatial-temporal samples, capturing genuine 4D scene geometry.

Highlights & Insights

• Continual learning principles for test-time sequence computing: Adapting EWC from multi-task continual learning into in-forward chunked TTT transforms the fast-weight memory into an adaptive spring that prevents unbounded parameter drift.
• Decoupling activation memory from sequence length: By enabling stable adaptation with small chunks (e.g., 2048 tokens per chunk), LaCET alleviates the activation memory bottleneck of Transformers, providing an architectural foundation for streaming 4D world models.
• Versatile dual-decoder framework under photometric supervision: The architecture switches seamlessly between direct image token reconstruction (LVSM) and explicit 4D Gaussian Splatting (LRM), achieving state-of-the-art performance without requiring multi-view diffusion priors.

Limitations & Future Work

• Reliance on accurate camera poses: The system constructs Plücker ray embeddings from known camera intrinsics and extrinsics, and does not yet support unposed in-the-wild dynamic video inputs.
• Pure photometric supervision may produce geometric artifacts: Although LaCET mitigates frame interpolation shortcuts, photometric losses (\(\ell_2\) + LPIPS) cannot fully guarantee 3D geometric consistency under extreme occlusion or rapid object deformation. Incorporating depth, optical flow, or multi-view geometric priors could improve physical accuracy.
• Dataset and compute scaling constraints: Current pretraining is demonstrated on Stereo4D and DL3DV; scaling to web-scale multi-modal 4D datasets remains an open opportunity.

Related Work & Insights

• vs LaCT (Zhang et al., ICLR 2026): While LaCT introduces large-chunk fast-weight adaptation for efficient inference, its unconstrained plasticity causes severe parameter drift in dynamic 4D sequences. LaCET introduces Fisher importance estimation and Streaming-EMA anchors to stabilize multi-chunk adaptation.
• vs 4D-LRM (Ma et al., NeurIPS 2025) & MoVieS (Lin et al., CVPR 2026): Prior feedforward 4D models rely on full-attention sequence layers or short-range motion scaffolds, suffering from linear or quadratic activation memory growth. FSM compresses historical context into neural fast weights, outperforming MoVieS by nearly 5 dB PSNR on Stereo4D.
• vs tttLRM (Wang et al., CVPR 2026): tttLRM applies TTT to static 3D autoregressive reconstruction, whereas FSM extends test-time adaptation to dynamic 4D space-time representations with elastic stabilization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant integration of Elastic Weight Consolidation into test-time training, addressing fast-weight drift in dynamic 4D spatial memory.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across 4D and 3D datasets, with rigorous ablations on anchor dynamics, importance metrics, and context density scaling.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulations, insightful analysis of LaCT failure modes, and coherent structure.
• Value: ⭐⭐⭐⭐⭐ Provides a viable, hardware-efficient architecture for streaming 4D reconstruction and embodied world modeling.