Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training¶
Conference: ECCV 2026
Paper: ECCV Official Page
Area: VLM Efficiency
Keywords: test-time training, streaming spatial understanding, fast-weight memory, spatiotemporal convolution, dense scene descriptions
TL;DR¶
Spatial-TTT combines online fast-weight updates, local sliding-window attention, and a small set of full-attention layers, using dense 3D scene descriptions to teach spatial evidence retention; its 2B model scores 64.4 on VSI-Bench while using less memory and computation than Qwen3-VL-2B at 1024 input frames.
Background & Motivation¶
Understanding a room is not the same as recognizing furniture in one frame. As a camera moves, objects reappear or become occluded; answering where the door is relative to a table or how many chairs exist requires organizing observations over time into consistent spatial memory. Most vision-language models (VLMs) learn semantics from 2D image-text data. Spatial question-answering fine-tuning can improve short-video performance without necessarily teaching persistent object identity, layout, and relative position in long streams.
Simply enlarging the context increases attention and KV-cache costs, while sparse frame sampling may discard the observations needed to associate object instances or recover spatial relationships. Test-time training (TTT) offers a different storage mechanism: write history into a small set of adaptable parameters instead of retaining every historical token. However, replacing all pretrained attention layers can damage cross-modal alignment, and writing isolated tokens into fast weights does not specifically encourage geometric memory. Short question-answering targets also supervise only a small part of a scene, leaving the memory module weakly informed about what should persist.
The paper therefore changes the memory architecture, the inputs to its updates, and the supervision together. It retains some full-attention layers, introduces local spatiotemporal context into fast-weight updates, and trains memory behavior using comprehensive scene descriptions rather than short answers alone. Core Idea: learn to write spatial evidence from successive observations into online-adaptable nonlinear memory while preserving fine-grained local interactions and pretrained semantic reasoning.
Method¶
Overall Architecture¶
The model is initialized from Qwen3-VL-2B-Instruct, takes video frames and a question, and generates a spatial description or an answer. During training, "Dense Supervision and Curriculum Training" shapes memory behavior; at inference time, the "Hybrid Memory Architecture" retains cross-chunk information in fast weights, while "Spatial-Predictive Updates" condition memory writing and reading on local spatiotemporal context rather than isolated visual tokens.
There are two distinct meanings of training here. Offline training learns the backbone, new modules, and fast-weight update behavior. Online adaptation updates fast weights while processing the current video, using key-value pairs derived from the input itself rather than correct answers to test questions. Dense descriptions are offline supervision, not additional ground-truth scene information supplied at test time.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
D["Dense Supervision and<br/>Curriculum Training"] -.->|Offline training only| H["Hybrid Memory<br/>Architecture"]
I["Video frames and question"] --> H
H -->|TTT branch Q/K/V| P["Spatial-Predictive<br/>Updates"]
P -->|Write completed chunk| M["Fast-weight memory"]
M -->|Read previous-chunk memory| P
H -->|Sliding-window and full-attention paths| O["Branch fusion and subsequent decoding<br/>Spatial description or answer"]
P --> O
The memory loop represents state updates as the video advances, not permission for early tokens in a chunk to read future tokens. Full-attention anchor layers remain part of the hybrid decoder, so the diagram should not be interpreted as a wholly fixed-state recurrent network.
Key Designs¶
1. Dense Supervision and Curriculum Training: retain the scene before learning question-specific retrieval
Ordinary spatial questions often require only an option, an integer, or a relation between two objects, supervising a narrow slice of the scene. The authors derive coherent descriptions from SceneVerse object-centric 3D scene graphs, covering scene type and function, object categories and precise counts, and layouts and pairwise relations. A model that loses earlier objects or relationships becomes less able to produce the complete description. Offline learning is therefore encouraged to preserve evidence useful for future questions rather than merely solve the current local query. The first stage has approximately 16k samples, including approximately 3.6k ScanNet descriptions and 12.5k ARKitScenes descriptions; these are approximate counts.
The attention window also shrinks gradually instead of immediately assigning all cross-chunk communication to newly introduced fast weights. In the first stage, the window is linearly annealed from 5600 to the chunk size of 2648. As direct local attention covers less history, fast weights progressively assume more responsibility for long-range information. The second stage uses spatial VQA to train relative direction, distance, counting, route planning, and room-scale reasoning, keeping the window equal to the chunk size. This is language supervision of implicit memory, not training an explicit 3D reconstructor and passing its map to a language model.
2. Hybrid Memory Architecture: fast weights handle cross-chunk history while attention preserves local interaction and semantics
Three of every four decoder attention layers become TTT layers, while one remains a full self-attention anchor, giving a 3:1 ratio. Anchors retain access to the entire context to reduce damage to pretrained cross-modal capabilities. Each TTT layer contains parallel sliding-window attention (SWA) and fast-weight branches sharing the original Q/K/V projections, with additional learnable scale and shift for TTT queries and keys. The fast-weight network is a bias-free SwiGLU-MLP: keys train a mapping to values, and queries retrieve historical information from the small network as it changes with the video. This stores compressed associations rather than a losslessly recoverable copy of every frame.
Compared with updates every 16 or 64 tokens, larger chunks improve parallelism and reduce arbitrary fragmentation of visual structure. They also require causal discipline: the complete current chunk cannot be written into memory and then exposed to its earliest tokens, since that would reveal later content. TTT handles previous chunks, while parallel SWA restores valid within-chunk interactions; its window satisfies \(w\geq b\) to cover the chunk's causal lower triangle. At inference, the SWA cache retains the latest \(w\) tokens, and a separate pending KV cache accumulates \(b\) tokens before one fast-weight update and a reset. These are local mechanisms in TTT layers, not a description of the full-attention anchors' caches.
3. Spatial-Predictive Updates: write neighborhood context rather than isolated tokens
Point-wise Q/K/V projections do not directly exploit the spatial adjacency and temporal continuity of video tokens. The method reshapes visual Q/K/V in the TTT branch into a time-height-width grid and applies \(3\times3\times3\) depthwise convolutions, aggregating local spatiotemporal neighborhoods independently per channel. Predictive key-value associations then include nearby regions and observations, and queries carry corresponding context. "Spatial-predictive" does not mean an auxiliary task that predicts a future RGB image; it means learning mappings between spatiotemporal contexts inside fast weights to better preserve geometric correspondence and continuity.
Dirac initialization makes the convolutions initially behave like identity mappings, avoiding an abrupt change to pretrained features. Online optimization uses momentum-based Muon updates, orthogonalizing gradients with Newton-Schulz iterations and applying L2 weight normalization that preserves the original magnitude. The 3D convolution supplies a local inductive bias, not explicit 3D coordinates, camera poses, or a global map; long-range organization still depends on memory dynamics and offline supervision. Equations (1) and (3) through (7) in the supplied text contain missing brackets, operators, or symbols, so this note does not reconstruct the exact loss, branch-fusion expression, or Muon recurrence.
A Worked Example¶
Imagine a camera first observing a table and two chairs, turning toward a doorway, and then returning to the table. A user asks for the total number of chairs and the direction from the table to the door. This illustrates the mechanism; it is not a separately reported experimental case.
During the first observation, visual tokens enter the hybrid decoder. Local attention handles nearby interactions, while spatiotemporal convolution in the TTT branch produces context-enriched keys and values. Once the pending cache reaches 2648 tokens, the completed chunk is compressed into fast weights and the cache is cleared. The chunk size counts tokens, not 2648 video frames.
After the camera turns toward the door, earlier table-and-chair tokens eventually leave the TTT layers' sliding windows, but fast weights can still supply compressed historical evidence to later queries. When a chair reappears, the model must combine local observations with historical associations to recognize a previously seen instance rather than simply increment a detection count. Object-count and relation descriptions during training are intended to encourage this persistence.
The final answer is generated by the hybrid decoder, not by deterministic path search over an explicit map. The mechanism offers a way to retain spatial evidence, but it does not guarantee correct counting or navigation over arbitrarily long videos; the reported counting and recall scores are also far from perfect.
Loss & Training¶
The online inner loop uses input-derived key-value associations as a self-supervised signal. Offline dense-description training and subsequent spatial VQA shape the backbone and update behavior. The readable main text does not fully specify the exact inner-loop loss or the weighting of outer-loop objectives, so it would be unjustified to invent a mean-squared-error loss or a weighted joint objective.
- First stage: uniformly sample 32 frames, anneal the window from 5600 to 2648 over the first two epochs, and keep the chunk size at 2648.
- Second stage: increase input length from 64 to 128 frames while keeping \(w=b=2648\).
- Use a backbone learning rate of \(10^{-6}\) and a new-TTT-parameter learning rate of \(10^{-5}\), with 1k warmup steps and cosine scheduling.
- Initialize the new gating projection to zero, scales to one, and shifts to zero, with Dirac initialization for convolution. These choices reduce disruption when modifying the architecture.
- The source contains a data-size discrepancy: Section 3.4 reports 2M spatial VQA samples for the second stage, whereas Section 4.1 reports 3M. The supplied cache contains no supplementary material, so both statements are retained rather than selecting one as definitive.
Key Experimental Results¶
Main Results¶
VSI-Bench contains over 5,000 question-answer pairs from 288 indoor videos. Numerical questions use Mean Relative Accuracy (MRA), multiple-choice questions use accuracy (ACC), and the overall score is a macro average across tasks. MRA measures how closely numerical predictions match ground truth in relative terms; the available main text does not specify its threshold protocol. The following selected columns reproduce the scores from the paper's Table 1.
| Model | Object Count MRA | Absolute Distance MRA | Relative Direction ACC | Route Planning ACC | Overall Average |
|---|---|---|---|---|---|
| Qwen3-VL-2B-Instruct | 62.1 | 40.2 | 42.0 | 30.4 | 50.3 |
| Qwen3-VL-8B-Instruct | 67.5 | 47.0 | 50.9 | 35.0 | 57.9 |
| VST-7B-SFT | 72.0 | 44.4 | 55.8 | 44.9 | 60.6 |
| Cambrian-S-3B | 70.7 | 40.6 | 61.9 | 27.3 | 57.3 |
| Spatial-TTT-2B | 70.8 | 47.8 | 73.0 | 47.4 | 64.4 |
Spatial-TTT exceeds its initialization backbone by 14.1 points and VST-7B-SFT by 3.8 points. Relative Direction reaches 73.0, but Object Count remains below VST's 72.0, so the model is not best on every subtask. Training data and model sizes differ across methods; this table compares final systems and does not attribute the entire 14.1-point gain to TTT alone.
Table 2 reports 76.2 overall accuracy on MindCube-Tiny, 12.3 percentage points above Gemini-3-pro's 63.9. This set contains 1,050 questions and uses micro averaging across tasks. Spatial-TTT scores 74.0, 89.8, and 55.5 on Among, Around, and Rotation, respectively. Rotation is clearly not its strongest category and is far below GPT-5's 94.5.
Ablation Study¶
The following results are from Table 4 on VSI-Bench. Removing the hybrid architecture means using pure TTT, not reverting to a full-attention-only model.
| Config | Numerical | Multiple-Choice | Overall Average | Change from Full Model |
|---|---|---|---|---|
| Full model | 64.0 | 64.8 | 64.4 | 0.0 |
| Without spatial-predictive mechanism | 60.7 | 63.4 | 62.1 | -2.3 |
| Without dense-description data | 61.0 | 61.5 | 61.3 | -3.1 |
| Without hybrid architecture | 55.4 | 52.4 | 53.9 | -10.5 |
Retaining full-attention anchors has the largest effect, with a particularly large 12.4-point change on multiple-choice questions. Spatial convolution and dense supervision also help, but these ablations do not isolate the independent contributions of SWA, large-chunk updates, Muon, or window annealing.
Key Findings¶
Table 5 reports peak memory and TFLOPs per forward pass at \(352\times480\) resolution. The following excerpt compares Spatial-TTT with its similarly sized backbone.
| Input Frames | Qwen3-VL-2B Memory GB | Spatial-TTT-2B Memory GB | Qwen3-VL-2B TFLOPs | Spatial-TTT-2B TFLOPs |
|---|---|---|---|---|
| 128 | 6.2 | 6.2 | 75.9 | 74.3 |
| 256 | 8.3 | 7.0 | 179.9 | 156.2 |
| 512 | 12.6 | 8.4 | 473.9 | 341.9 |
| 1024 | 21.2 | 11.9 | 1403.1 | 799.4 |
- At 1024 frames, memory is approximately 43.9% lower and TFLOPs approximately 43.0% lower. These are relative reductions calculated from the table, not measured latency speedups.
- Table 3 evaluates streaming tasks at 1 fps: at 120 minutes, VSI-SUPER-Recall reaches 30.0 ACC and VSI-SUPER-Count reaches 38.4 MRA. Counting substantially exceeds the listed alternatives, but recall is relatively stable rather than evidence of near-perfect long-term memory.
- Table 3 records some OOM cases as 0. Its caption says "longer than 120 minutes," although the 120-minute column already contains zeros, leaving a threshold inconsistency. Such scores partly reflect whether a model can run, not just accuracy after successful inference.
Highlights & Insights¶
- The coverage of memory supervision matters. Dense descriptions jointly require global context, instances, and relations, directly constraining what must be retained rather than merely increasing the number of short questions.
- Fast weights and attention have different responsibilities. The substantial pure-TTT ablation loss shows that history compression does not automatically replace pretrained semantics and global reasoning.
- Geometric bias is introduced at memory writing and reading. Local 3D convolution gives the update access to spatiotemporal neighborhoods without first constructing an explicit map, suggesting a useful intervention point for other video-memory systems.
Limitations & Future Work¶
- Constant memory for the entire model is not established. Section 3.4's bounded dual caches concern the TTT/SWA paths, whereas anchors retain full attention; Table 5 also shows memory increasing from 6.2 GB to 11.9 GB. Linear complexity in one branch cannot simply be assigned to the complete hybrid network.
- The paper reports computation and memory, not end-to-end streaming latency, update throughput, or sustained power consumption. Deployment evaluation should include online optimization overhead and clarify the full-attention anchors' caching policy for very long streams.
- The ablations do not separately evaluate Muon, window annealing, chunk size, anchor ratio, or long-term forgetting. Repeated objects, changing scenes, and dynamic objects deserve targeted tests for memory interference.
- The experiments focus mainly on indoor spaces and controlled spatial reasoning, not reliable closed-loop robot navigation. The 2M/3M discrepancy, damaged equations, and unavailable supplementary material also limit reconstruction of all training details from the supplied main text.
Related Work & Insights¶
- vs LaCT: The method inherits large-chunk fast-weight updates for hardware efficiency and adds video-neighborhood-aware spatial prediction and scene-level supervision. It does not originate the large-chunk TTT idea.
- vs Spatial-MLLM / VLM-3R: These approaches introduce explicit geometry encoders or reconstruction features, whereas Spatial-TTT injects spatiotemporal bias into memory updates. It reduces dependence on an additional geometry encoder but does not directly provide an inspectable 3D map.
- vs VST / Cambrian-S: These methods also address spatial training and long-video evidence, while Spatial-TTT emphasizes continuously updated parametric memory at test time. A concrete extension is to compare parametric and explicit instance memory for repeated-object counting under matched spatial data and inference budgets.
- Resources: The project page listed in the paper can support later verification of implementation and supplementary material; no code repository URL is inferred here.
Rating¶
- Novelty: 4/5. A coherent combination of fast weights, spatial neighborhoods, and dense supervision, with clear antecedents for hybrid TTT and large-chunk updates.
- Experimental Thoroughness: 4/5. Covers spatial QA, long-stream sensing, ablations, and resource usage, but lacks online latency and finer-grained update-strategy ablations.
- Writing Quality: 3/5. The main argument is clear, but training-data counts and constant-memory claims need clarification.
- Value: 4/5. A promising direction for compact models that accumulate spatial evidence, not yet equivalent to reliable memory over an unlimited time horizon.