Attention-based Vision-Language Memory for Spatial Reasoning

Conference: ECCV2026
Paper: Official paper page · PDF
Project: https://sairlab.org/vlm2/
Area: Vision-Language Reasoning / Video Spatial Reasoning
Keywords: semantic-geometric alignment, viewpoint consistency, adaptive position injection, working memory, episodic memory

TL;DR

VLM² aligns semantic features, predicted geometry, and viewpoint information from monocular video before using sliding-window working memory and fixed-capacity episodic memory to retrieve, fuse, and update scene representations, reaching 68.8 on VSI-Bench with video-only inputs, 7.9 score points above VLM-3R.

Background & Motivation

Counting chairs in a room looks like a recognition problem until the camera starts moving. The same chair appears from different angles and may temporarily leave the image. A model must avoid counting it twice while remembering chairs that are no longer visible. A conventional visual encoder can recognize a chair without reliably grounding its location in the scene; retaining several video frames in context does not necessarily create a spatial memory that can be updated consistently.

Adding a geometry foundation model does not automatically resolve these problems. Geometry tokens describe structure, but similar local patterns can remain ambiguous across viewpoints. Simply concatenating geometric and semantic features may disrupt an otherwise useful representation. Another approach converts ground-truth depth and camera parameters into positions and injects them into visual tokens, but this depends on reliable additional inputs. Replacing those coordinates with predictions introduces a different risk: errors in unreliable regions can contaminate semantic features. The challenge is therefore not merely to add 3D information, but to establish useful correspondences among semantics, position, and viewpoint first.

Even reliable per-frame representations will be forgotten when they leave a recent-context window, while retaining every observation increases storage and retrieval costs. This paper connects the two problems: construct viewpoint-consistent, 3D-aware representations, then preserve and retrieve them through bounded memory. Core idea: use semantic-geometric representations with gated position cues and viewpoint alignment as memory units, retaining both recent detail and history beyond the current window instead of simply stacking more video tokens.

Method

Overall Architecture

The inputs are a monocular video and a language question; the output is a language answer. After uniform frame sampling, a frozen visual encoder extracts semantic tokens, while the frozen geometry foundation model π3 produces geometry tokens, view tokens, and per-pixel 3D point maps. Adaptive 3D Position Injection and Viewpoint-Aware Geometry Alignment operate on the two branches, followed by Semantic-Geometric Fusion to construct a 3D-aware representation for each frame.

Bounded Dual Memory then uses the current representation to retrieve relevant information from both a recent window and a longer-term memory bank. Gated fusion combines the retrieved features and supports episodic-memory updates. Memory-enhanced representations and question embeddings enter the LLaVA-Video language backbone to generate an answer. Here, video-only input means that inference requires no additional ground-truth depth or point cloud; it does not mean the system avoids 3D priors learned by its pretrained geometry model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Monocular video frames"] --> Enc["Frozen visual encoder and π3"]
    Enc --> Pos["Adaptive 3D Position Injection"]
    Enc --> View["Viewpoint-Aware Geometry Alignment"]
    Pos --> Fusion["Semantic-Geometric Fusion"]
    View --> Fusion
    Fusion --> Memory["Bounded Dual Memory<br/>Dual retrieval, gating, and updates"]
    Memory -->|Updated memory for the next step| Memory
    Memory --> Answer["Language backbone generates answer"]
    Question["Language question"] --> Answer

Key Designs

1. Adaptive 3D Position Injection: ground semantics without unconditionally accepting prediction noise

Visual tokens primarily describe appearance and category. The model first pools the per-pixel point maps predicted by π3 into 3D coordinates corresponding to visual patches. Sinusoidal position encoding and a two-layer MLP map these coordinates into the visual feature dimension. Each patch therefore receives its own position cue, rather than an unrelated point cloud with no established correspondence to the semantic sequence. Position information is added through a residual path, preserving the original visual content.

The important detail is that a learnable gate controls the injection strength. The model can retain position cues useful for answering questions and suppress unreliable or irrelevant regions. The paper uses chairs versus walls to illustrate usefulness, but specifies neither separate confidence supervision nor the complete gate network. This should not be described as a confidence estimator trained with geometric-error labels: whether to use a geometric cue is learned through QA training. In Table 7, removing adaptivity yields 58.9, while gated injection reaches 61.6, supporting the conclusion that more coordinates are not automatically better.

2. Viewpoint-Aware Geometry Alignment: supply local viewing cues and global frame context together

Geometric similarity does not establish either instance identity or viewing direction: the front and back legs of a chair can produce similar structural features. The model first linearly projects each patch's view token into the geometry feature dimension, concatenates it with the corresponding geometry token, and passes the result through an MLP. This is not explicit instance matching. Instead, the geometry tokens themselves gain information that can reduce ambiguity about how a local structure is being observed.

Local cues alone do not describe the overall viewpoint of a frame. The model therefore also pools the view tokens, projects the result, and appends a global view descriptor to the geometric sequence. With \(hw\) original patches, this branch now contains \(hw+1\) tokens. Local viewpoint information and the global descriptor jointly provide context for subsequent attention. No explicit cross-frame correspondence loss is specified here, so viewpoint consistency should be understood as a design objective supported by experiments, not a guarantee that the same object's features become identical across views.

3. Semantic-Geometric Fusion: position-aware semantics query viewpoint-aware structure

After the two branches have been aligned, position-enhanced visual tokens serve as queries and viewpoint-enhanced geometry tokens serve as keys and values in cross-attention. The query side now conveys what was observed and where it may be, while the retrieved side supplies structure and viewing context. This avoids forcing two differently distributed feature streams directly into a single concatenated vector. The output \(H_t\) is the per-frame 3D-aware representation subsequently used by memory.

The cached equations contain extraction damage. The following expression only normalizes the query, key, and value relationship explicitly described in Section 3.1; it does not invent attention-head counts, projection dimensions, or residual connections:

\[ H_t=\operatorname{Attn}\!\left(Q(F_t^{\mathrm{pa}}),K(G_t^{\mathrm{vc}}),V(G_t^{\mathrm{vc}})\right). \]

Here, \(F_t^{\mathrm{pa}}\) denotes position-enhanced visual tokens and \(G_t^{\mathrm{vc}}\) denotes viewpoint-enhanced geometry tokens. The geometry backbone alone does not determine the outcome. With π3 held fixed, concatenation followed by an MLP scores 40.4, cross-attention scores 61.0, and adding both alignment designs reaches 63.8. The fusion interface can therefore determine whether geometric information helps or interferes.

4. Bounded Dual Memory: retrieve recent detail and earlier history separately, then refresh redundant entries

Working memory stores the most recent \(L_w\) representations and advances as a sliding window. The current \(H_t\) queries that window through cross-attention, producing \(M_t^w\). Episodic memory maintains a fixed-capacity bank of \(L_e\) entries and is also queried by \(H_t\), producing \(M_t^e\). Both retrieval paths are driven by the current visual representation. The paper does not directly feed the language question into these queries, so the mechanism should not be mistaken for explicit question-conditioned retrieval of historical frames. Task relevance is learned primarily through end-to-end QA training.

The retrieved features are not simply added together. Their concatenation passes through an MLP and sigmoid to produce a gate that controls the relative contributions of recent and episodic information. Normalizing the prose and legible equation fragments in Section 3.2 gives:

\[ \gamma_t=\sigma\!\left(\operatorname{MLP}([M_t^w;M_t^e])\right),\qquad M_t=\gamma_t\odot M_t^w+(1-\gamma_t)\odot M_t^e. \]

Rather than continually appending entries, the model replaces the episodic entry with the highest cosine similarity to the current fused representation \(M_t\). Intuitively, a new observation refreshes similar history while leaving different content in the bank. The replacement target is the most similar entry, not the least similar or oldest one. The cache does not explain empty-bank initialization or how sequence-valued representations are reduced for cosine similarity, so these details cannot be expanded into a complete algorithm. Fixed capacity bounds the memory bank and associated retrieval costs, but does not make video encoding and language generation independent of total input length.

A Worked Example

Consider the paper's chair-counting scenario: the camera first sees a chair near the entrance, moves deeper into the room, and later returns. On the first observation, position injection adds spatial cues to chair-related patches, viewpoint alignment supplies viewing context, and cross-attention constructs a representation suitable for memory retrieval. As the camera turns, working memory provides recent viewpoint context. Once the entrance chair falls outside the recent window, episodic memory may still retain its associated representation.

In the paper's best setting, \(L_w=8,L_e=32\): the recent window retains 8 representations and the episodic bank has capacity 32. Upon revisiting the entrance, the model retrieves from both memories, fuses the results, updates the most similar episodic entry, and lets the language backbone answer the question. This illustrates the mechanism rather than reproducing a reported test trajectory with intermediate predictions. There are no explicit chair IDs or symbolic counters, so the architecture does not guarantee that every revisit avoids double counting.

Loss & Training

The main model uses LLaVA-Video-7B with a Qwen2-7B language backbone and π3 as the geometry foundation model. Both the visual encoder and geometry model are frozen. The language model is fine-tuned with LoRA, using rank 128 and scaling factor 256. Training runs for 1 epoch on a mixed dataset with AdamW and batch size 128. The text reports an LLM peak learning rate of \(10^{-5}\) during warmup and uses 8 NVIDIA H200 GPUs.

The paper states that it follows VLM-3R's learning objective, but the cache does not enumerate the complete data mixture, objective terms, frame-sampling count, or loss weights. This note does not substitute a conventional autoregressive cross-entropy formula for an objective explicitly specified by the paper. The component-ablation section separately names LLaVA-NeXT-Video-7B as its baseline and states that the language backbone, training schedule, and compute budget are controlled. This differs from the main-model naming and should remain explicit when interpreting those ablations.

Key Experimental Results

Main Results

VSI-Bench contains over 5,000 QA pairs and VSTI-Bench approximately 6,000, drawn from indoor egocentric video. Multiple-choice tasks use Accuracy, while numerical tasks use Mean Relative Accuracy. The latter evaluates numerical answers through relative error, but the cache does not spell out its thresholds or aggregation rule; the metric name is retained without inventing a formula. The reported averages mix task-specific scores and should not all be called ordinary classification accuracy.

Benchmark / metric	Comparator	Comparator score	VLM²-7B	Score difference
VSI-Bench Avg., Table 1	VLM-3R-7B	60.9	68.8	+7.9
VSTI-Bench Avg., Table 2	VLM-3R-7B	58.8	65.3	+6.5
ScanQA val CIDEr, Table 3	Spatial-MLLM-4B	91.8	105.5	+13.7
ScanQA val BLEU-4, Table 3	Spatial-MLLM-4B	14.8	17.7	+2.9
SQA3D test EM-1, Table 3	Spatial-MLLM-4B	55.9	60.4	+4.5

These results support an advantage among the listed video-input models, not superiority over every method with additional inputs. Ross3D, which uses additional 3D information, reaches 107.0 ScanQA CIDEr and 63.0 SQA3D EM-1, both higher than this method. Nor does VLM² outperform every general-purpose model on every VSI-Bench subtask: its object-size score is 70.8, compared with 76.3 for Qwen3-VL-8B.

Ablation Study

The following table retains the Avg. column from the paper's Table 6, together with absolute distance, relative direction, and route planning, to distinguish the contributions of representation and memory.

3D-aware representation	Working memory	Episodic memory	Avg.	Absolute distance	Relative direction	Route planning
No	No	No	55.2	43.3	67.8	40.2
Yes	No	No	63.8	52.9	84.8	48.5
Yes	Yes	No	65.9	56.3	86.2	51.3
Yes	No	Yes	66.1	57.7	85.9	50.5
Yes	Yes	Yes	67.8	59.6	87.8	52.6

The 3D-aware representation first adds 8.6 score points. Working memory or episodic memory alone adds another 2.1 or 2.3, while full dual memory adds 4.0 over the representation-only variant. Route planning improves from 40.2 to 52.6, but the representation already contributes 8.3 of that gain; attributing the entire 12.4 improvement to memory would be incorrect. Table 6 reports 67.8 for the full model, unlike the 68.8 in main Table 1. The cache does not clearly explain the aggregation difference, so the values are neither merged nor silently corrected here.

Geometry Interface Analysis

The next table selects π3 configurations from the paper's Table 7. All rows except the first use cross-attention. A3PI denotes Adaptive 3D Position Injection, and VAGA denotes Viewpoint-Aware Geometry Alignment.

Configuration	Avg.	What the comparison tests
π3 + Concat-MLP	40.4	Direct concatenation can harm the original baseline
π3 + Cross-Attn	61.0	Whether a better fusion interface can use geometry
Position injection without adaptive gating	58.9	Whether unconditional coordinates introduce interference
Add A3PI	61.6	Whether gating improves coordinate use
Add VAGA	62.9	Whether viewpoint cues reduce geometric ambiguity
Add A3PI + VAGA	63.8	Whether the two alignment mechanisms complement each other

Table 8 further analyzes memory capacity. With \(L_w=8\), increasing \(L_e\) through 8, 16, and 32 gives averages of 66.3, 67.3, and 68.8. Keeping \(L_e=32\) but increasing \(L_w\) from 8 to 16 instead reduces the score to 68.3. A longer recent window is not necessarily better. The result is consistent with irrelevant recent tokens diluting attention, but is not direct causal proof of that explanation.

Key Findings

Long-video gains fit the motivation for memory. In Table 5, on videos longer than 2 minutes, VSI-Bench improves from VLM-3R's 60.0 to 68.1, and VSTI-Bench from 66.4 to 74.6. However, this table reports averages of 69.4 / 69.5, whereas the main tables report 68.8 / 65.3. Aggregations from different tables should not be mixed without clarification.

Zero-shot results are also not uniformly best. Table 4 reports 66.8 on SPAR-Bench and 67.2 on BLINK, but the CV-Bench average of 79.4 is below VG-LLM-8B's 81.7. The defensible conclusion is that the representation transfers, with benefits that remain task-dependent.

Highlights & Insights

• Alignment precedes fusion. The semantic branch gains position, the geometry branch gains viewpoint, and only then does cross-modal retrieval occur. This reframes whether geometry helps as a question about the interface through which it enters semantic representations, with Table 7 providing both a counterexample and supporting evidence.
• Memory stores representations that have already been aligned. Dual memory is not merely an isolated cache added to a video model; it operates on more reliable 3D-aware tokens. A transferable lesson for embodied QA is to examine cross-view stability of memory units before increasing capacity.
• Replacing the most similar entry is a useful design choice to examine. Instead of automatically evicting the oldest content, this strategy refreshes redundant content to preserve diversity. It offers a simple bounded spatial-memory policy, although its suitability for dynamic scenes still needs separate validation.

Limitations & Future Work

• Reproduction details are incomplete. The cache does not spell out the gate architecture, memory initialization, sequence reduction for similarity, training mixture, or complete loss details. The original implementation or supplementary material is needed; the main text alone does not establish a full training recipe.
• Tables and claims need clearer qualification. Main results, component ablations, and duration analysis report different averages, while some broad superiority claims are limited by subtask counterexamples. Per-question predictions, grouping weights, and a unified evaluation script would help resolve these issues.
• Bounded memory does not establish measured end-to-end efficiency. The text provides no latency, throughput, or GPU-memory comparison, and does not show that the geometry frontend processes arbitrarily long video at fixed cost. Memory-specific cost and complete inference-pipeline cost should be measured separately.
• Long-term updates may confuse similar instances or changing states. This is an inference in this note, not an author-validated failure case: if similar objects occupy the same feature neighborhood, replacing the most similar entry may overwrite useful history. Spatial consistency constraints or temporal tags require testing on dynamic scenes and long-range revisits.
• Input and task coverage remain limited. Evidence mainly comes from indoor video QA and does not directly establish reliability outdoors, in strongly dynamic environments, or in closed-loop robotics. The reported tables also lack multi-seed variance and error bars.

Related Work & Insights

• Compared with Video-3D LLM: Both enhance visual representations with position, but VLM² predicts geometry from video and gates coordinate injection to reduce noise, lowering dependence on ground-truth depth inputs. It nevertheless remains limited by geometric prediction quality.
• Compared with VG-LLM, Spatial-MLLM, and VLM-3R: All exploit geometry foundation-model priors. VLM² emphasizes semantic-position alignment and geometry-viewpoint alignment before fusion, followed by persistent memory. Main-table gains cannot all be attributed to memory; the component ablations are needed to interpret them.
• Compared with 3DLLM-Mem and CUT3R: The former represents long-term memory for spatial tasks, while the latter motivates continuously updated state. VLM² combines a sliding window with a fixed-capacity episodic bank, but the discussion of prior memory growth is not a matched-budget efficiency experiment.
• Resource boundary: The official page, PDF, and project address come from the supplied manifest and cached paper. Code-release status was not checked online, so the project page is not labeled as a verified open-source repository.

Rating

• Novelty: 4/5. Combining semantic position, geometric viewpoint, and bounded dual memory targets a clear problem, although the underlying components remain gating, attention, and memory updates.
• Experimental Thoroughness: 4/5. Multiple spatial QA benchmarks, zero-shot transfer, and component ablations are covered, but efficiency measurements, statistical variation, and cross-table aggregation explanations are missing.
• Writing Quality: 3/5. The main argument and motivations are clear, while implementation details and some result claims need clarification; damaged cached equation formatting also complicates verification.
• Value: 4/5. The representation and memory interfaces are useful for video-only spatial reasoning, but long-term robotic deployment requires further evidence.