EchoVLA: Robotic Vision-Language-Action Model with Synergistic Declarative Memory for Mobile Manipulation¶

Conference: ECCV2026
Paper: Official ECCV Page
Area: Robotics & Embodied AI
Keywords: mobile manipulation, declarative memory, scene memory, episodic memory, diffusion policy

TL;DR¶

EchoVLA separately maintains voxel scene memory for what exists where and episodic memory for recent interactions, retrieves both at different granularities to condition base and arm diffusion policies, and achieves 0.31 mean success on RoboCasa mobile manipulation versus 0.20 for π0.5 in the same table.

Background & Motivation¶

In tabletop manipulation, the current image often provides enough information to decide where the arm should move next. Mobile manipulation extends this process: the robot must approach a workspace, manipulate an object, and potentially leave or move between rooms. Viewpoint changes temporarily hide targets, while base motion changes the arm's position relative to them. Even similar-looking observations can require different actions depending on whether an object has not yet been grasped or has already been picked up for transport. A vision-language-action model (VLA) therefore needs reusable spatial structure and ordered interaction history, not just recognition of visible objects.

Simply stacking past frames does not automatically satisfy both requirements. Environment layout changes slowly and benefits from accumulation across interactions, whereas task progress changes with each action and quickly becomes stale. Mixing them in one cache makes persistent geometry and transient action cues compete for capacity. The paper draws on the spatial and experiential distinction in declarative memory, but its implementation is not a precise simulation of brain regions: one bank stores voxel features indexed by 3D coordinates, while another maintains a bounded queue of recent multimodal states. Compared with a generic perceptual cache, this separation explicitly specifies storage format, update timing, and retrieval targets.

EchoVLA aims to prevent a robot from forgetting its surroundings after turning or repeating completed subtasks after seeing a familiar scene. The authors also build the MoMani data pipeline to generate and filter demonstrations requiring coordinated base and arm motion, rather than learning mobile control solely from stationary tabletop trajectories. Core Idea: store spatial context in slowly updated 3D scene memory and recent task progress in rapidly updated, time-indexed episodic memory, then use their retrieved context to condition both base and arm action generation.

Method¶

Overall Architecture¶

At each decision step, the model receives a natural-language instruction, multi-view RGB-D observations, and proprioceptive robot states, and produces continuous base and arm actions. Instead of feeding the current observation directly to a history-free policy, it encodes the inputs, updates and queries two memory banks, and supplies retrieved context together with the current state to two action-denoising branches.

A frozen SigLIP text tower encodes the instruction, while its frozen vision tower independently processes RGB images from three views; visual features are then concatenated and projected into a shared embedding space. Depth observations are fused into a point cloud and processed by a trainable PointAttn backbone. A small MLP converts proprioception into configuration tokens. Language, visual, point-cloud, and proprioceptive tokens form the current multimodal sequence, retaining both the requested goal and the present relationship between the robot and objects.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["Instruction, RGB-D<br/>Proprioception"] --> ENCODE["Multimodal encoding"]
    ENCODE --> SCENE["Scene Memory<br/>Voxel features and slow updates"]
    ENCODE --> EPISODE["Episodic Memory<br/>Historical tokens and fast updates"]
    SCENE --> RETRIEVE["Coarse- and Fine-Grained<br/>Memory Retrieval"]
    EPISODE --> RETRIEVE
    ENCODE --> RETRIEVE
    RETRIEVE --> POLICY["Per-Part Diffusion<br/>Action Generation"]
    POLICY --> ACTION["Base actions + Arm actions"]
    DATA["MoMani demonstrations<br/>Training supervision only"] -.-> POLICY

Scene and episodic memory are parallel information sources, not a sequence that first turns the map into text and then writes that text into history. MoMani belongs to offline training: its planner does not plan each deployed action online. At runtime, actions come from the memory-conditioned policy.

Key Designs¶

1. Scene Memory: preserve spatial structure through change-driven voxel updates

Scene memory uses 3D coordinates as keys and aggregated PointAttn features as values to maintain a persistent, environment-specific voxel representation. The map starts as an empty grid in a new environment and is progressively populated through interaction, rather than reconstructing the entire layout from scratch at every step. New depth observations become local 3D feature volumes through PointAttn and are compared with existing memory decoded by an MLP in the corresponding region. Only regions whose reconstruction discrepancy exceeds a threshold are updated; sufficiently unchanged regions retain their values. This avoids writing every sensor fluctuation back into the map as an environmental change.

The paper uses an update threshold of 0.5 and an exponential moving average coefficient of 0.2. The former is a feature reconstruction-error threshold, not a distance in meters; the latter controls the contribution of new observations, not retention of 20% of the voxels. Although the global map accumulates spatial information over time, downstream retrieval selects relevant entries within the current local camera frustum to avoid reading the entire map. Persistent global memory therefore does not imply unrestricted access to every distant object at every step. Its benefit depends on local visibility, consistent localization, and map coverage.

2. Episodic Memory: retain recent encoded states rather than abstract task summaries

Episodic memory stores historical multimodal token sequences with time indices in a fixed-capacity, first-in-first-out buffer. It preserves encoded state details rather than language-generated summaries such as a statement that the first step is complete. Recent end-effector configurations, whether an object has been grasped, and whether a drawer has just opened can therefore remain available to subsequent queries. Here, original tokens means encoded tokens without additional summary compression, not lossless storage of camera pixels.

This branch primarily resolves ambiguity between visually similar states at different stages of a task. Its distinction from the scene map concerns not only input modalities but also lifetime: the map accumulates stable structure, whereas the queue deliberately discards old experiences. The sensitivity study selects a window length of 8. This is bounded recent history, not evidence of a complete task record or a database of experiences across tasks. Time indices are stored in the buffer, but the paper does not fully detail their encoding within attention computations.

3. Coarse- and Fine-Grained Memory Retrieval: select relevant records before reading them through cross-attention

Both retrieval paths separate similarity matching from cross-attention interaction. The scene branch uses the current voxelized 3D features as its query, selects relevant scene memory through cosine similarity, and applies coarse-grained cross-attention. The episodic branch queries historical states with the current multimodal state tokens, selects a relevant subset, and applies fine-grained cross-attention. Coarse-to-fine thus primarily refers to the semantic granularity of the two memories, not a requirement to produce a coarse action first and then search for a finer action nearby.

Retrieval before attention has a concrete purpose: not every map entry or buffered state is relevant to the current instruction and action stage. Passing all stored information to the policy increases cost and may introduce stale context. Scene features provide a spatial reference, while episodic features provide a recent-interaction reference; their outputs combine with the current state to condition the policy. The prose establishes this role, but the cached equations contain substantial extraction damage. This note therefore does not force the fusion into a specific concatenation, addition, or gating operator, or invent a retrieval count.

4. Per-Part Diffusion Action Generation: share context while learning distinct motion spaces

Base translation and rotation differ from contact-rich arm manipulation in their dynamics and precision requirements. EchoVLA uses independent diffusion denoising processes for these two action spaces. Each branch reads the same memory-augmented context but generates its own action subspace. Shared conditioning gives them a common spatial and task reference, while separate action modeling avoids forcing a single output head to represent both motion types in exactly the same way.

During training, each branch receives noisy demonstration actions, predicts the injected noise, and minimizes squared noise-prediction error. The overall objective adds the base and arm denoising losses. At inference time, demonstration actions are unavailable; the model generates actions through iterative denoising conditioned on current observations and retrieved memory. This architecture supports coordinated control, but shared context is not a collision-safety guarantee. The paper does not provide a theoretical constraint ensuring that the independently generated branches always satisfy joint feasibility.

A Worked Example¶

Consider an illustrative instruction to move an object from a counter to a location by the stove. Starting at a distance, the robot forms its current encoding from RGB-D observations and proprioception. Scene memory gradually retains the counter, stove, and surrounding geometry, while episodic memory records recent observations and robot configurations. This example explains the mechanism; it is not an additional quantitatively evaluated rollout from the paper.

As the base approaches the counter and the arm begins grasping, the map branch continues to provide a spatial reference for the target area. The history branch helps distinguish approaching the object from already holding it. Seeing the counter again does not imply that grasping should restart: recent states provide evidence that the action stage has changed. Retrieved context enters the two denoising branches, which generate the required base movement and arm motion.

During placement, new states continue to enter the length-8 queue while old states are removed, and sufficiently changed voxel regions are updated according to the threshold. Neither memory is a static attachment: one maintains spatial continuity, the other recent temporal continuity. If pose drift writes an object into incorrect coordinates, historical cues may mitigate the resulting decision error, but the mechanism does not guarantee that they repair the map.

Loss & Training¶

Training uses the two-branch denoising objective described above, with frozen SigLIP visual and text encoders and a trainable PointAttn backbone. The authors report using 8 NVIDIA A100 GPUs with multi-view RGB-D observations and robot states. The main text does not sufficiently specify parameter count, optimizer, learning rate, training duration, diffusion steps, or action-chunk length, so training cost and closed-loop control frequency cannot be inferred. Equations 1 through 10 in the cache contain missing or misordered content. This note retains only mechanisms and hyperparameters supported by the prose rather than presenting guessed repairs as the authors' equations.

MoMani first uses multimodal large language model guidance to generate simulation candidates through target-aligned sampling, safety-aware navigation, and continuous navigation-manipulation stitching, supporting simultaneous base and arm execution. Candidates must pass hard quality gates: zero collisions, position error below 0.05 meters, orientation error below 5 degrees, and task success. Feasible candidates are then ranked lexicographically by path length and planning cost, with a Top-K set retained for a scene-camera audit. The requirement of 100% task success is a filtering condition for accepted candidates, not a claim that the learned policy succeeds 100% of the time.

The dataset contains 7,889 simulation episodes and 1,200 real-world episodes. Pure navigation accounts for 57.0% of the simulation set; most remaining episodes cover four mobile manipulation tasks. Real demonstrations use a TidyBot++-style holonomic base and a Kinova Gen3 seven-degree-of-freedom arm, with teleoperation recorded at 30 Hz, followed by segmentation and replay verification; failed attempts are discarded. The 30 Hz rate describes demonstration collection, not reported EchoVLA inference speed. Although the paper describes generation and real-world collection, it does not sufficiently specify train/test scene splits or map reuse across episodes, which limits interpretation of generalization results.

Key Experimental Results¶

Main Results¶

Success rate (SR) is the fraction of successful task executions, expressed below on a 0-to-1 scale, with higher values being better. Simulation uses RoboCasa; the text reports three random seeds and 50 evaluation episodes per task. Real experiments use TidyBot++ in a 7-meter × 7-meter arena, with 20 trials per task and randomized initial base positions. The following summary combines original Tables 2 and 5; its rows cover different task sets and should not be treated as the same test set.

Evaluation scope	EchoVLA mean SR	π0.5 mean SR	Diffusion Policy mean SR	Absolute gain over π0.5
Simulation: four manipulation tasks + navigation only, original Table 2	0.52	0.32	0.01	+0.20
Simulation: four mobile manipulation tasks, original Table 2	0.31	0.20	Not listed	+0.11
Real world: six mobile manipulation tasks, original Table 5	0.44	0.33	0.32	+0.11

Simulation mobile manipulation increases difficulty by placing the base farther from targets or requiring additional navigation after manipulation. The four tasks are counter-to-stove placement, sink-to-counter placement, turning on the faucet, and turning on the stove. EchoVLA achieves SRs of 0.17, 0.34, 0.29, and 0.43, respectively. Gains are absolute success-rate differences: +0.11 means 11 percentage points, not an 11% relative improvement.

Ablation Study¶

The following table reproduces the counter-to-stove placement ablations from original Table 3. Mobile requires coordinated movement, while Static is the stationary variant; SR is summarized over 50 episodes. PC denotes point clouds, EM episodic memory, and SM scene memory. Each row removes only the indicated component while retaining the others.

Config	Mobile SR	Static SR	Mobile difference from full model
Full model	0.17	0.21	0.00
Without RGB	0.02	0.13	-0.15
Without PC	0.08	0.15	-0.09
Without SM	0.09	0.16	-0.08
Without EM	0.14	0.13	-0.03

Original Table 4 also tests the window length and update threshold on the same Mobile task. Window lengths of 2, 4, 8, and 16 yield SRs of 0.08, 0.12, 0.17, and 0.15. Thresholds of 0.1, 0.3, 0.5, and 0.7 yield SRs of 0.11, 0.14, 0.17, and 0.13. Neither longer history nor more frequent map updates improves performance monotonically.

Key Findings¶

The two memories have task-dependent benefits: removing SM costs 0.08 on Mobile, while removing EM costs 0.03. On Static, removing EM costs 0.08 and removing SM costs 0.05. This supports different spatial and progress-related roles, but does not establish a strict interaction effect across tasks because there is no control removing both memories together.
EchoVLA reaches only 0.10 on the real long-horizon EnP task. Although better than π0.5 at 0.00, this is not reliable cross-room execution. On refrigerator opening, OR, it scores 0.40 versus 0.50 for π0.5. The authors attribute this failure to door motion and dynamic occlusion degrading explicit geometric memory.
The paper contains reporting inconsistencies. Table 5 gives Diffusion Policy an EnP SR of 0.03, which is incompatible with a single group of 20 trials whose success fractions must be multiples of 0.05; additional averaging is not explained. The prose describes RK baselines as achieving only 0.10, but Table 5 gives π0.5 a score of 0.40; 0.10 applies only to Diffusion Policy. This note retains the table values rather than correcting them on the authors' behalf.

Highlights & Insights¶

Memory design depends on update semantics, not merely cache capacity. Separating slowly changing geometry from rapidly changing interactions prevents stability and rapid forgetting of obsolete states from being imposed through one rule.
Retrieval precedes action generation, so memory is not an auxiliary log for inspection. It becomes shared conditioning for both denoising branches, directly connecting spatial and temporal context to continuous control.
Data quality control targets the full navigation-manipulation transition. Hard filtering establishes feasibility, while offline ranking selects preferable trajectories; these have distinct responsibilities, although safe demonstrations do not automatically produce a safe deployed policy.

Limitations & Future Work¶

The authors acknowledge dependence on high-quality depth and pose streams. Accumulated odometry drift causes map misalignment or ghosting; proposed future directions include loop closure or visual SLAM, and active exploration or generative 3D priors for cold starts in new environments.
The authors report failures under dynamic occlusion, but recent history only provides an additional reference and cannot guarantee map correctness. Updating movable doors separately from static background is a direction suggested by this failure mechanism, not an implemented component of EchoVLA.
Experimental interpretation in this note is limited by insufficiently specified cross-scene splits, map-reset protocols, retrieval latency, and full training budgets, along with missing error bars in the main tables. Advantages should be restricted to the reported settings, not extrapolated to open-world generalization or statistical significance.
This note recommends jointly removing both memories, adding parameter-matched long-history baselines, and controlling the data budget to distinguish benefits from memory structure, input modalities, and MoMani data.

vs MemoryVLA: According to the paper's comparison, MemoryVLA augments manipulation with perceptual and cognitive memory, whereas EchoVLA explicitly separates coordinate-indexed maps from time-indexed history. No direct quantitative comparison with MemoryVLA is provided, so this is an architectural distinction rather than a performance ranking.
vs BSC-Nav: BSC-Nav emphasizes landmarks and cognitive maps for language-model planning; EchoVLA feeds retrieved context directly into a continuous-action diffusion policy. The former emphasizes planning representations, while the latter emphasizes how memory participates in base and arm control.
vs π0.5 / DP3: The main table uses these as strong VLA and 3D-policy baselines. EchoVLA combines dual memory with per-part action generation. A transferable lesson is to identify whether a task lacks spatial persistence or recent progress information before selecting a memory type, rather than uniformly increasing the history window.

Rating¶

Novelty: 4/5. The spatial-temporal memory division and its integration into mobile manipulation are well targeted, although the underlying components are not entirely new.
Experimental Thoroughness: 3/5. Simulation, real robots, and modality and memory ablations are included, but map protocols, error bars, and stronger interaction controls are missing.
Writing Quality: 3/5. The central argument is clear, but some statistical descriptions conflict; damaged formula extraction in the local full text also limits precise reproduction.
Value: 4/5. The work offers a useful memory organization and data pipeline for mobile manipulation while exposing problems with dynamic geometry and cold starts.