RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics¶

Conference: ECCV2026
Paper: ECCV Official Page
Project: RoboStream
Area: Robotics & Embodied AI; Vision-Language Model Reasoning
Keywords: long-horizon manipulation, spatio-temporal grounding, causal memory, object permanence, vision-language models

TL;DR¶

RoboStream binds object-level visual evidence to 3D geometry through STF-Tokens and continuously records action-induced state changes in a CSTG, enabling a vision-language model without task-specific fine-tuning to check manipulation preconditions, track occluded objects, and achieve 90.5% average success on eight RLBench long-horizon tasks.

Background & Motivation¶

Executing one grasp differs fundamentally from completing a sequence of building, covering, uncovering, and restoring objects. A single-step planner can use the current image to select a grasp, but long-horizon manipulation also requires knowing why an object is there, what it supports, and whether the next action will invalidate an established condition. VoxPoser connects language to motion planning through 3D value maps, while SoFar associates language constraints with position and orientation, strengthening the spatial representation of individual actions. The remaining problem addressed here is that reconstructing the scene from each new image lets small positioning errors affect later stacking, while a block just covered by a box can disappear from the planning context. Failure may therefore arise from losing the evolving world state rather than misunderstanding the instruction.

Simply retaining old images does not resolve this problem because the model must still associate objects across frames and infer executable geometry from pixels. Conversely, attaching 3D coordinates only to currently visible objects cannot explain where an occluded object went or distinguish planned motion from an unintended collision. Two interdependent requirements follow: observations must connect to persistent object identities, and historical records must explain how actions changed those objects. For example, restoring an initial stack requires retrieving its bottom block from beneath a container rather than mistaking the currently visible container for a new support surface. RoboStream consequently modifies the VLM's input representation and execution memory instead of retraining an action policy.

The objective is not merely to make the model reason for longer in language, but to provide geometrically supported current states and traceable history at each decision. Visual representations identify objects, 3D attributes constrain their locations and extents, and event logs explain how their states arose. These information sources are updated during execution to support retrieval after occlusion, action-order checks, and replanning after failures. Core Idea: replace independent frame observations with a stream of geometrically grounded states tied to persistent object identities, then use action-linked event memory to preserve causal continuity across long-horizon manipulation.

Method¶

Overall Architecture¶

Inputs comprise the current RGB-D observation, a goal image or natural-language instruction, and the scene memory maintained so far. The output is not a final textual answer but a six-degree-of-freedom action for robotic execution; language planning and numerical geometry resolution are separate. After task-relevant object identification and segmentation, the system proceeds through Spatio-Temporal Fusion Tokens, the Causal Spatio-Temporal Graph, and Causal Reasoning and Action Instantiation. These stages respectively provide grounded object states, maintain relations and events across time, and use that information to select and instantiate the next action.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["RGB-D + goal"] --> Perception["Object identification<br/>and segmentation"]
    Perception --> Tokens["Spatio-Temporal<br/>Fusion Tokens"]
    Tokens --> Graph["Causal Spatio-Temporal<br/>Graph"]
    Graph --> Planning["Causal Reasoning<br/>and Action Instantiation"]
    Input -->|Goal and annotated observation| Planning
    Planning --> Execution["6-DoF action execution"]
    Execution -->|New observation| Perception
    Graph -->|Historical identities and states| Perception

The feedback loop operates during inference and execution; it does not denote back-propagation or policy training. The front-end VLM uses the observation, goal, and memory to describe relevant objects in an open vocabulary, after which SAM3 produces instance masks. Combining these masks with depth yields object-specific point clouds and focuses subsequent visual processing on relevant entities. Goal images suit configurations that are difficult to enumerate fully in text, whereas language suits temporal requirements such as hiding and restoration. Memory conditions object description generation to encourage identity continuity, but the main text does not fully specify identity matching or conflict resolution. Persistent identity should therefore be understood as a key framework objective, not a formally established tracking guarantee.

Key Designs¶

1. Spatio-Temporal Fusion Tokens: bind object appearance to queryable geometry

The basic STF-Token unit is an object instance, not the entire scene image or an imprecise positional description. For each object, its mask is applied to the RGB image, and the VLM visual encoder produces a \(16\times16\) patch grid. Relevant visual evidence is retained using a patch-mask IoU threshold \(I_{th}\), then projected and aggregated into the language representation space. The intention is to reduce redundant background interference and place an object's appearance alongside its associated geometric fields in the same context. The main text does not specify the value of \(I_{th}\) or report threshold sensitivity, so an exact retention ratio cannot be inferred. The geometric branch extracts a 3D center and shape description from the object's point cloud rather than asking the model to re-estimate every numerical relation from appearance.

The reported center is the coordinate-wise point-cloud median, not the arithmetic-mean centroid commonly implied by that term. The shape representation records a mean, standard deviation, minimum, and maximum along each spatial axis; the following quantities correspond directly to Section 3.3:

\[ \mathbf{c}_i^t=\operatorname{median}(P_i),\qquad \mathbf{s}_i^t=\{(\mu_a,\sigma_a,a_{\min},a_{\max})\mid a\in\{x,y,z\}\}. \]

Means and standard deviations characterize the principal point distribution, while extrema supplement geometric extent to reduce what a Gaussian approximation alone might miss. Together with visual evidence and a timestamp, these quantities form the object state; geometric fields are serialized by the text tokenizer and enter the same model context as visual representations. Fusion here does not introduce a newly trained multimodal network; it organizes evidence through existing visual projection and text interfaces. Nor is this a full 3D mesh or a full covariance model: axis-wise statistics should not be mistaken for an exact reconstruction of arbitrary shapes. The benefit is that subsequent actions can retrieve numerical geometry by object identity rather than relying on unconstrained coordinate generation by the VLM.

2. Causal Spatio-Temporal Graph: retain spatial relations and action-induced state history

The CSTG combines a current spatial scene graph with a causal memory log; 4D refers to 3D states evolving over time, not a new four-dimensional sensor. Scene graph nodes correspond to object identities, and each node retains a sliding window of \(K\) STF-Tokens. Edges describe inter-object geometry through Euclidean center distance and directional offset, as defined in Section 3.4:

\[ d_{ij}=\|\mathbf{c}_i-\mathbf{c}_j\|_2,\qquad \Delta\mathbf{c}_{ij}=\mathbf{c}_j-\mathbf{c}_i. \]

These relations let the planner inspect relative positions instead of reading disconnected object coordinates. Distance and direction are not equivalent to a complete physical support model, however, and the main text does not introduce a separate mechanics solver for stability checks. The system compares states within the temporal window to record planned displacement, unintended collision, occlusion, action execution, and subtask completion. Each event includes a timestamp, location, and causal source, connecting an observed movement to the action or external disturbance associated with it.

When an object becomes temporarily invisible, its historical identity, last observed location, and occlusion event remain available to planning. The model can therefore propose removing an occluder on the basis of history rather than treating the hidden object as nonexistent. A last-known location is nevertheless conditional historical evidence; if an unobserved disturbance moves the object during occlusion, the log does not automatically guarantee its current position. Causal here primarily means structured associations between actions and state changes, not a general causal world model identified through interventions. The paper does not specify the value of \(K\), event detection thresholds, or long-term history compression rules, leaving these as reproduction details to resolve. The structure supplies evidence for precondition checks and replanning, rather than an unconditionally correct hidden-state estimate.

3. Causal Reasoning and Action Instantiation: check semantic preconditions before resolving execution geometry

At planning time, the VLM receives the CSTG, an observation annotated with object identities, and the task goal. It uses chain-of-thought (CoT) reasoning to review historical states, check action preconditions, and resolve conflicts involving occlusion or action dependencies. For example, selecting a bottom block requires considering whether objects above it must first be removed, not simply whether it belongs to the goal set. If observations diverge from expectations, state and event records support replanning instead of continuing an entire previously generated action sequence. This is language reasoning supported by structured evidence; the main text does not provide a formal verifier guaranteeing every CoT judgment. The model first produces a semantic directive referencing particular objects, and a geometric resolution stage instantiates the six-degree-of-freedom action using STF-Tokens in the graph.

Separating these stages preserves semantic generalization while reducing errors from language models inventing precise coordinates. This separation does not imply that object centers and axis-wise statistics completely determine arbitrary grasping and rotation policies. For spatial reasoning evaluation, the paper explicitly follows SoFar by incorporating PointSO for six-degree-of-freedom prediction. Training-free must therefore not be interpreted as independence from perception modules, geometric tools, or low-level controllers. After execution, the system observes again and updates tokens, the graph, and event logs; loop reliability still depends on errors in these components. In particular, a failed grasp can change the subsequent state and require renewed precondition checks even when the semantic action was correct.

A Worked Example¶

Consider hiding a block and restoring an initial stack; the following is an explanatory walkthrough based on the paper's task mechanisms, not an additional measured trial. The initial observation establishes identities and 3D states for the target block, an upper block, and the occluding container, while retaining the initial stacking relationship. Before moving the bottom block, the system uses current relations to determine whether the upper object must be removed to satisfy manipulation preconditions. After the target moves to the designated region, a new observation confirms the position change and the log records the movement and its action source. When the container covers the target, direct visual evidence disappears, but event memory retains its identity and last-known location. On receiving the restoration requirement, the planner can first remove the container using that history, then refresh geometry from the reappearing visual and depth evidence. It finally arranges objects according to the initial relationships instead of simply producing a currently plausible-looking arrangement. The example illustrates the division of labor: geometry grounds each placement, while memory allows action order to depend on the past.

Loss & Training¶

RoboStream introduces no new loss function and performs no adaptation training or fine-tuning specific to these evaluation environments. The paper uses Qwen3-VL-8B, Qwen3-VL-32B, and Qwen3-VL-235B as planning backbones for its three model scales. Zero-shot here means that the framework requires no in-domain fine-tuning for target tasks, not that the foundation model or components such as SAM3 were never trained. STF-Tokens, CSTG, and prompt organization operate at inference time, while execution history accumulates during operation rather than serving as supervised labels. Several equations are damaged in the available text extraction; this note retains only geometric expressions unambiguously supported by the prose and does not reconstruct damaged planning-operator equations.

Key Experimental Results¶

Main Results¶

Table 2 on page 12 evaluates eight long-horizon tasks constructed by extending and reconfiguring RLBench assets, with 25 episodes using different random seeds per method per task. The metric is whole-task success rate (%), with equal episode counts across tasks; this is not an aggregate over all native RLBench tasks. Five complex-configuration tasks use goal images, while covering the top/bottom block and unstacking then restacking use text instructions.

RLBench task	SoFar	VoxPoser	RoboStream-8B	RoboStream-32B	RoboStream-235B
Bridge Between Towers	52.0	8.0	76.0	88.0	88.0
Cover Top Block with Box	4.0	4.0	40.0	84.0	92.0
Cover Bottom Block with Box	0.0	0.0	16.0	68.0	96.0
Place Blocks in Two Containers	100.0	96.0	100.0	100.0	100.0
Place Blocks in Two Cont. (Hard)	4.0	4.0	24.0	76.0	88.0
Stack Five Colors	4.0	4.0	60.0	72.0	80.0
Stack Three Colors	60.0	96.0	96.0	96.0	96.0
Unstack then Stack Four Colors	0.0	0.0	52.0	76.0	84.0
Average	28.0	26.5	58.0	82.5	90.5

RoboStream-235B improves the average over SoFar by 62.5 percentage points and over VoxPoser by 64.0 percentage points. Advantages concentrate on occlusion, longer stacking sequences, and reconstruction after disassembly; simple placement into two containers is less discriminative. Performance improves from 8B to 235B, but Table 2 is not a representation-only ablation with identical backbones for every baseline, so the entire gap cannot be attributed to memory.

Ablation Study¶

Table 5 on page 13 fixes RoboStream-235B and removes STF-Tokens, CSTG, or both on the same eight long-horizon tasks. The following selection retains average success and two representative tasks; all numerical entries are percentages.

STF-Tokens	CSTG	Cover Bottom Block	Unstack then Stack	Average success
Absent	Absent	0.0	0.0	12.0
Present	Absent	0.0	0.0	14.5
Absent	Present	88.0	64.0	79.5
Present	Present	96.0	84.0	90.5

Removing CSTG decreases average success by 76.0 percentage points, whereas removing STF-Tokens decreases it by 11.0 percentage points. STF-Tokens alone improve over neither component by 2.5 percentage points; adding them when CSTG is present improves success by 11.0 percentage points. This supports a complementary interpretation: better single-step geometry cannot compensate for missing long-horizon state, while memory allows geometric gains to persist across execution steps. However, CSTG includes state history, relations, and events rather than only a textual log, so this ablation does not separate the contributions of different memory contents.

Key Findings¶

Real-world experiments use a Franka Research 3, a parallel gripper, and a front-view Intel RealSense D435i, with 17 colored objects and 21 tasks repeated three times each. Section 4.2 and Figure 4 on page 10 report 44.4% success for 235B on Hard building versus 11.1% for both SoFar and VoxPoser; its Hard disassembly success is 66.7%. The same section reports 88.9% for 235B and 33.3% for 8B on hide-and-restore, versus 0% for both baselines, but small real-world experiments do not establish general open-world reliability. Table 1(b) on page 11 reports a four-task SIMPLER WidowX + Bridge average of 74.8% for RoboStream-8B versus 58.3% for zero-shot SoFar, showing that gains are not confined to long-horizon tasks. Spatial evaluation adds another line of evidence: Tables 3 and 4 on page 13 report 48.9% overall SpatialBench accuracy and 52.2% integrated 6-DoF success on Open6DOR V2 for 32B. The corresponding SoFar-32B results are 45.3% and 48.4%; these metrics follow different evaluation definitions and cannot be directly pooled with RLBench task success.

Highlights & Insights¶

Bind appearance and geometry to the same identity. The important step is not merely adding coordinate fields, but keeping those coordinates attached to an object that can be referenced again, reducing mismatches between visual reference and execution geometry.
Remember the source of changes, not only the latest frame. Action-linked events provide historical justification for uncovering an object, supplying information that single-frame systems lack on restoration tasks.
Separate reasoning from numerical resolution. The VLM selects semantic actions while geometric records help instantiate them, an interface useful for manipulation systems requiring precise positioning without retraining their planner.
Ablations reveal conditional gains. STF-Tokens contribute little without CSTG, suggesting that single-step spatial competence depends on maintained state during long-horizon execution rather than module improvements simply adding together.

Limitations & Future Work¶

Execution bottlenecks acknowledged by the authors. Section 5 notes that unstable grasping and perceptual uncertainty can still cause failure; correct planning does not guarantee successful physical control.
Incomplete reproduction details for memory. The main text does not specify cross-frame association rules, window length, event thresholds, or confidence handling when external forces move invisible objects; these are reproduction questions arising from the available text.
Limited efficiency evidence. Token compression is motivated by reduced background processing, but no end-to-end latency, memory-usage, or context-growth curves are provided in the main text, so real-time deployment cannot be inferred.
Bounded causal capability. Logs track actions and state changes, but general causal inference under unobserved disturbances, complex contact, or incorrect attribution is not established.
Evaluation scope matters. Real-world tasks mainly involve building, disassembling, and hiding colored objects with few repetitions; the authors identify contact-rich, dexterous, and open-world manipulation as future directions.
Possible next steps. The authors suggest a downstream VLA controller or an end-to-end VLA internalizing spatio-temporal reasoning and causal memory; an additional reader-proposed direction is uncertainty-aware historical states with active reobservation.

Compared with SoFar. SoFar emphasizes language-grounded orientation and six-degree-of-freedom manipulation; RoboStream adds cross-step object states and event tracking while retaining PointSO in spatial evaluation, so it is not a wholly independent low-level geometry system.
Compared with VoxPoser. VoxPoser expresses action objectives through 3D value maps; RoboStream focuses on preserving the state changes on which those objectives depend, making past action consequences available to current planning.
Compared with Inner Monologue. Perceptual feedback supports self-correction, whereas CSTG organizes feedback into objects, times, locations, and event sources, reducing dependence on unstructured historical descriptions.
Compared with SAM2Act and MemoryVLA. Section 4.4 claims better performance on four RLBench cases, but the supplied main text contains no corresponding numerical table, so this note does not extend that claim quantitatively.
Research direction. Separately ablating geometric states, action sources, and occlusion events under controlled external disturbances could distinguish gains from history volume, structured representation, and correct causal attribution; this is a reader-proposed study.

Rating¶

Novelty: 4/5. Integrating persistent object geometry and action events into a training-free planning loop is well targeted, although scene graphs, memory, and geometric tools have precedents.
Experimental Thoroughness: 4/5. Five evaluation settings and fixed-backbone ablations provide breadth, but real-world trials are small and analyses of memory subcomponents and compute costs are limited.
Writing Quality: 3/5. The two core deficits map clearly to the proposed modules, but identity matching, event detection, and several interfaces remain underexplained.
Value: 4/5. The approach offers practical insight for occlusion, restoration, and multistep stacking, serving as a useful reference for explicit state management in robotic planning.