OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams¶

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Code: https://github.com/Go2Heart/OmniStream
Area: Robotics & Embodied AI
Keywords: streaming visual representations, causal attention, 3D-RoPE, multi-task pretraining, vision-language-action models

TL;DR¶

OmniStream converts the DINOv3 image encoder into a causal video backbone with a historical KV-cache, jointly trains it with self-supervised representation learning, geometric reconstruction, and language alignment, and reuses the same frozen visual features for perception, spatial reasoning, and robotic manipulation, reaching 3.885 average completed sequence length on CALVIN ABC-D.

Background & Motivation¶

Robot cameras, egocentric devices, and video assistants receive images continuously rather than working with an already complete video file. They must recognize content, understand object motion, camera position, and spatial relations, and avoid using future frames in current decisions. Image models such as DINOv3 specialize in spatial features, video models such as V-JEPA 2 emphasize temporal change, and geometric models such as CUT3R focus on reconstruction; sharing the Transformer architecture does not make these capabilities interchangeable. For example, the paper reports only 54.0% SSv2 action recognition for DINOv3 and 44.2 J&F on DAVIS'17 dense propagation for V-JEPA 2, illustrating that motion understanding and spatial correspondence are distinct objectives.

Converting every task into text generation can unify the output interface without necessarily producing visual representations that different tasks can directly reuse. In particular, features that generate the correct nouns may not preserve depth, displacement, and precise spatial relationships when the downstream task changes from describing a video to controlling a robot arm. The paper therefore asks a narrower question: can a visual backbone first learn semantics, dynamics, and geometry together, then remain frozen while task-side modules are trained? This evaluation exposes deficiencies in the representation more directly than end-to-end adaptation, which lets the visual encoder relearn each task.

The authors start from existing image-based spatial priors, introduce a causal architecture for streaming inputs, and complete the representation with complementary supervision. A single image becomes a stream of length 1, while videos and geometric sequences provide different temporal and spatial constraints to the same encoder. Core Idea: unify a reusable visual backbone that sees only the past and present and carries semantics, dynamics, and geometry, rather than merely unifying the output formats of different tasks.

Method¶

Overall Architecture¶

The input is a sequence of RGB frames, processed by a backbone initialized from DINOv3 ViT-L and modified with causal spatiotemporal attention and 3D-RoPE. At each time step, it produces dense patch features, a global semantic token, and an optional camera token; historical keys and values remain in a KV-cache. During pretraining, the shared backbone receives static and temporal distillation, streaming geometry supervision, and language alignment in parallel, before being reused with different downstream readout modules. The three supervisory branches in the diagram are parallel constraints, not an inference pipeline that first reconstructs a point cloud and then sequentially generates text and actions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current RGB frame<br/>and historical KV-cache"] --> B["Causal Streaming Backbone"]
    B --> C["Static and Temporal Distillation"]
    B --> D["Streaming Geometry Supervision"]
    B --> E["Language Alignment"]
    C --> F["Joint pretraining<br/>then freeze the visual backbone"]
    D --> F
    E --> F
    F --> G["Task-side modules<br/>perception, reasoning, and action"]

Here, freezing applies only to the visual backbone during downstream adaptation; it does not mean that the language model, projector, or action head is untrained. In particular, the VLA setting allows training the vision-to-language projector, unlike the original VLM4VLA frozen setting that freezes both the visual encoder and projector. This is therefore a study of transferring frozen visual representations, not zero-shot robotic control without downstream learning.

Key Designs¶

1. Causal Streaming Backbone: read history without accessing future frames

Each frame is divided into non-overlapping patches and augmented with one global [CLS] token, four register tokens, and an optional [CAM] token for the camera head. The attention restriction operates at the frame level rather than following the order of tokens within a frame, so any current-frame patch can read every other current-frame patch and all preceding frames. This preserves full spatial interaction within each frame while excluding information from future images. Writing \(\tau(u)\) for the frame containing token \(u\), the mask in Eq. (4), page 6, is:

\[ M_{uv}=\begin{cases}0,&\tau(u)\geq\tau(v),\\-\infty,&\tau(u)<\tau(v).\end{cases} \]

Streaming inference computes representations for the new frame and reuses historical keys and values instead of re-encoding the entire past; with the full cache retained, it produces the same result as full-sequence causal attention. The positional encoding allocates each attention head's positional dimensions to time, height, and width in a \(2:3:3\) ratio, interleaving temporal components into the original two-dimensional RoPE while retaining the DINOv3 arrangement of the remaining spatial components. The authors also retain RoPE-box jittering to improve positional robustness, rather than learning an entirely new positional table for time. This modification gives videos temporal order while attempting to preserve the spatial priors already present in the initialization. However, KV-caching removes redundant computation, not the growth of history: page 12 explicitly reports per-step temporal complexity of \(O(T)\), which should not be interpreted as constant latency or constant memory.

2. Static and Temporal Distillation: preserve image discrimination and sensitivity to motion

Static images are single-frame streams and videos are multi-frame streams in a shared DINOv3-style student-teacher distillation framework, with the teacher updated by an exponential moving average of the student parameters. Global and local views provide cross-view constraints, while the student's temporal access still follows the causal mask. The DINO term constrains global semantics, iBOT constrains patch features, KoLeo discourages excessive concentration in feature space, and the Gram term maintains consistency of patch-level structure. Together, these terms discourage the backbone from learning shortcuts useful only to a particular task head while losing general visual information.

Video data contributes more than additional images: it constrains representations through continuous motion of the same objects. Removing VideoSSL reduces first-stage SSv2 performance from 69.3 to 63.0, while ImageNet changes from 85.2 to 85.4; static classification is therefore not a substitute for measuring dynamic representation quality. The paper places the full formulations of the four self-supervised sub-losses in supplementary material, and some equations in this cache have extraction damage, so their internal weights are not reconstructed here.

3. Streaming Geometry Supervision: constrain appearance tokens to reconstructable space

The model reads dense patch features from selected intermediate layers and feeds them to a dual-DPT depth head that predicts depth and ray maps, while a lightweight MLP reads [CAM] tokens to predict camera parameters. Each pixel of a ray map has 6 components: a three-dimensional ray origin and direction; each camera output has 9 components: a 4-dimensional rotation quaternion, 3-dimensional translation, and 2-dimensional field of view. The principal point is assumed to lie at the image center, so this is not unrestricted estimation of arbitrary camera intrinsics. Depth and rays also form a point map, as specified on page 8:

\[ \hat{P}_t=\hat{o}_t+\hat{D}_t\odot\hat{d}_t. \]

Supervision uses normalized ground-truth depth, rays, point maps, and camera parameters, requiring the outputs to explain the same scene jointly. Depth uses confidence-weighted L1 regression with a spatial gradient term and logarithmic confidence regularization; rays, points, and camera parameters use L1 regression. The gradient term emphasizes depth boundaries, while point-map supervision constrains the combined result of depth and rays instead of letting each branch fit labels independently. The purpose of this branch is to incorporate three-dimensional constraints into shared features, so downstream spatial question answering does not require a separate geometric encoder. It nevertheless requires geometric supervision data and should not be described as three-dimensional understanding learned solely from ordinary unlabeled video.

4. Language Alignment: make visual details readable by language models early

During pretraining, last-layer visual tokens pass through an MLP projector into a Qwen3-0.6B autoregressive language decoder for dense captioning, OCR, and object grounding. Language-modeling gradients pass through the decoder into the visual backbone, changing the visual representation rather than merely training a text head to interpret fixed features. OCR encourages preservation of local text, grounding connects words to regions, and captioning supplies broader semantic associations. This explains why representations with reasonable image probing performance can still transfer poorly to VLMs and VLAs when this branch is removed.

The small pretraining language decoder is distinct from the downstream Qwen2.5-7B-Instruct model. The downstream VLM projects frozen visual tokens into the language space and learns video and spatial question answering within the LLaVA-Video framework. The downstream VLA adds a lightweight MLP action head to the language model outputs; action prediction reads visual observations that already contain geometric and dynamic information, without requiring a textual scene explanation first. Perception tasks do not require this language interface: image tasks use linear heads, video action recognition uses attentive pooling, and geometry tasks use their reconstruction heads.

A Worked Example¶

Consider the instruction in Figure 1 to put a spoon on a blue towel; the following illustrates the method rather than reporting an additional experimental trajectory. On the first frame, the visual backbone represents objects and layout and initializes historical keys and values. As the camera or objects move, each new frame can read the historical context but cannot access the next frame; 3D-RoPE distinguishes temporal change from positions within an image. During pretraining, distillation maintains discriminative local features, geometry supervision constrains depth and viewpoint, and language supervision connects the spoon, towel, and their locations to instruction concepts. During control, these pretraining supervisory branches do not all need to run: frozen visual features, the trainable projector and language model, and the action head produce control outputs. The distinction matters: the pretraining branches shape the representation, whereas the downstream action module learns the task-specific control mapping.

Loss & Training¶

From Eq. (10) and its accompanying text on page 8, the joint objective is:

\[ \mathcal{L}_{\mathrm{total}}=0.1\mathcal{L}_{\mathrm{ssl}}+\mathcal{L}_{\mathrm{geo}}+\mathcal{L}_{\mathrm{cap}}. \]

Each training step sequentially processes batches from the different tasks and accumulates gradients, updating the parameters once after all tasks rather than training three separate backbones. The authors use 29 datasets and approximately 200M frames: Table 1 lists approximately 113M for image SSL, 20M for video SSL, 18M for 3D/4D scenes, and 50M for captioning-related data; these are approximate component counts. Image data includes DataComp-100M and ImageNet-21K; video data includes Kinetics, SSv2, and PE-Videos; geometric data spans real and synthetic scenes; language data includes GRIT, the RefCOCO series, Blip3-OCR, and SA1B-Caption. Pretraining uses 64 NVIDIA H200 GPUs, with 60K steps at \(224\times224\) in the first stage and 120K steps at \(512\times512\) in the second stage. The optimizer is Adam with a peak learning rate of \(1\times10^{-4}\), 4K warmup steps followed by cosine decay, and no weight decay in either stage. Every multi-frame sample has length \(T=16\); the causal mask gives different frames in the same clip supervision over historical contexts ranging from 1 to 16 frames. These resources are part of the meaning of downstream freezing: avoiding downstream visual fine-tuning does not imply inexpensive pretraining.

Key Experimental Results¶

Main Results¶

The following selection retains the original metrics rather than collapsing different tasks into a single score; table and page references use the printed page numbers of the supplied PDF. The visual backbone remains frozen downstream, while task heads, language models, and policy-side components are trained under their respective protocols.

Task and metric	OmniStream	Comparison	Source and conditions
ImageNet classification accuracy, higher is better	84.7	DINOv3-L 86.7	Table 2, p. 10; frozen-feature probing
SSv2 ACC@1, higher is better	68.5	DINOv3-L 54.0; V-JEPA2-L 73.7	Table 2, p. 10; attentive pooling
DAVIS'17 J&F, higher is better	71.6	DINOv3-L 73.2; V-JEPA2-L 44.2	Table 2, p. 10; same VOS probing pipeline
Sintel online depth Abs Rel, lower is better	0.314	CUT3R 0.417	Table 3, p. 11; online evaluation
KITTI online depth Abs Rel, lower is better	0.136	CUT3R 0.118; Point3R 0.093	Table 3, p. 11; online evaluation
VSI-Bench average, higher is better	70.6	SpaceMind 69.6; LLaVA-Video-7B 35.6	Table 5, p. 12; different complete VLM systems

Abs Rel averages the absolute pixelwise depth error divided by ground-truth depth, while J&F combines region overlap and contour quality. VSI-Bench demonstrates strong spatial reasoning by the complete system, but comparisons between systems with different data and architectures are not strictly controlled single-variable experiments. For the robotic tasks most closely related to this note's classification, the frozen-vision entries below provide the more relevant comparison.

Frozen-vision VLM4VLA model	CALVIN ABC-D average sequence length	CALVIN Task-5 success fraction	SimplerEnv-Bridge average success rate
Qwen2.5VL-7B	2.905	0.334	18.5%
LLaVA-Video-7B	2.898	0.340	30.2%
OmniStream-7B	3.885	0.634	45.8%

Sources: Table 6, p. 13, and Table 7, p. 14. CALVIN uses the ABC-D long-horizon instruction setting; SimplerEnv uses Bridge V2 for real-to-sim evaluation. The CALVIN value 3.885 is average completed sequence length, not a 3.885% success rate; Task-5 instead measures the fraction completing the five-task chain. Relative to frozen-vision Qwen2.5VL-7B, OmniStream improves average sequence length by 0.980 and SimplerEnv success by 27.3 percentage points. However, fully fine-tuned Qwen2.5VL-7B reaches 4.057 on CALVIN, and pi0 reaches 60.4% on SimplerEnv in Table 7, so the model does not universally outperform specialized policies or fully fine-tuned models.

Ablation Study¶

These results use only first-stage \(224\times224\) checkpoints and must not be directly subtracted from main-table results after the second stage; the source is Table 8, p. 14.

First-stage configuration	SSv2 ACC@1	NYUv2 RMSE, lower is better	VSI-Bench	VideoMME	CALVIN average sequence length
Full model	69.3	0.379	57.3	54.1	3.80
Without VideoSSL	63.0	0.420	57.9	55.8	3.42
Without 3D Geometry	68.4	0.471	52.5	53.8	3.34
Without Captioning	67.4	0.395	44.9	45.0	2.38

Key Findings¶

Language alignment has the largest effect on downstream interfaces: removing Captioning reduces CALVIN from 3.80 to 2.38, a decrease of 1.42, and VSI-Bench from 57.3 to 44.9, a decrease of 12.4 percentage points.
Geometry supervision benefits more than depth maps: its removal increases NYUv2 RMSE from 0.379 to 0.471 and reduces CALVIN by 0.46, supporting the usefulness of three-dimensional information for control transfer.
Multi-task learning does not improve every metric monotonically: without VideoSSL, VSI-Bench is 57.9 and VideoMME is 55.8, both above the full first-stage model. The evidence supports overall complementarity, not an absence of task conflicts.
Length extrapolation is demonstrated within a limited range: page 12 reports evaluation on up to 110 frames after training on 16 frames, not stability on indefinitely long streams.

Highlights & Insights¶

Testing whether a representation remains useful when frozen makes geometry, question answering, and action jointly constrain the claim of generality beyond image classification. This tests backbone reuse more directly than output-format unification.
The three supervisory signals contribute dynamics, physical structure, and linguistic readability; the Captioning ablation particularly shows that visual probing does not fully predict robotic transfer. VLA encoder design should test both whether information is retained and whether the policy interface can read it.
The causal architecture aligns information availability during training and online deployment, while 3D-RoPE retains part of the two-dimensional initialization structure. This adaptation strategy may transfer to other strong image models, but it does not replace long-term memory management.

Limitations & Future Work¶

The authors explicitly acknowledge that the model does not exceed specialists on every task and leave model scaling to future work; ImageNet, SSv2, and KITTI illustrate these trade-offs.
The method implies growing storage for a full historical KV-cache and growing attention cost per step. The main paper does not provide a fixed-memory policy or long-running online frame-rate measurements, so streaming alone does not establish compliance with robotic real-time budgets.
Robotic evidence comes from CALVIN and SimplerEnv simulation evaluations rather than new physical-robot deployments in this paper; frozen vision also does not mean that the projector and entire policy stack are frozen.
This cache does not include supplementary material, preventing verification of all downstream training hyperparameters and self-supervised sub-loss details. Unreported implementation settings should not be inferred from the main text.
The overview and detailed tables contain discrepancies: Table 2 lists 53.7 for OpenVLA on Simpler, whereas Table 7 lists 4.2; some CUT3R depth and pose entries also differ. Specific comparisons here follow the corresponding detailed tables and avoid relying on conflicting entries for the central conclusions.
Future evaluation could measure long-term geometric drift and closed-loop manipulation under a fixed cache budget, alongside pretraining ablations with a matched data budget. These are proposals motivated by the evidence boundaries, not validated results.

DINOv3, paper reference [74]: supplies image-based spatial initialization and the distillation framework. OmniStream adds causal temporal modeling and joint geometric and linguistic constraints rather than reinventing the underlying ViT.
V-JEPA 2, paper reference [6]: specializes in video representations and performs better on SSv2, but the paper's VOS probing reveals weaker dense spatial correspondence. General video representation evaluation should include both motion recognition and dense correspondence.
CUT3R, paper reference [87]: uses persistent state for continuous three-dimensional perception. OmniStream instead emphasizes geometry as supervision for a general representation that also supports language understanding and action.
VLM4VLA, paper reference [111]: supplies the robotic adaptation and re-implantation evaluation framework. This paper studies transfer after changing the visual representation rather than introducing a new action decoding mechanism.
Resources: official paper page, author project page, and code repository. The project and code URLs appear on pages 1 and 3 of the paper and were not checked online during this writing task.

Rating¶

Novelty: 4/5. Individual components have clear precedents, but a causal visual backbone and three supervisory signals evaluated through frozen transfer form a coherent research question.
Experimental Thoroughness: 4/5. The evaluation covers perception, geometry, question answering, and robotics with key ablations; physical closed-loop validation and long-stream resource measurements are missing.
Writing Quality: 3/5. The narrative and component responsibilities are clear, but some overview and detailed results disagree, and the prose describing the CALVIN metric is imprecise.
Value: 4/5. The work provides cross-task design evidence for embodied visual backbones, but it is not a general solution that makes training costs and deployment constraints irrelevant.