Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/1ranGuan/VST
Area: Multimodal Reasoning
Keywords: Streaming video understanding, chain-of-thought, dual memory, causal attention, reinforcement learning

TL;DR

VST moves video reasoning ahead of the user query, continually updating textual memory through causally trained streaming thoughts, enabling VST-7B to reach 79.5% on StreamingBench and 41.9% accuracy with 0.56-second query-response latency on VideoHolmes.

Background & Motivation

An online VideoLLM cannot inspect a complete video and then select its key frames: future frames have not arrived, while past frames gradually exceed the context budget. TimeChatOnline and Streamforest emphasize visual-token compression, and other approaches retrieve historical information from the KV cache. These mechanisms address how much information can be retained, but do not necessarily make the language model analyze relationships between events as they unfold.

Offline video reasoning often generates a long chain-of-thought after a question arrives, exchanging additional test-time compute for better accuracy. A real-time assistant cannot always make the user wait for that reasoning to finish. When answers depend on subtle, widely separated cues, retrospective inspection can be both slow and incomplete. Video playback itself provides an underused opportunity: the model can organize current observations and historical events before knowing the eventual question.

The paper therefore treats visual compression as only part of memory management and asks the model to continually produce semantic memory for later answers. Core Idea: amortize explicit reasoning over pre-query video playback, combine short-term visual buffering with long-term textual memory for direct answering, and train this behavior through causal supervision and final-answer rewards.

Method

Overall Architecture

VST receives a stream of video clips and a question that may arrive during playback, and answers using evidence available at that point. During playback, the model reads the current visual buffer and existing textual memory, then generates a new streaming thought. When the question arrives, it uses the latest visual context and completed memory rather than starting a long new chain-of-thought.

This protocol requires dedicated training rather than an assumption that an offline model will execute it correctly. The authors first synthesize cross-clip evidence chains from video knowledge graphs, teach the causal streaming protocol through VST-SFT, and then use VST-RL to generate trajectories on-policy and optimize earlier memory through the final answer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Source["Training videos"] --> Data["Graph-Grounded Supervision"]
    Data --> SFT["Causal Streaming Fine-Tuning"]
    SFT --> RL["Trajectory-Level Reinforcement Learning"]
    RL --> Memory["Dual Memory and Asynchronous Thinking"]
    Stream["Incoming video clips"] --> Memory
    Memory -->|Next clip| Memory
    Memory -->|User query| Answer["Latest visuals and completed memory<br/>Direct answer generation"]

The first three stages are training procedures; the last stage is the deployment-time inference loop. The knowledge graph is not a third memory structure maintained at inference time.

Key Designs

1. Graph-Grounded Supervision: make training questions depend on dispersed visual evidence

An ordinary offline CoT can explain a current clip using events that have not happened yet, violating streaming causality if used directly for training. The authors segment scenes with PySceneDetect and extract entities and head-relation-tail triples within a sliding window. An entity bank maintains identities across clips. When the window advances, the oldest clip is removed while the other overlapping clips remain, reducing discontinuities in entity tracking across adjacent scenes.

After processing the entire video, an LLM removes noisy entries such as duplicate entities and subtitles. NetworkX constructs a graph, and depth-first search from random starting nodes samples multi-hop evidence chains containing timestamps, entity relations, and scene descriptions. Entity overlap between any two sampled chains from one video must be below 10%. Gemini 3.0 flash produces streaming thoughts and QA pairs requiring multiple pieces of evidence, followed by world-knowledge, format, logical-consistency, repetition, and thought-validity checks. The graph grounds supervision in traceable events rather than unconstrained plausible descriptions; it does not itself prove that every possible form of future-information leakage has been eliminated.

2. Causal Streaming Fine-Tuning: constrain both visibility and prediction targets

VST-SFT orders each training example as initial memory, video clip, corresponding thought, next clip, corresponding thought, and eventually the final clip, question, and answer. In addition to ordinary causal masking, attention can access only a fixed budget of the most recent visual tokens. Historical non-visual tokens remain visible under the causal constraint. The model therefore cannot revisit old images indefinitely and must write useful information into text as it becomes available.

Long videos are further divided into consecutive training segments, with the previous segment's memory conditioning the next. Unlike treating segments as unrelated short videos, this preserves narrative state while bounding individual segment length. Next-token prediction loss applies only to newly generated thoughts and the final answer, not to visual tokens or supplied historical memory. The learned behavior is thus how to extend useful memory and answer, rather than how to reproduce the entire input.

3. Trajectory-Level Reinforcement Learning: select useful thoughts through final correctness

Imitating teacher thoughts can preserve irrelevant details, while deployment requires the model to rely on its own earlier memory. VST-RL therefore runs the policy in a streaming-video interaction loop, generating all intermediate thoughts and the final answer on-policy. It samples a group of trajectories for the same question, computes verifiable rewards solely from final-answer correctness, and assigns the group-relative advantage to every generated token in each trajectory, including thoughts produced before the query.

The paper defines the advantage as follows: reward minus the group mean, without division by the standard deviation in this definition.

\[ \hat A_i = r_i - \operatorname{mean}(R). \]

Optimization uses GRPO-style probability-ratio clipping with DAPO-style lower and upper clipping thresholds; the training diagram also includes a reference-model KL penalty. Final rewards connect what was remembered earlier to how well the model answers later, but do not independently label each thought for faithfulness. This objective improves task utility rather than verifying the truth of every reasoning sentence.

4. Dual Memory and Asynchronous Thinking: replace post-query waiting with pre-query work

At deployment, short-term memory holds current visual content and long-term memory stores earlier textual thoughts. Clip boundaries are set when accumulated visual tokens reach the capacity. The model combines the current clip with the previous memory to generate a thought, then uses FIFO eviction to remove the oldest memory entries. Using the notation in Section 2.1, the central recurrence is:

\[ z_k \sim p(z\mid c_k,m_{k-1}),\qquad m_k=\operatorname{Update}(m_{k-1},z_k). \]

This textual memory is intended to preserve event changes, temporal cues, and cross-clip relationships, not merely image labels, so later answers need not reprocess the entire visual history. However, FIFO is not a learned optimal retriever, and semantic compression cannot guarantee preservation of every detail that a future question might require.

The implementation generates thoughts asynchronously every 16–32 seconds, taking 7.0 seconds on average with a P99 latency of 11.2 seconds, both below the minimum triggering interval. If a query or a new step interrupts background thinking, the model uses the latest completed memory state. Low response latency therefore comes from scheduling and precomputation: background LLM computation and token consumption still increase, rather than disappearing.

A Worked Example

The VideoHolmes example in Figure 6 asks what rule governs the appearance of a man with a blurred face. An earlier clip shows a wall clock at 9:50; a later clip shows 10:00 before the man appears. The relevant evidence is the temporal relationship across clips, rather than simply the woman writing or interacting with an object.

During playback, VST records clock readings and event order in memory, then selects the answer that the man appears automatically at a fixed time, with a 0.51-second response. Video-R1 instead reasons retrospectively after the query, incorrectly selects a specific-object trigger, and takes 9.53 seconds. These are individual-example latencies, not substitutes for the aggregate comparison in Table 6.

Loss & Training

The backbone is Qwen2.5-VL with video sampled at 2 fps; the visual encoder and projection layer remain frozen throughout both stages. VST-SFT uses 100K synthetic streaming-thought examples plus 50K open-ended LLaVA-Vid QA examples. Each training sample is limited to 128 seconds, with longer sequences handled through consecutive segments.

VST-RL uses 11K questions, combining multiple-choice questions from LLaVA-Vid, Video-Marathon, and Onethinker with counting questions from RepCount. Both training stages for the 7B model use 32 GPUs with 80GB VRAM each. RL uses verl, vLLM, and FSDP, with rollout batch size 256, 8 trajectories per group, and precomputed video embeddings during rollout.

Evaluation uses lmms-eval, caps every inference step, including streaming thinking and final answering, at 8,192 video tokens, and permits at most 4 thinking steps. This is the efficient evaluation configuration and should not be conflated with ablations using more thinking steps.

Key Experimental Results

Main Results

Accuracies are percentages, and accuracy gains are percentage points. Table 1 evaluates StreamingBench real-time understanding, Table 2 evaluates OVO-Bench, and Table 3 uses VideoMME without subtitles. Latency comes from the same-setting VideoHolmes comparison in Table 6.

Source Table	Dataset / Metric	VST-7B	Comparison Method	Comparison Value	Difference
Table 1	StreamingBench Overall	79.5	Streamforest-7B	77.3	+2.2 points
Table 2	OVO-Bench Overall	59.3	Streamo-7B	57.9	+1.4 points
Table 2	OVO-Bench Backward Avg.	56.7	Streamforest-7B	52.0	+4.7 points
Table 3	VideoMME without subtitles Overall	64.9	TimeChatOnline-7B	62.4	+2.5 points
Table 3	VideoHolmes accuracy	41.9	Video-R1-7B	36.5	+5.4 points
Table 6	VideoHolmes query-response latency	0.56 seconds	Video-R1 w/CoT	8.80 seconds	About 15.7 times faster response

These are specific comparisons from the respective tables, not a claim that VST leads every model or subtask. For example, Gemini 1.5 pro scores 63.0 overall on OVO-Bench, above VST's 59.3. VST's strength is primarily the combination of open-source online performance and responsive reasoning.

Ablation Study

The following results come from Table 4. The first three data configurations each use 50K examples and isolate data composition; full VST-SFT uses the larger corpus described above, so these groups should not be treated as one equal-budget ablation.

Config	OVO Backward	OVO Forward	OVO Overall	VideoMME without subtitles Overall
Native Qwen2.5-VL-7B baseline	47.5	41.9	50.5	62.9
LLaVA-Vid 50K	49.9	42.4	52.3	61.8
LLaVA-Vid 30K + VST 20K	52.0	50.1	56.8	62.5
LLaVA-Vid 20K + VST 30K	53.3	50.0	57.1	63.1
VST-SFT only	56.7	48.5	57.4	63.0
VST-RL only	49.3	54.6	56.8	62.8
VST-SFT + VST-RL	56.7	54.0	59.3	64.9

Key Findings

• At the same 50K-example budget, replacing part of ordinary QA with 30K VST examples raises OVO Overall from 52.3 to 57.1. The structure of supervision matters, not just the quantity of data.
• SFT improves Backward by 9.2 points over the native baseline; RL alone improves Forward by 12.7 points. Combining them produces the best Overall result, but its 54.0 Forward score remains slightly below RL alone at 54.6, so not every subtask improves monotonically.
• In Figure 5, increasing thinking steps from 1 to 16 raises Backward from 53.3 to 57.5; real-time understanding and forward prediction plateau at at least 4 steps. More recorded detail helps tracing but can add redundancy elsewhere.
• Table 5 evaluates Qwen zero-shot under the VST inference protocol, giving 55.0 OVO Overall for 7B. This is not the native baseline of 50.5 in Table 4; mixing them would misstate the training gain.

Highlights & Insights

• The timing of reasoning is itself a design variable. VST uses playback intervals to organize evidence in advance, separating user waiting time from total reasoning cost rather than merely increasing thought length.
• Training and deployment share the same visual-visibility constraints. Restricting access to older frames makes semantic memory necessary and avoids teaching with global access before demanding streaming behavior at test time.
• Final-answer rewards can shape pre-query memory. They provide task-level feedback about what deserves preservation, although sparse-reward credit assignment remains an open issue.

Limitations & Future Work

• The authors explicitly acknowledge non-negligible additional LLM token consumption and suggest latent reasoning for efficiency. The 0.56-second figure measures query-response latency, not total video-processing time or aggregate compute.
• Reported failures include retaining salient but query-irrelevant information, overcompressing early evidence, missing fine temporal spans, and failing to connect distant weak cues. More faithful temporal memory and cross-event relationship tracking are direct improvement targets.
• From the mechanism, query-agnostic textual compression and FIFO eviction can both discard information irreversibly. Combining the approach with visual memory or retrieval is reasonable and is also proposed as future work, not an already validated component.
• From the evaluation scope, hiding background reasoning depends on completing it within clip intervals. The reported latency table does not establish real-time guarantees for denser inputs, slower hardware, or concurrent workloads.

Related Work & Insights

• vs TimeChatOnline / Streamforest: These methods emphasize streaming visual-context efficiency, whereas VST adds a language-model-generated semantic reasoning stream. The approaches can complement each other; VST has not replaced every visual compression mechanism.
• vs Video-R1: Video-R1 primarily generates CoT after the query, while VST accumulates evidence beforehand. Tables 3 and 6 show accuracy and response benefits, respectively, but do not establish lower total compute.
• vs offline CoT synthesis: VST organizes streaming supervision through timestamped graph evidence chains and causal masking. The transferable principle is alignment between supervision and deployment-time information boundaries, not an assumption that graph structure automatically creates reasoning ability.

Rating

• Novelty: 4/5. Proactive reasoning during video playback becomes a trainable protocol, combining scheduling, supervision, and post-training.
• Experimental Thoroughness: 4/5. Online and offline benchmarks, model scales, and training ablations are covered; total compute and more demanding real-time workloads remain underexplored.
• Writing Quality: 4/5. The pipeline and comparisons are clear, but readers must distinguish baseline protocols and query latency from total cost.
• Value: 4/5. The approach provides a practical reference for responsive video assistants that depend on historical evidence.