Thinking in Streaming Video¶

Conference: ECCV2026
Paper: ECCV Paper
Code: https://github.com/CASIA-IVA-Lab/ThinkStream
Area: VLM Reasoning
Keywords: streaming video, incremental reasoning, semantic memory, response timing, reinforcement learning with verifiable rewards

TL;DR¶

ThinkStream makes a video model watch, reason, and decide whether to respond chunk by chunk, retains historical reasoning as a semantic substitute for evicted visual tokens, and jointly optimizes answers and response timing through streaming RLVR, improving Qwen2.5-VL-3B's overall OVO-Bench score from 51.00 to 59.66.

Background & Motivation¶

Offline video question answering typically consumes an entire video before generating an answer or a long chain-of-thought (CoT). A real-time assistant cannot wait for the recording to end or use future frames to make a current judgment. As video arrives continuously, it must remember previous events while responding promptly to new ones. Repeatedly applying an offline model to an expanding video prefix increases visual encoding, attention, and generation costs together. Retaining only the latest few seconds is cheaper, but risks forgetting events that a user asks about later.

Existing online video systems control context through visual memory or KV cache compression, while some use classification heads or event triggers to determine when to speak. These mechanisms do not necessarily maintain an explicit, continuously updated reasoning state. This paper examines a specific connection: can a short reasoning update support both the current answer and future memory? Once reasoning has recorded that a person placed a chopping board near the sink, the old frames' dense visual tokens need not remain indefinitely. However, independent chunk captions may omit event relationships and may not determine when enough evidence exists to answer.

The goal is therefore not to attach a lengthy explanation to every frame, but to distribute a limited reasoning budget over video time and train state updates and response timing together. Core Idea: use short reasoning both for online decisions and historical semantic memory, retain only recent dense visual evidence, and train the complete streaming interaction trajectory with verifiable accuracy, format, and timing rewards.

Method¶

Overall Architecture¶

ThinkStream builds on Qwen2.5-VL-3B and takes a user instruction, the current video chunk, and the existing historical state. For each arriving chunk, the model first generates a short <think>...</think> segment, followed by either <silent> or a <response> with an answer. Silence means continuing to observe, not skipping the memory update.

The historical state combines recent visual tokens with accumulated reasoning and interaction tokens. Reasoning-Compressed Streaming Memory (RCSM) evicts only old visual tokens outside the visual window. Streaming-Context RLVR provides feedback on complete interaction trajectories during training, while the High-Efficiency Streaming Backend executes variable-length input processing, decoding, and cache eviction. Solid edges below represent the online state loop; dotted edges represent training. Reward computation is not an additional decision model invoked during deployment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    INPUT["New chunk + instruction"] --> LOOP["Watch-Think-Speak"]
    LOOP --> MEMORY["Reasoning-Compressed Streaming Memory"]
    MEMORY -.->|Training trajectories| RL["Streaming-Context RLVR"]
    DATA["Cold-start traces + verifiable labels"] -.-> RL
    MEMORY -->|Online cache update| ENGINE["High-Efficiency Streaming Backend"]
    RL -.->|Trained policy| ENGINE
    ENGINE --> OUTPUT["Response or silence"]
    ENGINE -->|Reuse history for the next chunk| LOOP

Key Designs¶

1. Watch-Think-Speak: replace one long reasoning pass with continuous short state updates

Reasoning triggered by a new chunk is not an independent caption. It interprets new evidence in light of the existing history: it can record a newly observed event, revise a hypothesis about an ongoing activity, or refine temporal and causal relationships. The second part of the output is the interaction action. Observing new information and deciding whether to address the user are therefore connected within the same autoregressive generation process. No external classifier first decides whether to activate a separate question-answering model; even when no response is needed, useful observations can enter the model's history.

The causal constraint is that each step accesses only current and past chunks, without referring to events that have yet to arrive merely to produce a plausible explanation. Reasoning does not start from scratch only when a question appears: it accumulates along video time, allowing a later answer to reuse an interpretation already formed. The maximum number of reasoning tokens per second of video controls the added cost. The selected 20 tokens/s is a reasoning budget, not the video frame rate or the model's generation throughput. It specifies how much text can be spent updating state per unit of video.

2. Reasoning-Compressed Streaming Memory: evict old frames while retaining their interpretation

RCSM maintains a mixed KV cache. Visual tokens within a recent sliding window preserve detailed evidence; older visual tokens are evicted, while tokens representing earlier reasoning and interaction actions remain in context. The paper's state definition retains both reasoning and actions, so this is neither a single latest summary nor a procedure that repeatedly compresses all history into a fixed-length text. New reasoning depends on previous memory, allowing relationships such as object locations and event ordering to propagate over time.

The key benefit is that old frames and old events need not disappear together. Questions about the past can use the retained semantic state, while recent details remain available in the visual window. Information loss is nevertheless selective: colors, text, or spatial details never recorded in reasoning may become unrecoverable once visual tokens are deleted. Two growth processes must also be distinguished. The window bounds the number of visual tokens, but reasoning tokens continue to accumulate more slowly. The experiments therefore support low-cost operation over the tested video lengths, not an algorithmic guarantee of strictly constant total memory for arbitrarily long streams.

3. Streaming-Context RLVR: train state updates with both answer and timing feedback

Cold-start training first teaches time-grounded short reasoning and the interaction format. Subsequent training samples multiple trajectories in an actual streaming context. Each trajectory proceeds chunk by chunk and uses RCSM to manage history, so the policy faces prefix-only observations and disappearing old visual evidence during training, rather than reasoning offline with the entire video. The authors use Group Relative Policy Optimization (GRPO), compare trajectories sampled for the same instance, and apply the standard clipped objective with KL regularization. The local extraction of that objective is damaged, so it is not reconstructed here as an exact author-provided equation.

The verifiable reward has three components: accuracy checks whether responses match reference answers, format checks the structured output protocol, and timing measures the distance between the actual and annotated response steps. RLVR data are restricted to multiple-choice, binary, and counting questions to enable rule-based answer verification; the same matching rule is not used to assess arbitrary free-form responses. The clearly stated textual definition of the timing reward is reproduced below. The prose specifies summing the three components. This joint trajectory-level feedback indirectly encourages useful reasoning; it does not provide an independent factual correctness label for every reasoning sentence.

\[ R_{\mathrm{time}}=\max\left(0,1-\frac{|t_{\mathrm{resp}}-t_{\mathrm{gt}}|}{w}\right). \]

Here \(t_{\mathrm{resp}}\) is the model's response step, \(t_{\mathrm{gt}}\) is the annotated step, and \(w\) is the temporal tolerance window. Closer responses receive higher reward; both premature and late responses are penalized, and the component becomes zero outside the window. This explains why eventually answering correctly is insufficient: an assistant may guess the right answer before current evidence supports it. The main text does not specify the numerical value of \(w\), and the 20-second memory window must not be mistaken for this reward window.

4. High-Efficiency Streaming Backend: retain manual eviction while reducing decoding overhead

RCSM needs to modify the KV cache by token type and age during ongoing generation. A standard transformers eager implementation permits these operations, but repeated kernel launches add overhead. The authors argue that existing high-throughput engines do not readily accommodate the required custom cache updates. Their backend therefore processes variable-length visual input through eager prefill, runs decoding and eviction through replayable CUDA Graphs, and maintains the cache through in-place memory shifting.

This is not another model that determines answer content. It executes the same streaming loop and also supports online trajectory sampling for RLVR. The division follows the input characteristics: new chunks introduce variable numbers of tokens, whereas repeated decoding and eviction benefit from replaying captured execution graphs. Engineering acceleration and semantic compression address different costs, so backend tokens/s gains should not be attributed entirely to shorter reasoning. The main text refers detailed implementation to an appendix, but the assigned cache contains only the main paper and references. Kernel, positional encoding, and batching details are therefore not invented here.

A Worked Example¶

Following the kitchen scenario in Figure 1, the process can be understood as four consecutive updates rather than four independent video questions. Initially, the scene shows a person at a stove. The model records the scene and activity, has no answer to deliver, and remains silent. In the next chunk, the person moves toward the sink. The model connects this observation to the preceding cooking activity, but this alone does not establish where the chopping board was placed. When the user asks where the board was put, a later chunk provides evidence of the person handling it near the sink. The model updates the spatial relationship and then responds. As cleanup begins, the model continues recording new state without repeatedly answering the same question.

If the corresponding kitchen frames later leave the visual retention window, the board's location may remain available through retained reasoning and response tokens. However, a later question about a small pattern that was never recorded cannot necessarily be answered from semantic memory. The example illustrates that silence still updates state and that preserving events is not lossless frame storage; it is not an additional quantitative experiment.

Loss & Training¶

Data construction first segments scenes with PySceneDetect and then obtains dense chunk descriptions from Qwen3-VL-235B-A22B-Instruct. Timestamped descriptions ground subsequent instructions and reasoning traces; this large model is not a required deployment branch. Instruction combinations cover real-time dialogue, event triggers, and continuous output, together with past, current, and future temporal scopes and seven content semantics. Filtering leaves 39 valid scenario combinations. The pipeline then synthesizes time-grounded reasoning traces, requiring every thought and response to respect the evidence available at its timestamp. Training uses 110K cold-start instances and 9K verifiable RLVR instances. Open-ended questions can enter instruction construction, but RLVR uses deterministically verifiable formats.

Cold-start training uses a batch size of 64 and a learning rate of \(1\times10^{-5}\). RL uses a batch size of 8, 8 trajectories per GRPO group, and a learning rate of \(2\times10^{-7}\). Both stages use AdamW, sample video at 2 FPS with native dynamic resolution, and train on a single node with 8 NVIDIA H20 GPUs. The budget experiment selects 20 tokens/s, while the visual cache experiment selects a 20-second window. These two values respectively control text generation and the duration of recent visual evidence.

Key Experimental Results¶

Main Results¶

Table 1 selects reported scores from the paper's Tables 1 and 2; higher is better throughout. The overall OVO score aggregates real-time visual perception and backward tracing. StreamingBench reports only its Real-Time portion, not all capabilities of the complete benchmark. The main text does not specify finer evaluation splits or uniform sampling and hardware settings across baselines, so these are reported benchmark comparisons rather than resource-controlled experiments.

Model	OVO Real-Time	OVO Backward	OVO Overall	StreamingBench Real-Time
Qwen2.5-VL-3B	60.03	41.98	51.00	67.96
Qwen2.5-VL-32B	64.40	50.33	57.37	74.27
Streamo-3B	61.51	41.76	51.64	Not reported
Dispider-7B	54.55	36.06	45.31	67.63
ThinkStream-3B	67.03	52.30	59.66	75.00

Against the same-size base model, the overall OVO gain is 8.66 points and the StreamingBench Real-Time gain is 7.04 points. The latter score exceeds the reported GPT-4o result of 73.28 but remains below Gemini 1.5 pro at 75.69, so this is not a universal advantage over proprietary models. Raw offline results are 61.9 on VideoMME and 56.4 on LongVideoBench, indicating retained offline understanding after the streaming adaptation.

Ablation Study¶

Table 2 corresponds to the paper's Table 6 and compares memory representations and post-training variants within ThinkStream. Its average is the mean of the three displayed benchmark scores, not the overall OVO score above or an average over offline tasks alone.

Memory Config	StreamingBench Real-Time	OVO Backward	OVO Real-Time	Average
No memory	69.6	41.8	59.2	56.9
Discrete caption memory	59.0	36.9	50.1	48.7
Cold-start CoT memory	70.6	47.6	63.3	60.5
RLVR-optimized CoT memory	75.0	52.3	67.0	64.8

Table 3 corresponds to the paper's Table 4 and varies the maximum reasoning token budget per second of video. Latency uses the table's reported units. The main text does not fully specify the timing granularity or single-GPU configuration of this ablation, so these values should not be treated as per-frame latency on arbitrary devices.

Reasoning Budget (tokens/s of video)	StreamingBench Real-Time	OVO Backward	Latency (ms)
0	69.6	41.8	130
5	70.2	46.9	193
10	72.3	49.7	255
20	75.0	52.3	380
30	75.0	52.6	505

Key Findings¶

Reasoning memory is not equivalent to concatenating captions: discrete captions score 48.7 on average, below 56.9 without memory. This experiment alone does not establish that every caption-compression approach is ineffective.
RLVR improves cold-start CoT memory from 60.5 to 64.8 on average, a gain of 4.3 points. The paper does not separately ablate format, timing, and accuracy rewards, so the full gain cannot be attributed to timing reward.
Increasing the budget from 20 to 30 tokens/s adds only 0.3 points on backward tracing while latency rises from 380 to 505 ms. In the paper's Table 5, expanding the visual window from 5 to 20 seconds raises the backward score from 46.7 to 52.3, while a 30-second window scores 51.6.
Section 6.3 reports 154.07 versus 30.06 tokens/s for the custom backend and eager baseline at batch size 1, and 766.87 versus 276.10 tokens/s at batch size 8. Figure 4 reports end-to-end latency below 0.5 seconds over the tested lengths, not a guarantee for infinite streams or every hardware configuration.

Highlights & Insights¶

Reasoning text is reused as memory rather than discarded after explaining an answer. Its cost therefore supports both current interaction and future backward tracing, while also creating a risk that erroneous reasoning persists.
Silence belongs to the generative action space, allowing response timing and answer content to be trained jointly. This formulation can transfer to interactions that require waiting for evidence without assigning the entire decision to an external trigger.
The relationship between algorithm and backend is explicit: semantic memory reduces historical visual load, while CUDA Graphs reduce execution overhead. Deployment evaluations should measure both separately to avoid conflating algorithmic gains with hardware optimization.

Limitations & Future Work¶

The authors explicitly control cost with short reasoning and a visual window, but the main paper contains no dedicated limitations section. Efficiency claims come from a finite experimental range and should not become strict complexity claims for infinite video.
Note analysis: historical reasoning keeps accumulating, and its original visual evidence may already have been deleted, allowing an early mistake to become a premise for later reasoning. Hierarchical text compression, memory correction, and retention of selected keyframes deserve testing, but are not implemented components of this paper.
Note analysis: timing supervision depends on an annotated response step and tolerance window. Open-ended dialogue may admit several reasonable response times, and verifiable rewards for multiple-choice, binary, and counting tasks may not cover this ambiguity.
Numerical caveat: the paper's Table 3 labels the gain from 61.5 to 61.9 as 0.8 and the gain from 54.2 to 56.4 as 1.8. Subtracting the listed raw values gives 0.4 and 2.2, respectively, so those two gain annotations are not adopted here. The base model's StreamingBench score is 68.9 in Table 6 but 67.96 in Table 2, without an explanation of the setting difference in the main text.
Reproducibility boundary: the assigned full-text cache does not include the cited appendix, preventing verification of source-data splits, complete prompts, the timing-window hyperparameter, and backend implementation details. The code URL is disclosed by the paper; its repository contents were not checked online for this note.

vs VideoLLM-online: this represents a direct online perception-response approach. ThinkStream adds continuous reasoning and semantic memory, with benefits accompanied by extra generation cost; comparisons should account for the budget, not only final answers.
vs StreamMind / EgoSpeak: according to this paper's related-work discussion, these approaches rely on event triggers or dedicated decision mechanisms. ThinkStream places silence and response in one generative protocol and optimizes the action through timing reward.
vs offline video reasoning and LightThinker: the former generally reasons over complete video context, while the latter focuses on reasoning-context compression. This paper instead makes causally constrained incremental reasoning itself serve as long-term memory for streaming video.

Rating¶

Novelty: 4/5. Online reasoning, interaction timing, and memory reuse are integrated coherently, although the underlying techniques have precedents.
Experimental Thoroughness: 4/5. Online and offline evaluation, memory, budgets, and efficiency are covered, but independent reward ablations and longer-horizon validation remain missing.
Writing Quality: 3/5. The core mechanism is clear, but inconsistent gain annotations and cross-table baselines complicate verification.
Value: 4/5. A practical direction for compact streaming video assistants, with information loss and memory growth still requiring scrutiny in long-term deployment.