CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management¶

Conference: ECCV 2026
Paper: ECCV paper page
Area: Video Understanding
Keywords: streaming video understanding, feature trajectory curvature, hierarchical visual memory, dynamic thresholds, training-free

TL;DR¶

CurveStream detects semantic changes through the turning behavior of visual feature trajectories and uses online dual thresholds to retain frames at high resolution, retain them at low resolution, or discard them, improving Qwen2.5-VL-7B from 73.31% / 59.90% to 84.00% / 73.48% on StreamingBench / OVOBench without training.

Background & Motivation¶

A streaming video has no predetermined endpoint, whereas a multimodal large language model (MLLM) can accommodate only a limited visual context. Keeping every historical frame causes visual tokens and the KV cache to grow continuously; truncating the oldest content can remove the causes needed to answer a current question. Uniform sampling is simple, but it can spend the budget on static backgrounds while missing a brief action or a newly appearing object. The challenge is therefore not merely to reduce frame counts, but to decide continuously which visual evidence deserves context space before future questions are known.

Adjacent-frame differences alone are insufficient. Camera translation can change the image continuously without introducing a new event, while an important action may last only briefly. Query-conditioned retrieval can recover information after a question arrives, but requires additional storage and a post-hoc query. CurveStream instead focuses on the decision before memory insertion: if consecutive frames move in similar directions in feature space, even substantial displacement may represent smooth change; a sudden turn is a stronger candidate for a semantic transition.

The method links geometric change intensity to memory resolution instead of compressing every retained frame to the same quality. Core Idea: use a curvature proxy on feature trajectories to identify important transitions, preserve strong transitions with high-resolution detail and ordinary transitions with low-resolution context, and control long-term cost through adaptive thresholds and a bounded queue.

Method¶

Overall Architecture¶

The inputs are arriving video frames and natural-language questions issued at arbitrary times. CurveStream adds online selection and memory management to the visual input side of an existing MLLM: it computes a curvature-aware score, estimates dynamic thresholds for the current video pace, and updates hierarchical visual memory. Question answering uses the retained visual context.

The geometric scoring frontend is a frozen DINOv2-small model. It supplies global frame features; this does not mean that the base model's visual encoder, projector, or language model is retrained into a new network. After the resolution decision, retained frames still enter the base model's visual encoding and language generation pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Arriving frame and recent features"] --> Score["Curvature-Aware Scoring"]
    Score --> Threshold["Online Dual-Threshold Adaptation"]
    Threshold --> Memory["Hierarchical Visual Memory<br/>High resolution, low resolution, or discard"]
    Query["Current frame at query time"] -->|Force high-resolution retention| Memory
    Memory -->|FIFO eviction when over budget| Context["Bounded visual context"]
    Context --> Model["Existing MLLM + question<br/>Generate answer"]

Memory updates in the diagram occur at inference time; there is no additional training-supervision branch. High-resolution memory is not a permanent storage tier: the oldest content can still be evicted when the capacity limit is reached.

Key Designs¶

1. Curvature-Aware Scoring: distinguish persistent movement from a sudden trajectory turn

The Curvature-Aware Scorer (CAS) first L2-normalizes global frame features. It combines two signals. The first is motion variation between the current and previous frames, described as 1 minus their feature cosine similarity. The second compares the directions of two consecutive feature displacements rather than the features themselves. For three frame features, the displacements are defined as:

\[ d_1=F_{t-1}-F_{t-2},\qquad d_2=F_t-F_{t-1}. \]

The curvature proxy also uses 1 minus cosine similarity, but its arguments are these displacement vectors. It approaches 0 when the two displacements point in the same direction and increases when their directions change abruptly. CAS is therefore not simply a renamed optical-flow magnitude: it supplements ordinary frame differences with information about whether the direction of change itself has changed. The authors linearly combine the first-order motion term and the second-order geometric term, with \(\lambda\) controlling the latter's contribution.

Here, curvature is a turning-angle proxy for a discrete feature trajectory. It should not be interpreted as an exact differential-geometric curvature of a continuous manifold or as a direct judgment of event importance. Smooth camera movement is more likely to produce directionally consistent trajectories, but abrupt camera turns, noise, and unstable features can also produce high scores. The connection between geometry and semantic importance is an empirical assumption, not a mathematical guarantee.

The supplied full-text extraction corrupts some numerators, operators, and subscripts in Eqs. (3)–(6). The explanation above follows the intact surrounding prose and displacement definitions; guessed reconstructions of the score or variance recurrence are not presented as the authors' exact equations.

2. Online Dual-Threshold Adaptation: interpret the same change magnitude relative to the current video pace

A fixed threshold struggles with both prolonged stillness and sudden vigorous activity. Hierarchical Visual Memory Management (HVMM) updates the score mean and variance using an exponential moving average (EMA), with momentum \(\gamma\) controlling the decay of historical information. The evolving mean and standard deviation define two K-Sigma thresholds:

\[ g_1=\mu_t+k_1\sigma_t,\qquad g_2=\mu_t+k_2\sigma_t,\qquad k_1<k_2. \]

These thresholds assess how unusual the current change is relative to recent activity, instead of measuring every video against one absolute scale. A small transition during a quiet period may deserve retention. When activity becomes persistently intense, the score distribution and thresholds rise, preventing every frame from entering high-resolution memory.

Adaptation requires only a small set of statistics and does not wait for a user query to compute relevance. However, it depends on how quickly the EMA tracks distribution shifts: excessive history causes lag, whereas too little history increases sensitivity to transient noise. The paper does not provide a sufficiently complete set of default hyperparameters and initialization details, so illustrative thresholds should not be mistaken for a verified implementation configuration.

3. Hierarchical Visual Memory: allocate limited tokens to critical details and temporal transitions

A frame whose score reaches or exceeds \(g_2\) enters Clear Memory and retains the base model's native dynamic high-resolution input. A score between \(g_1\) and \(g_2\), inclusive at the lower bound and exclusive at the upper bound, routes the frame to Blurred Memory at \(224\times224\). A score below \(g_1\) discards the frame. The current frame at query time is forcibly assigned to high-resolution memory, preventing a stable scene from losing immediately relevant detail merely because its change score is low.

Low-resolution frames are not redundant copies kept without purpose: they preserve transitional evidence between actions and events. Retaining only high-resolution turning points can preserve the before and after while losing the connection between them; keeping every frame at high resolution quickly consumes the budget. Hierarchical retention concentrates detail on key frames while using cheaper visual context to preserve temporal continuity.

When the queue exceeds capacity, FIFO removes the oldest tokens regardless of their original resolution tier. Thus, the score controls admission and representation precision, not long-term retention priority. This distinction matters: the method bounds storage cost, but cannot guarantee that an arbitrarily old important event will never be forgotten. It does not add an external retrieval store that can restore deleted content.

The paper denotes queue capacity by \(N_{\max}\) and emphasizes a maximum visual-token budget. The 10–20 frames in Table 1 describe the observed size of the dynamic memory queue, not the total number of frames processed over the entire stream. High- and low-resolution frames have different token costs, so capacity management must account for tokens rather than frame counts alone.

A Worked Example¶

Consider a camera moving smoothly along a street, turning toward a billboard, and then becoming stable again. This example illustrates state transitions; it is not a quantitative case study from the paper.

During smooth movement, adjacent features may differ substantially, but successive displacement directions remain similar. The curvature term therefore does not stay high simply because movement continues. Turning toward the billboard changes the feature trajectory; if the combined score exceeds the upper threshold, the frame enters memory at high resolution, preserving details that may support reading the sign.

Intermediate-score frames during the turn enter at \(224\times224\), connecting the earlier and later views. Redundant frames can be discarded after the camera stabilizes. If the question is then "What brand is on the billboard?", the current frame is still forcibly retained at high resolution rather than filtered out for having a low change score.

As the video continues, the queue evicts its oldest content when the budget is reached. If a question about the billboard arrives much later and its frames have already been evicted, CurveStream itself provides no guarantee that the answer can be recovered.

Loss & Training¶

This is a training-free inference-time method: it adds no supervised loss and does not fine-tune the base model. The paper states the policy objective as increasing the conditional probability of a correct answer from the current memory under a fixed capacity constraint. The implementation uses geometric scoring, EMA thresholds, and queue rules rather than gradient-based optimization of that objective.

Experiments use DINOv2-small for scoring features, fix low-resolution inputs at \(224\times224\), preserve the base model's native high-resolution settings, and run on a single inference GPU. Efficiency experiments ingest video at 1 FPS. Table 4 sweeps \(\lambda\) from 0.2 to 1.0; Figure 4 studies one threshold parameter while fixing either \(k_1=0.0\) or \(k_2=1.0\). These sweeps do not fully specify the defaults for all experiments.

Key Experimental Results¶

Main Results¶

Table 1 reports average accuracy across 10 real-time visual understanding subtasks in StreamingBench and 6 real-time visual perception subtasks in OVOBench. The selection below emphasizes matched-backbone comparisons. Scores are percentages; gains are percentage points.

Backbone and configuration	StreamingBench	OVOBench	Gain over matched backbone
LLaVA-OneVision-7B	71.34	63.06	Baseline
+ CurveStream	75.12	70.57	+3.78 / +7.51
Qwen2-VL-7B	69.04	60.65	Baseline
+ FreshMem	74.20	66.67	+5.16 / +6.02
+ CurveStream	81.04	70.73	+12.00 / +10.08
Qwen2.5-VL-7B	73.31	59.90	Baseline
+ HERMES	79.44	68.98	+6.13 / +9.08
+ CurveStream	84.00	73.48	+10.69 / +13.58
Qwen3-VL-8B	73.20	70.10	Baseline
+ CurveStream	85.56	80.76	+12.36 / +10.66

These results support effectiveness across multiple backbones, but not the claim that every model gains more than 10 percentage points on every task. LLaVA-OneVision-7B, for example, has smaller improvements. The FreshMem and HERMES gains shown here are calculated by subtracting their respective backbone rows.

Offline results below come from Table 2 and all use Qwen2.5-VL-7B. Both gains and regressions are retained to avoid treating streaming advantages as a guarantee for all long-video tasks.

Configuration	MVBench	EgoSchema	VideoMME
Baseline	65.00	58.47	64.52
+ HERMES	65.53	59.47	60.63
+ CurveStream	66.03	64.29	62.97
CurveStream gain over baseline	+1.03	+5.82	-1.55

Ablation Study¶

Table 3 compares sampling strategies on StreamingBench with a selected-frame budget of \(N=10\). Training-free methods use Qwen2-VL-7B. StreamForest uses Qwen2-7B and is a trained reference system, not a strictly controlled replacement of the sampler alone.

Sampling strategy	Accuracy (%)	Interpretation
Uniform sampling	69.04	Ignores content changes
Cosine similarity	73.28	Uses adjacent-frame differences
Optical flow	46.54	The ordinary optical-flow variant in this table
Pyramid optical flow	75.69	A different flow variant; not interchangeable with the previous row
StreamForest (trained)	77.26	Reference with different training conditions
CurveStream curvature sampling	77.31	4.03 percentage points above cosine similarity

The 77.31% in Table 3 and 81.04% in Table 1 belong to different evaluation settings and are not interchangeable. Figure 3b indicates a favorable trade-off when adaptive mixed retention yields approximately 50% high-resolution frames, with roughly 40% less computational overhead reported by the authors. This is not an algorithmic rule fixing the high-resolution ratio at 50% for every video.

Key Findings¶

Table 4 reports accuracies of 65.83, 62.50, 63.33, 62.50, and 65.00 for \(\lambda\) values of 0.2, 0.4, 0.6, 0.8, and 1.0. The range is 62.50%–65.83%, consistently above the 60.65% baseline, but distinct from the full-configuration 70.73% in Table 1.
Table 5 shows that, for 15-minute video ingested at 1 FPS, E2E decreases from 63.28 to 44.69 ms/token and time to first token (TTFT) from 5.606 to 1.234 s, corresponding to 1.42-fold and 4.54-fold speedups. TTFT is not per-frame processing time.
Section 4.6 reports approximately 10K–12K visual tokens and about 20 GB of memory. These are empirical measurements for the tested implementation, not universal resource limits across all backbones and resolutions.

Highlights & Insights¶

Second-order directional information complements first-order change magnitude. It distinguishes a moving image from a change in the pattern of movement, providing candidate event boundaries without additional training.
Content selection is coupled with representation precision. Transitional frames need not face a binary choice between full retention and complete deletion: low-resolution memory can preserve action continuity with fewer tokens.
Forcing the current frame into high-resolution memory is an important practical safeguard. A change score helps select history but does not measure the detail needed for the current question, so this rule protects static scenes from inappropriate filtering.

Limitations & Future Work¶

The authors explicitly report a VideoMME regression from 64.52% to 62.97%, attributing it to global detail lost under a fixed memory budget. Preserving transitions is insufficient when tasks depend on distant, sparse, low-change clues.
FIFO makes no exception for high-resolution key frames. Supporting an infinite stream means bounded storage during continued operation, not lossless access to infinite history. Compact summaries or external retrieval could complement the method, but are not components already provided here.
From a reader's perspective, high curvature need not imply high semantic value. Further evaluation should separate abrupt camera turns and noise from genuine event changes, and specifically test low-curvature scenes with important fine details.
Reproducibility remains incomplete: the supplied main text does not fully specify budgets, EMA initialization, numerical handling of cosine similarity for zero displacement, or all default parameters. Corrupted equation extraction also limits exact implementation recovery from this text alone.
Efficiency measurements show lower overall latency, but Table 5 does not separate DINOv2 scoring, visual encoding, and language generation time. Additional scoring overhead still needs to be measured separately on different hardware.

vs FreshMem: FreshMem uses frequency-space hybrid memory, whereas CurveStream bases scheduling on feature-trajectory geometry and online thresholds. In the Qwen2-VL-7B comparison in Table 1, CurveStream leads by 6.84 / 4.06 percentage points.
vs HERMES: Both address hierarchical streaming memory, but CurveStream explicitly makes curvature-driven resolution decisions at frame admission. Its matched Qwen2.5-VL-7B results lead by 4.56 / 4.50 percentage points; these differences must not be confused with the FreshMem comparison.
vs ReKV: ReKV retrieves in-context video KV-cache information, whereas CurveStream emphasizes query-independent online selection before a question arrives. Recovering information afterward and deciding what to retain beforehand are different design choices, not equivalent substitutes.
Transferable insight: Robotics and egocentric perception can use feature turns as candidate event boundaries, but should model task value and recency separately. Otherwise, geometrically salient yet irrelevant camera movement may still occupy memory.

Rating¶

Novelty: 4/5. Combining discrete feature curvature with online hierarchical resolution scheduling is concise, but remains a heuristic visual-memory policy.
Experimental Thoroughness: 4/5. Multiple backbones, online and offline tasks, and efficiency analyses are covered, but budget descriptions and component-level costs remain incomplete.
Writing Quality: 3/5. The main argument is clear, although some broad improvement claims exceed the table-level evidence and evaluation settings require careful separation.
Value: 4/5. Useful for resource-constrained streaming VLM prototypes, without establishing lossless long-term historical memory.