X-Stream: Benchmarking MLLMs as Multiplexers for Multi-Stream Understanding

Conference: ECCV 2026
Paper: ECCV official page
Area: Multimodal VLM
Keywords: multi-stream understanding, streaming video, proactive QA, benchmark evaluation, multiplexing strategies

TL;DR

X-Stream is the first benchmark for streaming understanding over multiple concurrent video streams: a sufficiency+necessity dual-verification pipeline builds 4,220 cross-stream QA pairs, and MLLMs are abstracted as bandwidth-limited naive multiplexers to systematically compare spatial, temporal, and semantic multiplexing strategies; the best model, Gemini 3 Pro, scores only 49.60 overall, and forward (proactive) QA scores generally stay below 21, exposing a large gap in concurrent-stream and proactive response capabilities.

Background & Motivation

Video streaming understanding is moving from one stream to many. Real applications are inherently multi-stream: a World Cup broadcast runs over 40 cameras, a robotic arm wears both shoulder and wrist cameras, autonomous driving watches the map and the street together, and multi-screen collaboration squeezes several feeds into one display. These scenarios demand answering at the right time from the right stream — not watching a complete video file and then replying. Yet existing evaluations are almost all single-stream: multi-video/multi-view benchmarks (MVU-Bench, EgoLife, WaymoQA) are offline, while streaming benchmarks (StreamingBench, OVO-Bench, ProactiveVideoQA) carry only one stream. The authors organize multi-video data forms into a pyramid — Multi-Video has the fewest constraints, above it Multi-View (multiple perspectives of the same activity) and Multi-Angle (multiple angles of the same subject at the same time), while Multi-Stream sits in the middle introducing the critical constraint of timestamp alignment — so multi-stream streaming understanding is an unmeasured gap layer.

Building such data hides a subtle trap: automatically generated "pseudo multi-stream" QA. A cross-stream question may look like it needs two videos while actually being answerable from any single stream. The authors observed two typical forms during construction: pseudo reference (the cross-stream anchor becomes invalid — if "sitting" in Stream 1 spans the whole video, querying it against a momentary event in Stream 2 makes temporal alignment meaningless) and pseudo cooperation (overlapping fields of view create redundant information, so a single stream suffices). Unchecked, the benchmark silently degenerates into an ordinary single-stream evaluation.

Core idea: block the single-stream shortcut at the data-protocol level with a sufficiency+necessity dual verification — given the multi-stream video clipped to the annotated timestamp, a strong model must answer correctly (sufficiency); given only any individual stream, a strong model must fail (necessity) — and conceptualize the MLLM as a bandwidth-limited naive multiplexer, systematically evaluating how concurrent streams are integrated through the lens of signal multiplexing theory.

Method

Overall Architecture

X-Stream's "method" consists of two parts: a data construction pipeline that guarantees multi-stream necessity, and an evaluation protocol that organizes online inference by treating MLLMs as multiplexers. Starting from 20+ sources and 857 hours of raw multi-stream material (2-10 streams), pairing and preprocessing select 160 hours with 2-5 streams, forming 932 videos, 451 takes, and 4,220 QA pairs; about 20% of takes have 3-5 streams and about 30% of questions involve audio/speech. Coverage spans eight domains: driving, sports, robotics, daily routine, chat, surveillance, live streaming, and interface.

Evaluation is organized along capability × task dimensions: the four core capabilities are single-stream understanding, cross-stream anti-interference, cross-stream reference alignment, and cross-stream cooperation (9.6%, 28.6%, 33.4%, and 28.4% respectively); the 11 subtasks span three progressive levels — foundational multimodal perception (visual/audio/temporal grounding, object counting, saliency detection), high-level logical cognition (3D spatial, causal, counterfactual, and commonsense reasoning, anomaly detection), and decision-making (behavior planning). Questions split into instant and forward types: the former can be answered immediately; the latter are proactive tasks that must wait until the answer conditions occur.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw multi-stream material<br/>857h / 2-10 streams / 20+ sources"] --> B["Preprocessing<br/>2FPS resampling + 50MB chunking"]
    B --> C["QA generation<br/>templates + Gemini-3-Pro refinement"]
    C --> D["Multi-stream sufficiency check<br/>clipped multi-stream must answer correctly"]
    D --> E["Multi-stream necessity check<br/>every single stream must fail"]
    E --> F["Two-round human review<br/>31 experts"]
    F --> G["4,220 QA benchmark<br/>11 subtasks / online streaming evaluation"]

Key Designs

1. Sufficiency+necessity dual verification: rule out the single-stream shortcut at the data-protocol level

Against pseudo multi-stream QA, the pipeline applies two machine checks to every candidate QA: first clip the video to the annotated timestamp and let a strong model answer with the full multi-stream input — retain only if correct (sufficiency, ensuring the question is answerable and the timestamp is right); then feed each stream individually — discard the sample if any single stream alone yields a correct answer (necessity, ensuring the answer truly requires joint multi-stream understanding). Verification deliberately uses high-resolution, high-bitrate video, since the target timestamp is already fixed, ruling out false failures caused by low-quality input. The whole flow is adversarial: generation tends to create shortcuts, and verification is designed specifically to catch them. This differs fundamentally from the common "generate then spot-check" pipeline — the necessity check turns "must be multi-stream" into a hard constraint rather than a statistical tendency.

2. Timing evaluation for forward questions: being right is not enough; being right on time is what counts

For forward (proactive) questions, the answer condition has not occurred at question time; the model must keep monitoring the streams and respond the moment the condition first holds. Evaluation uses a 2-second response window over 6 time slots: any response other than silence before the proactive timestamp counts as Early Response (ER, lower is better); staying silent throughout the response window counts as No Response (NR, lower is better); full credit requires the correct answer exactly within the 2-second window after the timestamp. When computing capability-level scores, only temporally accurate answers are averaged, preventing answer-timing differences from contaminating capability scores. This design separates "when to speak" from "what to say" — and most models fail at the former: Qwen3-Omni-30B has solid comprehension (Instant 63.92) but Forward is only 0.61 with ER at 98.81, meaning it almost always jumps the gun.

3. MLLMs as naive multiplexers: a systematic treatment of three multiplexing strategies

An MLLM can consume only one token stream at a time, so concurrent videos must first be merged into one token stream — precisely the multiplexing problem in telecommunications. The authors enforce a fixed average video token rate ceiling \(C_{\max}=250\) tokens/second (analogous to channel bandwidth) and implement three strategies. Spatial division: pixel-level concatenation of downsampled frames from all streams (\(X_t=\text{Concat}(D(M_t,r_m),D(N_t,r_n))\), retention ratios maximized within the budget), preserving cross-stream spatial relations within a single frame, but requiring video re-encoding and audio mixing. Temporal division: streams enter as independent inputs, separated by a <stream N> identifier, with indicator variables α, β sampling each stream's frames uniformly under the budget; temporal embeddings are explicitly aligned, though synchronized temporal encoding is feasible only for open-source models. Semantic division: token-level salient-sample selection — an auxiliary visual encoder builds a conditional DPP kernel (relevance × similarity), and greedy MAP inference iteratively selects top-k tokens balancing task relevance and diversity, then interleaves streams via time division; since proprietary models cannot be modified at the token level, the workaround converts the stream with the most retained tokens back into frames. The authors are explicit that this is an abstraction of current practice rather than a new architecture: the three strategies' strengths shift with stream count and bandwidth, and no universal winner exists.

Loss & Training

This is a benchmark and evaluation study; no models are trained. The online inference pipeline processes continuous streams at 1-second intervals while maintaining a sliding memory window; scoring uses LLM-as-a-Judge, calibrated against humans: 31 experts and the judge model independently scored 200 QAs, yielding a Spearman correlation of 0.62 (p<0.05). The human preference baseline under the same pipeline scores 91.84 overall, showing that the ceiling is far above current model levels and the benchmark is not saturated.

Key Experimental Results

Main Results

The main comparison adopts the most straightforward spatial division multiplexing, reporting Instant / Backward / Forward / Overall scores plus forward-timing metrics (ER/NR, lower is better). The table below excerpts the original Table 2 (scores on a 0-100 scale):

Model	Overall↑	Instant↑	Backward↑	Forward↑	ER↓	NR↓
Human Preference	91.84	91.73	95.19	85.10	9.50	2.55
Gemini 3 Pro	49.60	73.38	72.23	20.77	73.13	0.23
Doubao-Seed-1.8	36.79	55.49	57.18	14.52	66.19	3.95
Qwen3-Omni-30B-A3B	34.28	63.92	53.40	0.61	98.81	0.27
Qwen3-VL-30B-A3B	34.19	52.09	38.54	14.46	73.91	1.18
GPT-5	27.78	44.28	37.18	6.51	81.73	1.14
GPT-4o	22.46	37.28	32.72	4.05	87.14	0.74
Dispider (streaming)	15.44	21.71	19.29	8.09	55.63	7.26
VideoLLM-online-8B (streaming)	8.48	15.00	15.53	0.03	99.10	0.66
MMDuet2 (streaming)	6.79	11.76	10.37	1.44	31.49	54.11

On the multi-stream ability dimensions (also from Table 2), Gemini 3 Pro scores 72.45 single-stream / 71.16 cross-stream cooperation / 74.79 cross-stream reference / 66.96 anti-interference — the most balanced model; open-source Qwen3-Omni-30B scores 63.41 / 55.68 / 66.08 / 56.58 respectively. At the 11-subtask level (original Table 3) a clear capability gap emerges: foundational perception is generally passable (Gemini visual grounding 66.72, counting 76.37), while logical cognition collapses — causal reasoning peaks at only 41.79 (Gemini), counterfactual reasoning is oddly led by Qwen3-Omni (90.00) but with evidently few samples (subtask scores across models are discrete values like 10/15/20/40/50/85/90), and decision-making/behavior planning is weak across the board (Gemini 44.18, most models below 35).

Ablation Study

Multiplexing ablation by ability (original Table 4) and stream-count scaling (original Table 5, Overall):

Model / Scheme	Single	Coop.	Ref.	Inter.	N=2	N=3	N=4	N=5
Gemini-3-Pro / Spatial	72.45	71.16	74.79	66.96	57.47	19.86	30.48	22.81
Gemini-3-Pro / Temporal	79.62	75.13	67.08	80.74	58.09	24.19	31.76	22.14
Gemini-3-Pro / Semantic	70.30	66.64	70.92	68.94	55.50	26.89	41.25	31.06
Qwen3-Omni / Spatial	63.41	55.68	66.08	56.58	36.61	36.88	34.22	25.84
Qwen3-Omni / Temporal	69.78	58.76	58.62	69.14	40.55	36.17	35.83	25.44
Qwen3-Omni / Semantic	61.58	52.13	55.34	59.03	36.15	38.46	40.64	29.82

⚠️ The original Tables 4 and 5 are set side by side in a two-column PDF, and the cached extraction scrambles their row order; the values above are realigned using rows consistent with Table 2 (the Gemini spatial row matches Table 2 exactly) and the printed column headers. For individual cells, the original PDF should be considered authoritative.

Key Findings

• No multiplexing scheme is universally best: spatial division excels at cross-stream referencing (preserving pretrained intra-frame inference dynamics), temporal division wins in two-stream settings with relaxed bandwidth (Gemini temporal single-stream 79.62 is the highest cell in the table), and semantic division shows its advantage at ≥3 streams under tight token budgets (at N=4, Gemini semantic 41.25 vs spatial 30.48). These map onto the inherent drawbacks of frequency/time/code division in signal theory.
• Proactive timing is a universal weakness: nearly all models post high ER (Qwen3-Omni 98.81, VideoLLM-online 99.10) — they cannot resist answering early — while only MMDuet2 swings to the opposite extreme (NR 54.11, mostly silent). The large gap between Forward and Overall shows current architectures couple "perception" and "when to speak" poorly.
• Multi-stream necessity holds (original Table 7): single-stream input scores only 5.35 on X-Stream (Qwen3-Omni), while injecting distractor streams into single-stream datasets (StreamingBench, OVO-Bench) degrades 57.19/56.16 to 26.10/34.78 — X-Stream measures a genuinely new capability, not an extension of single-stream systems.
• Audio multiplexing is its own problem (original Table 6): spatial division superimposes audio tracks and causes overlap; temporal division avoids overlap but cuts speech continuity. Because audio is more strongly coupled than pixels, simple schemes trade wins (Qwen3-Omni spatial division drops 34.28→29.40 without audio, while temporal division rises 26.57→30.37).

Highlights & Insights

• Necessity verification is a transferable data-engineering pattern. Any multi-input benchmark (multi-document, multi-sensor, multi-agent) can harden "must use all inputs" into a checkable constraint — an individual input must fail — which is far more reliable than post-hoc manual shortcut filtering.
• Signal multiplexing theory as an analysis framework. Spatial/temporal/semantic division inherit the trade-offs of frequency/time/code division (resource waste vs latency-jitter vs code interference), giving "why each wins where" a priori structure instead of a purely empirical tuning conclusion — directly actionable guidance for next-generation multi-stream agent architectures.
• ER/NR turns timing into a diagnosable metric. Reporting the jump-the-gun rate and the silent rate separately localizes failure modes (does the model not understand, or not dare to speak, or cannot stop speaking) far better than a single Forward score.

Limitations & Future Work

• The stream count is bounded: despite raw material with up to 10 streams, the benchmark keeps 2-5; the regime where semantic division wins (≥3 streams) covers only about 20% of data, so that conclusion is extrapolated at higher stream counts.
• Some subtasks are evidently small (counterfactual scores are widely spaced discrete values), making per-task conclusions fragile; the 0.62 LLM-judge-to-human correlation is moderate, and the impact of judge noise on fine-grained rankings is unquantified.
• Semantic division relies on an auxiliary visual encoder for token pruning, and on proprietary models it can only be approximated by converting the most-retained stream back into frames — an approximation error across strategies.
• All three strategies are inference-time schemes; architectures trained natively for multi-stream (stream-identifier embeddings, cross-stream attention) remain unexplored. The authors also note that public datasets lack film-grade timestamp synchronization, and inter-stream drift is a long-term data-side hazard.

Related Work & Insights

• vs StreamingBench / OVO-Bench / ProactiveVideoQA: these streaming benchmarks do streaming and proactivity well but are entirely single-stream; X-Stream's increment is not "harder" but the new dimension of timestamp-aligned multi-stream plus a dual-verification data protocol.
• vs EgoLife / WaymoQA / MVU-Bench: multi-view/multi-video benchmarks process complete video files offline; X-Stream's pyramid organizes Multi-Video→Multi-View→Multi-Angle by constraint strength, with Multi-Stream wedged in the middle carrying timestamp alignment as its unique constraint — a clean positioning.
• vs VideoLLM-online / Dispider / MMDuet2: dedicated streaming models perform worst on this benchmark (6.79-15.44), showing their training distribution is badly mismatched to multi-stream scenes; X-Stream's multiplexing analysis hands them a concrete list of improvement directions.

Rating

• Novelty: 5/5. The first multi-stream streaming benchmark fills a real gap; both the "multiplexer" conceptual framework and the dual-verification pipeline are new contributions.
• Experimental Thoroughness: 4/5. Eleven models, online inference, and strategy × stream-count × audio ablations; yet some subtasks are small, the semantic strategy on proprietary models is only approximated, and cross-strategy conclusions partly rest on small samples.
• Writing Quality: 4/5. The pyramid positioning, multiplexing-theory mapping, and experimental conclusions connect smoothly; some tables are heavy to read under two-column layout.
• Value: 5/5. For real-time multi-stream agents (live directing, multi-screen collaboration, multi-camera robotics) it provides the first systematic diagnostic tool and design guide.