ECCV2026 Video Understanding Multimodal VLM video benchmark perception-centric reasoning compositional question answering test-time reasoning multimodal LLM

PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning¶

Conference: ECCV2026
Paper: ECCV
Area: Video Understanding / Multimodal VLM
Keywords: video benchmark, perception-centric reasoning, compositional question answering, test-time reasoning, multimodal LLM

TL;DR¶

PerceptionComp builds 1,114 five-way multiple-choice video questions (279 high-scene-complexity real-world videos, 100% manual annotation) by making several perceptual subconditions jointly necessary, so that no single moment or single viewing suffices; humans reach 100% with unrestricted rewatching but drop to 18.97% in a single-view setting, while the best MLLM (Gemini-3-Flash) reaches only 45.96%.

Background & Motivation¶

Video QA benchmarks are already numerous, but they split into two families that miss the same thing. The first family is perception-centric (VideoMME, Perception Test, LongVideoBench): the visual content is real, yet most questions can be answered after one viewing with little deliberation, so the gap between models reflects how much they remember rather than whether they can repeatedly retrieve evidence. The second family is logic-heavy (VideoMathQA for mathematical reasoning, VSP for spatial planning and mazes): difficulty is real, but it comes from the logical structure itself, since the visual input is synthetic or drastically simplified — getting the answer wrong has little to do with seeing the evidence clearly. Long-video benchmarks (LongVideoBench, LVBench) press on context length and narrative memory instead: longer videos are harder, but what is hard is remembering, not going back to find evidence.

This leaves an obvious gap: no benchmark is simultaneously long-horizon, perception-centric, and genuinely forcing repeated evidence gathering. The reason it had not been built is not a lack of interest but an underestimated construction cost — for a question to be truly unsolvable by a single frame or a language prior, the annotation stage must establish two things: the answer is uniquely determined by the video, and removing any one subcondition makes it no longer unique. That verification can only be done by hand, question by question; automatic template generation cannot do it. The cheap way to raise difficulty had been to lengthen videos, which does not touch the orthogonal axis of perceptual difficulty. Recent progress in test-time scaling for MLLMs is what makes the benchmark worth building now: if thinking longer demonstrably helps mathematical and code reasoning, the field needs a diagnostic that separates "did not think enough" from "never actually saw it," otherwise there is no way to decide whether compute should go to perception or to reasoning.

The paper pushes difficulty directly into the construction constraints. At the video level it does not stack duration but selects clips with high scene complexity using the number of SAM2-detected instances and optical-flow magnitude as proxies, deliberately favoring many objects, intense motion, and frequent cuts — videos that a single caption cannot replace. At the question level, each item composes several perceptual subconditions under two logics: conjunctive (all subconditions refer to the same target, and no proper subset uniquely determines the answer) and sequential (later subconditions depend on intermediate entities established earlier), forcing the solver to carry referents back and forth across temporally separated evidence. Core idea: make "repeated perception is mandatory" a hard construction constraint — select videos by scene complexity rather than duration, assemble questions under conjunctive/sequential logics, manually verify answer uniqueness and subcondition necessity for every item, and then prove with a single-view human study that the shortcuts really are blocked.

Method¶

Overall Architecture¶

PerceptionComp produces a static evaluation set, so its "method" is a three-stage pipeline plus an evaluation protocol: video selection → subcondition design and question assembly → fully manual annotation and difficulty grading, yielding 1,114 five-way multiple-choice questions over 279 videos. Evaluation reports only five-way accuracy (chance is 20%), with three human baselines and three controlled analyses attached.

Video selection decides what the material is, subcondition composition decides where the difficulty lives, and the annotation protocol decides whether the answers can be trusted. The three constrain each other: if the videos are not cluttered enough, no amount of subconditions can stop a single coarse glance from working; if annotation lacks cross-checking, hard compositional items are prone to ambiguous answers and the benchmark's validity collapses. On the evaluation side, models with native video interfaces (e.g., Gemini) are fed raw video, all others get 64 uniformly sampled frames (some GPT APIs use 50 because of input-length limits); proprietary models are prompted with chain-of-thought (CoT), open-source instruction-tuned models are asked to output the choice directly, and open-source thinking models are prompted with CoT (temperature 0.7, max generation length 16,384 tokens). Three extra controlled experiments (input frames, thinking-token budget, active rewatching) separate insufficient visual evidence from insufficient deliberation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["2–10 min real-world videos<br/>city walk / mall / sports / villa / variety / movie / game stream"] --> B["High scene-complexity video selection<br/>SAM2 instance count + optical-flow magnitude"]
    B --> C["Subcondition composition<br/>conjunctive / sequential logics"]
    C --> D["Same-category five-way choices<br/>distractors share the answer category"]
    D --> E["Fully manual annotation + cross-check<br/>unique answer + necessary subconditions"]
    E --> F["279 videos / 1,114 questions<br/>+ three difficulty levels"]
    F --> G["Evaluation: five-way accuracy<br/>human baselines / MLLMs / controlled analyses"]

Key Designs¶

1. High scene-complexity video selection: move difficulty from the duration axis to the content axis

Existing perception benchmarks often use clips depicting a single event with a handful of subjects, and the paper's judgment is blunt: such videos can be approximately replaced by a one-sentence caption without changing downstream performance much, so they cannot diagnose perceptual competence. PerceptionComp therefore selects by scene complexity: seven source categories — city-walk tours (outdoor), shopping in malls, sports competitions, indoor home/villa tours, variety shows, movie clips, and game livestreams — with clips of 2–10 minutes, explicitly requiring many objects, frequent scene transitions, and substantial camera motion. Complexity is not a human hunch but two automatic proxies: the number of instances detected by SAM2 (object density) and optical-flow magnitude (motion intensity and scene-change dynamics, with RAFT cited as the flow estimator), prioritizing clips that score high on all three. All videos are real recordings rather than synthetic renderings; one category (game livestreams) is screen-captured, but it still exhibits naturally occurring dynamics and clutter.

This is an axis orthogonal to "make the video longer," a point the paper stresses repeatedly: most PerceptionComp videos are not longer than 10 minutes, yet the benchmark takes far more human time than longer ones (see Table 2), showing that video thinking has more difficulty dimensions than context length.

2. Subcondition composition: conjunctive and sequential logics that make a single moment insufficient

Question difficulty is explicitly split into "how hard each subcondition is" and "how subconditions are glued together." At the subcondition level, the paper defines six perceptual skills and requires each question to draw on several of them: semantic understanding (recognizing object categories, attributes such as shape, color, and material, and higher-level semantic relations such as roles or interactions), spatial understanding (scene layout and relative geometry such as left/right, front/behind, near/far, plus occlusion), temporal understanding (following motion patterns and localizing events in time, e.g. what happens before or after a reference event), correspondence (matching instances or parts across time and views, e.g. tracking the same object across shots or doing part–whole matching), visual knowledge (commonsense tightly coupled to the visual content), and world modeling (simple near-future prediction from ongoing dynamics).

Two composition logics each block a different shortcut. Conjunctive composition points all subconditions at the same target, forming an "and" chain; crucially, the paper verifies that no proper subset uniquely determines the answer — each condition only removes part of the candidate set, and only the full conjunction collapses it to a single solution, so a model cannot ignore part of the query and still be right. Sequential composition requires subconditions to be resolved in order, with later ones depending on intermediate entities or states established earlier: a first subcondition identifies an object, a second constrains its behavior at a later time, and a third asks about a relation involving that same object after another event. The referent must be carried forward step by step, inducing multi-hop perceptual reasoning in which an early misreading propagates — which is exactly the source of the mid-chain error concentration reported in the failure analysis.

3. Same-category five-way choices: squeeze the language prior out of the answer space

The final answer is one of six kinds of perceptual information: objects (category names such as "car" or "sofa"), attributes (color, count, shape), relationships (semantic, spatial, or social relations between entities), location (place descriptors such as room type, country, or region), action (the name of an action performed by an agent), and event (a higher-level composite situation in the video). Every question is a five-way choice, but the distractors are not arbitrary: all options are drawn from the same answer category as the correct one (all colors, or all object categories), and each option is constrained to a single word or a very short phrase.

The purpose is to remove option-level linguistic cues. If distractors mixed categories or differed visibly in length, a model could pick the right answer because it "reads like" an answer or fits a commonsense collocation, and the item would stop testing video evidence. The metric is deliberately plain — five-way accuracy, with chance at 20%. That baseline is empirically confirmed by the single-view human study (18.97%): humans who cannot rewatch are essentially guessing, so the option design leaves no visible bias.

4. Fully manual annotation with cross-checking: guaranteeing a unique answer and necessary subconditions

Highly compositional items are prone to ambiguity, so PerceptionComp is 100% manually annotated: a single question takes 10–20 minutes from video selection to final annotation. Annotators first write the subconditions and the final answer, then self-check that the answer is uniquely determined by the video and that every subcondition is genuinely necessary; each question is then independently checked by at least one annotator who did not create it, confirming again that a single correct answer exists and that no proper subset of subconditions suffices, with failed items revised or discarded.

Difficulty grading is also done by experts, based jointly on the number of composed subconditions and the intrinsic difficulty of those subconditions — the paper explicitly avoids treating difficulty as a function of subcondition count alone. Statistically (Figure 2(b) of the original), higher difficulty levels do contain more subconditions, reflecting increased compositional depth and stronger long-horizon demands, but the correspondence is not definitional.

A Worked Example: from the glass cube to the cyclist's vest color¶

The example in Figure 1(a) shows how sequential composition works. The task is: ① locate the store with a giant glass cube; ② on the street next to that glass cube there are two food trucks of the same color; ③ when the camera passes the food truck closer to the glass cube, an SUV of the same color as the food truck appears on the right-hand side of the road; ④ what is the color of the vest of the cyclist passing by this SUV at that moment? The four steps form a strict forward dependency: the store determines the street segment, the street segment plus "closer to the glass cube" determines which food truck, the truck's color determines the SUV's identity, the SUV determines which frame counts as "that moment," and only then does reading the vest color become well posed.

The human trace in Figure 1(b) mirrors this progressive narrowing: find the glass cube → locate the two yellow food trucks → find the truck near the crossing → determine its position relative to the glass cube → find the yellow SUV on the right → then find the light-green cyclist. Each step builds on the previous localization; if any step locks onto the wrong target, the rest of the chain can stay internally coherent while drifting away from ground truth — precisely the pressure sequential composition is meant to apply.

Key Experimental Results¶

Main Results¶

The paper evaluates three human baselines and 18 MLLMs on the full 1,114 questions (Table 2 of the original). The human side is the most direct evidence: an expert with unrestricted rewatching and sufficient time reaches 100.00%, ordinary participants (allowed to rewatch but prone to losing patience) reach 85.10%, and humans allowed to watch only once with no rewatching drop to 18.97%, near the 20% chance line of five-way choice.

Model	Size	Frames	Five-way accuracy (%)
Expert (unrestricted rewatch, ample time)	-	-	100.00
Human (rewatching allowed)	-	-	85.10
Single-view Human (no rewatch)	-	-	18.97
Gemini-3-Flash [10]	-	raw video	45.96
Gemini-3-Pro [10]	-	raw video	44.43
Gemini-2.5-Pro [6]	-	raw video	44.34
Seed2.0 Pro [3]	-	64	44.34
Gemini-3.1-Pro [10]	-	raw video	43.72
GPT-o3 [29]	-	50	43.54
GPT-5.2 [28]	-	64	40.75
Gemini-2.5-Flash [6]	-	raw video	38.15
GPT-5 [26]	-	64	36.45
GLM-4.5V [37]	106B	64	36.69
GPT-4.1 [27]	-	50	34.02
Qwen3-VL [1]	30B	64	34.38
Qwen3-VL [1]	8B	64	34.06
Qwen3-VL-Thinking [1]	235B	64	38.20
InternVL-3.5 [40]	8B	64	32.32
Qwen2.5-VL [2]	72B	64	31.33
GPT-4o-latest [16]	-	50	32.50
VideoChat-R1 [21]	7B	64	28.63
Video-R1 [8]	7B	64	26.27
Qwen2.5-VL [2]	7B	64	22.73

⚠️ Gemini-3 / 3.1, GPT-5 / 5.2, Seed2.0 Pro, and GLM-4.5V are the version names used at the time of evaluation in the original paper; refer to the original and to the official model cards. This table takes only the Overall column of Table 2 — the per-category and per-difficulty columns are misaligned in the cached text and were not reconstructed column by column. Note also that Sections 4.2/4.3 of the original write Qwen3-VL-8B as 34.80%, while Table 2 reports 34.06% for that row; the two disagree, so 34.06% from Table 2 is listed here. ⚠️ Refer to the original paper.

The human time comparison makes the same point. On PerceptionComp the average answering time is 144 seconds, more than 2× VideoMMMU (65 s), more than 5× Minerva (32 s), LongVideoBench (27 s), and Video-Holmes (26 s), more than 10× VideoMME (13 s), and about 18× Perception Test (8 s) — even though PerceptionComp videos are mostly not longer than those benchmarks'.

Benchmark	Avg. human answering time (s)	Single-view human accuracy here
Perception Test	8	-
VideoMME	13	-
Video-Holmes	26	-
LongVideoBench	27	-
Minerva	32	-
VideoMMMU	65	-
PerceptionComp	144	18.97%

Ablation Study¶

There are no modules to remove, so the substitute is three controlled analyses that separate "not enough visual evidence" from "not enough deliberation." All three run on a fixed subset of 100 videos (500 questions) because of evaluation budget, so their absolute numbers are not directly comparable with the full-set table above.

Variable	Setting	Model	Accuracy change
Input frames	16 → 32 → 50	GPT-o3	34.0 → 43.54 (+9.5 pts, monotone)
Input frames	16 → 32 → 64	Qwen3-VL-8B	27.0 → 34.80 (+7.8 pts, monotone)
Thinking-token budget	1,024 / 2,048 / 4,096 / 8,192	Gemini-2.5-Flash	monotone increase (original gives curves only, no values)
Active rewatching (LongVT tool protocol)	fixed input → optional segment revisiting	Qwen2.5-VL	22.73 → 28.80 (+6.07 pts)
Active rewatching (lightweight agentic wrapper)	fixed input → optional segment revisiting	Gemini-3-Flash	45.96 → 50.60 (+4.64 pts)
Thinking vs. instruct	GPT-4o → GPT-o3	proprietary pair	+11.04 pts
Thinking vs. instruct	Gemini-2.5-Flash → Gemini-2.5-Pro	proprietary pair	+6.19 pts
Thinking vs. instruct	Qwen3-VL-8B → Qwen3-VL-Thinking-8B	open-source pair	34.06 → 33.82 (down 0.24 pts)

Key Findings¶

Both perception and reasoning budgets are real bottlenecks, and they are complementary in direction. More input frames lift GPT-o3 from 34.0% to 43.54% and Qwen3-VL-8B by 7.8 points, confirming that the questions depend on aggregating evidence across many frames and several temporally separated moments rather than on a few salient frames; enlarging the thinking-token budget improves accuracy monotonically as well, indicating that items also require maintaining intermediate hypotheses without premature commitment. Even combined, this only reaches around 50%, far from humans (85.10% with rewatching, 100% for experts).
Stronger language-side reasoning does not automatically mean stronger perception. On proprietary models the thinking variants win consistently (GPT-o3 over GPT-4o by 11.04 points, Gemini-2.5-Pro over Flash by 6.19 points), but the open-source side shows a counterexample: Qwen3-VL-Thinking-8B (33.82%) is slightly below the instruction-tuned Qwen3-VL-8B (34.06%). The paper's reading is that when the perceptual evidence is misread or underspecified, a longer reasoning chain amplifies the error rather than correcting it, because PerceptionComp requires every intermediate conclusion to be re-grounded in temporally separated visual evidence and uncertainty compounds over multi-hop chains.
Scale is not the cure. On the open-source side the 8B/30B/235B Qwen3-VL variants score 34.06/34.38/34.02, not a monotone trend in parameters; the bottleneck looks like stably extracting fine-grained evidence under clutter and temporal discontinuity rather than generic capacity.
A shared ceiling appears across difficulty levels. The paper reports substantial drops on Level 3 items (more subconditions and harder ones), which require holding several intermediate hypotheses while repeatedly gathering evidence from different moments; this echoes the human baselines, where even people need sustained effort and repeated verification to reach perfect accuracy.
Strong frontier models cluster in a narrow 40–46 band. Despite different architectures and interfaces, the Gemini-3 variants and GPT-o3 all land in the mid-40s, which the paper reads as a common bottleneck in perception-centric long-horizon video reasoning rather than one vendor's engineering gap.
Failures peak mid-chain and are largely spatial. Decomposing each question into a fixed sequence of subcondition-solving steps and locating the first step where the model's intermediate conclusion diverges from the expert trace, the failure rate is only 5% at step 1, 20% at step 2, peaks at 40% at step 3, then 25% at step 4 and 10% thereafter; expert review attributes 60% of these mid-stage failures to violated spatial subconditions (e.g., incorrect 3D spatial relationships). The typical pattern is anchoring on an object that partially matches identity keywords but violates a critical spatial or temporal constraint, after which the rest of the chain drifts coherently.

Highlights & Insights¶

Using humans rather than models to argue for the benchmark's validity is the paper's best move. The "single-view humans reach only 18.97%" control turns "repeated perception is mandatory" from a design intention into a measurable property, and incidentally confirms the five-way options carry no visible prior (near the 20% chance line). Any diagnostic benchmark can copy this argument: first show human degradation under a restricted setting, then interpret model scores.
Turning "complexity" into reproducible automatic proxies. Filtering videos with SAM2 instance counts and optical-flow magnitude converts "high scene complexity" from an adjective into two computable thresholds — directly transferable to any work that needs difficulty-based sampling of video/image data, including RL training-data curation.
The "no proper subset suffices" check is the key anti-shortcut rule for compositional items. It upgrades "every subcondition must matter" from writing advice into a verifiable construction constraint, applicable to any multi-constraint synthetic data (tool-use traces, multi-hop QA, agent tasks): keep only the samples where dropping any one constraint makes the answer non-unique.
A counterintuitive empirical finding: longer reasoning can amplify perceptual errors. Qwen3-VL-Thinking-8B scoring below its instruction-tuned sibling, together with the mid-chain error peak, points to a transferable judgment — on multi-hop perception tasks the payoff of reasoning budget is conditional on per-step perceptual accuracy, so fixing perception before adding CoT is the better order of operations.

Limitations & Future Work¶

Small scale. 279 videos / 1,114 questions is modest next to MMVU (1,529 videos / 3,000 questions) or VideoMME (900 / 2,700). The authors justify this with 100% manual annotation at 10–20 minutes per question, but a small set means limited samples per video category or difficulty level once stratified, weakening statistical power for fine-grained claims.
A single metric and no process metric. Evaluation reports only five-way accuracy, even though the failure analysis already uses a more informative protocol — decomposing a question into subcondition steps and locating the first divergence from the expert trace — which appears only in the supplementary and is not a public evaluation metric. Adding "per-subcondition accuracy" or a "first-error position" distribution to the leaderboard would point at what to fix far better than a total score.
The "real video" boundary needs stating. Although the authors stress that all videos are real recordings rather than synthetic renderings, the seven source categories include game livestreams and variety-show clips, whose visual statistics differ from the real deployments named in the motivation (robots, AI glasses); the "in-the-wild" framing has some tension here.
Difficulty grading lacks a consistency report. Levels 1/2/3 are assigned by experts based jointly on subcondition count and intrinsic difficulty, but no inter-annotator agreement figures are given, nor is it explained how intrinsic subcondition difficulty is quantified — making the difficulty stratification less reproducible than the "unique answer + necessity" protocol, which is documented carefully.
Residual option-bias risk in five-way choice. Same-category, single-word distractors are a reasonable mitigation, but no analysis of position/order bias is reported, and it is not shown whether some models answer via elimination rather than video evidence.
The controlled analyses need clear boundaries. All three (frames, thinking tokens, active rewatching) are run on a fixed subset of 100 videos / 500 questions and cannot be compared directly against the full-set numbers; whether that subset is distributionally matched to the full set is also not discussed.
Concrete improvements: make active rewatching a first-class setting that reports token/frame budget alongside accuracy, and plot the perception-vs-reasoning allocation curve at matched compute; add subcondition-level error attribution metrics; and test stability under multilingual question phrasing (currently all questions are English).

vs VideoMME / Perception Test / MMVU: these are perception-centric or expert-knowledge benchmarks with real visuals, but their questions can often be answered from a few salient moments. PerceptionComp differs by writing "evidence from temporally separated segments must be aggregated" into the construction constraints and quantifying it with 18.97% single-view human accuracy — not through longer videos but through denser constraints.
vs LongVideoBench / LVBench: long-video benchmarks press on context length and narrative memory, where longer means harder. PerceptionComp videos are mostly 2–10 minutes and difficulty comes from scene complexity and subcondition composition on an orthogonal axis; the human time contrast (144 s vs 27 s) shows "thinking longer" and "watching longer" are different things.
vs VideoMathQA / VSP: their difficulty comes mainly from logical structure (mathematics, mazes/spatial planning) with synthetic or heavily simplified visual input. PerceptionComp inverts this, making the visual evidence itself the irreplaceable bottleneck — which is also the precondition for diagnosing "insufficient perception" separately from "insufficient reasoning."
vs Video-Holmes / VCR-Bench / MINERVA: these complex video-reasoning benchmarks emphasize narrative, causal, or human-centric reasoning, often with automatic plus manual annotation. PerceptionComp is 100% manual and verifies both answer uniqueness and subcondition necessity per item, at the cost of smaller scale and higher per-item cost.
vs reasoning and tool-using methods such as Video-R1 / VideoChat-R1 / LongVT: these are evaluated rather than competed against; the finding is that explicit reasoning (VideoChat-R1) and active rewatching (LongVT and the agentic rewatch wrapper) both help, yet remain far from human level on PerceptionComp, showing that "being able to call a tool to rewatch" is not the same as "reliably grounding and composing evidence across segments."

Rating¶

Novelty: ⭐⭐⭐⭐ The benchmark concept is not new, but the construction-and-verification recipe (conjunctive/sequential subcondition composition, no-proper-subset verification, single-view human control) is a substantive advance that turns perception-centric difficulty into a checkable hard constraint.
Experimental Thoroughness: ⭐⭐⭐⭐ Coverage of 18 models across proprietary/open-source and instruct/thinking variants, plus three human baselines and three controlled analyses, is rare; points off for running the grouped analyses on a 500-question subset only and for the missing agreement report on difficulty grading.
Writing Quality: ⭐⭐⭐⭐ The motivation chain is clear (what each benchmark family lacks → construction constraints → human control validation) and the mid-chain error distribution is convincing; the cached tables have scrambled column ordering and a few prose numbers disagree with Table 2.
Value: ⭐⭐⭐⭐ A rare clean testbed for test-time perception scaling: scores cluster in a narrow band and errors concentrate in perception and mid-chain steps, which directly informs the judgment of where to spend perception-side compute.