Video-Holmes: Can MLLM Think like Holmes for Complex Video Reasoning?¶
Conference: ECCV2026
Paper: ECCV Paper
Project: Video-Holmes
Area: Multimodal Reasoning / Video Understanding
Keywords: multi-clue reasoning, active evidence seeking, suspense short films, chain-of-thought, reasoning diagnosis
TL;DR¶
Video-Holmes tests whether models can independently locate and connect scattered clues through 1,837 questions about 270 suspense short films; Gemini-3.0-Pro leads the main table at 49.6%, compared with 96.3% for humans watching each film once.
Background & Motivation¶
“Reasoning” in video question answering does not always entail understanding relationships between events. When a question already specifies a timestamp, object, or action, a model may answer from a local observation without explaining the story. Even a long chain-of-thought (CoT) can amount to a verbal elaboration of an explicitly supplied visual clue. General video accuracy therefore struggles to separate recognizing visual content from drawing an inference across multiple pieces of evidence. The paper asks whether models can identify which shots matter when the question does not directly reveal the crucial clues.
Suspense short films offer compact test material: foreshadowing, misdirection, cuts, and final twists distribute relevant evidence across time. Explaining why a character turns back, how two people are related, or why an ending occurs often requires linking an earlier action to a later outcome. Supernatural settings also require inferring rules from events within the film instead of rejecting unusual events through everyday knowledge. The value is not strangeness itself, but the availability of several clues that must be explained jointly to support one conclusion. The authors therefore annotate plots manually and design questions around evidence relationships rather than generating questions from arbitrary frames.
This changes the interpretation of failure: misidentifying an object, missing an event, and observing events but linking them incorrectly deserve separate diagnoses. A correct option is still insufficient evidence of a reliable process, since a model may infer the right theme from an invented plot. The paper consequently builds questions, analyzes explanations, and uses a separate auxiliary training set to test benefits for other video tasks. Core Idea: move video evaluation from answering with supplied clues to actively locating and connecting clues, then use human evidence references to distinguish perception, omission, and reasoning errors.
Method¶
Overall Architecture¶
This is primarily a benchmark paper, not a new reasoning model named Video-Holmes. Dataset construction takes short films as input, builds timestamped plot and evidence annotations, and generates questions, options, answers, and explanations. Model evaluation takes video or sampled frames together with a question and options, and produces a stepwise explanation followed by an answer. A diagnostic evaluator then combines the human plot, reference explanation, and model response to categorize the outcome. Human annotations thus support question construction and diagnosis; they are not default hints supplied to models in the video evaluation. Only the text-input ablation explicitly supplies human plots or generated descriptions to language models. The test set contains 270 films and 1,837 questions; the separate auxiliary training set contains 233 films and 1,551 questions. The four main designs are evidence annotation, seven task types, question quality control, and reasoning diagnosis; reinforcement learning tests the utility of the data.
Key Designs¶
1. Evidence annotation: establish traceable plot references first
The authors search YouTube for suspense short films lasting 1 to 5 minutes, using 8 sub-keywords to diversify the material. The initial pool exceeds 2,500 videos, with manual filtering for overly simple plots, insufficient inferential clues, and poor production quality. The goal is to retain stories with genuine evidence chains rather than merely collect videos that models answer incorrectly. Retained videos are divided into narrative segments, each described in text to establish the context of events. Annotations also record key character relationships and supporting evidence instead of treating character identity as self-evident. This turns the basis for question generation from isolated image labels into event descriptions that can be checked against the film.
The crucial “reasoning shots” annotation records timestamps, visual clues, and the conclusions supported by those clues. Annotators also describe the rules and effects of supernatural elements and summarize the film's core theme. These fields place observable facts, fictional rules, and higher-level interpretation within one structured reference, although a theme is not a purely visual fact. Trained annotators with at least an undergraduate degree perform the initial annotation, and senior staff randomly review 20%. If an annotator's submission is too brief or logically flawed, all of that annotator's assignments must be revised. This workflow controls reference quality; it does not by itself establish that every question admits no interpretive ambiguity.
2. Seven task types: trace scattered evidence through different relationships
Social Reasoning (SR) identifies relationships, including identity connections between a character's younger and older appearances. Physical Anomaly Reasoning (PAR) explains scenes that violate ordinary physical expectations and infers implicit rules within the film. Multimodal Hint Reasoning (MHR) interprets meaningful signals such as camera movements and changes in object positions. Intention & Motive Chaining (IMC) distinguishes a character's surface behavior from its underlying purpose. These tasks ask what an action contributes to the narrative rather than merely which action category is visible. For example, interpreting a character touching a pocket may require a later shot of a missing object to explain why the character turns back.
Temporal Causal Inference (TCI) connects events across times and scenes to determine the mechanism behind an outcome. Timeline Analysis (TA) reconstructs event order rather than simply repeating the sequence in which edited shots appear. Core Theme Inference (CTI) integrates plot, dialogue, and symbolism to identify a deeper message. All seven tasks require locating and connecting relevant clues, but emphasize different relationships. “Active seeking” here describes a task requirement, not an implemented retrieval agent that interactively revisits the video. In the main experiments, most evaluated models still receive predetermined sampled frames or native video input.
3. Question quality control: constrain generation with human evidence
DeepSeek-R1 generates questions from structured human annotations and predefined tasks, together with detailed answer explanations. Generation is instructed to follow the annotations strictly rather than substitute outside assumptions for evidence within the film. Reference explanations describe how multiple clues support the correct option, allowing diagnosis beyond answer-letter matching. After generation, the authors manually audit the entire question set and regenerate or correct erroneous and ambiguous items. The main text reports initial problems in “approximately 1% (46 pairs)” but does not clearly specify the denominator for that percentage. This note preserves that wording rather than reinterpreting 46 pairs as a fixed error rate within the final test set.
The workflow reduces factual deviations and ambiguity in generated questions; it does not guarantee the absence of language-model preferences. Using the same model family to produce reference explanations and diagnose responses may favor particular explanatory styles. The authors therefore add human studies of diagnostic reliability and task difficulty rather than relying only on automated judgments. The difficulty study samples 100 questions from each of three benchmarks and asks 10 evaluators to judge clue requirements and difficulty. The Active Seeking percentage measures questions that require searching for visual evidence instead of relying on predefined textual clues. The Chain-of-Clues percentage measures questions requiring multi-step logical connections; neither percentage is model answer accuracy.
4. Reasoning diagnosis: separate option correctness from evidence quality
Incorrect answers are categorized as Visual Perception Error (VPE), Visual Omission Error (VOE), Reasoning Error (RE), or Think Right Answer Wrong (TRAW). VPE means extracting incorrect visual information, VOE means missing a key object or event, and RE means making a logical mistake such as linking clues incorrectly. TRAW describes reasoning that broadly matches the reference explanation but ends in an incorrect answer, exposing an output-consistency problem. Correct answers are divided into Think Right Answer Right (TRAR) and Think Wrong Answer Right (TWAR). TWAR matters because selecting the correct option does not cancel out invented character relationships or an incorrect plot explanation. These labels assess agreement between visible responses and reference evidence, not direct observation of a model's internal reasoning.
DeepSeek receives the human plot, question, reference explanation, available model reasoning, and category definitions to perform diagnosis. Table 4 reports error-type ratios within incorrect answers and success-type ratios within correct answers, using different denominators. The authors identify omissions and reasoning errors as the main bottlenecks, but few perception errors do not imply that every model observed the whole video. An omission may arise from sparse sampling or attention selection, and the taxonomy does not fully separate these causes. Human verification uses 100 randomly sampled responses and 10 evaluators; Table 7 reports 95.0% overall agreement and Cohen's \(\kappa=0.90\). This supports the usefulness of automated diagnosis, but the sample is insufficient to establish equal reliability across every task and model.
Loss & Training¶
The main evaluation uses officially released checkpoints rather than training a new network on test questions; prompts request a stepwise explanation before the answer. Models without native video support generally receive uniformly sampled frames with timestamps; Table 2 mostly uses 32 frames, Claude uses 20, and Gemini uses native video. The human baseline averages 5 evaluators who read the questions first, watch each film exactly once, and then answer. The auxiliary experiment applies reinforcement learning post-training to Qwen2.5-VL-7B on the separate training set, following the GRPO-CARE protocol. The cached main text does not provide complete reward definitions, hyperparameters, or compute budgets; a generic GRPO equation would not constitute a new loss proposed here. The cache ends with the references and omits the cited Appendices A, B, and D, so detailed prompts and implementation settings cannot be verified from this material.
Key Experimental Results¶
Main Results¶
The following selection comes from Table 2, page 9; all metrics are accuracy (%), with TA denoting Timeline Analysis and TCI denoting Temporal Causal Inference. These are the main table's input conditions and should not be combined with later audio-enabled ablations into a single uniform leaderboard.
| Model | Video input | TCI | TA | Overall |
|---|---|---|---|---|
| Qwen2.5-VL-7B | 32 frames | 17.6 | 30.0 | 27.8 |
| Video-R1 | 32 frames | 28.9 | 34.5 | 36.5 |
| Video-RTS | 32 frames | 33.3 | 43.0 | 40.0 |
| Qwen3-Omni-30B-A3B | 32 frames | 38.4 | 55.2 | 46.2 |
| Gemini-2.0-Flash | Native video | 23.1 | 20.5 | 30.6 |
| Gemini-2.0-Flash-Thinking | Native video | 43.1 | 51.0 | 43.1 |
| Gemini-3.0-Pro | Native video | 48.5 | 54.0 | 49.6 |
| Human | Single viewing | 99.6 | 97.9 | 96.3 |
Gemini-2.0-Flash improves from 30.6% to 43.1% with its Thinking version: 12.5 percentage points, not a relative improvement of 12.5%. The overall gap between the best model and humans is 46.7 percentage points, although their input formats and observation strategies are not identical.
Ablation Study¶
The following selection comes from Table 5, page 12; accuracy is in percent, and text-input rows evaluate language models reading descriptions rather than directly viewing video.
| Model | Intervention and condition | Overall |
|---|---|---|
| Qwen2.5-VL-7B | 32 frames, with CoT | 27.8 |
| Qwen2.5-VL-7B | 32 frames, without CoT | 29.4 |
| Qwen2.5-VL-7B | 80 frames | 33.0 |
| Qwen2.5-VL-7B | 1 fps | 42.5 |
| Video-R1 | 32 frames, with CoT | 36.5 |
| Video-R1 | 32 frames, without CoT | 28.2 |
| Gemini-2.5-Pro | Without audio | 45.0 |
| Gemini-2.5-Pro | With audio | 51.3 |
| DeepSeek-R1 | Frame-level captions, FLC | 31.2 |
| DeepSeek-R1 | Video-level captions, VLC | 64.6 |
| DeepSeek-R1 | Human annotations, HA | 92.0 |
Human annotations already supply logical relationships between key events, so 92.0% does not show that a text-only model is better at seeing video. Video-level captions are generated by Gemini-2.5-Pro, whereas frame-level captions come from Qwen2.5-VL; the comparison changes both granularity and captioner capability. Table 5(a) labels the Qwen2.5-VL gain at 1 fps as +14.9, but subtracting 27.8 in Table 2 from 42.5 in Table 5 gives 14.7 percentage points; the endpoints conflict with the reported gain.
Key Findings¶
The transfer results below come from Table 6, page 14; post-training uses 233 videos and 1,551 questions, and metrics are accuracy (%).
| Model | Video-Holmes | Video-MME | TempCompass | Video-MMMU | MVBench |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | 27.8 | 56.6 | 72.6 | 48.1 | 59.0 |
| Same model post-trained on the auxiliary set | 40.7 | 58.9 | 74.2 | 50.2 | 63.8 |
Video-Holmes improves by 12.9 percentage points and Video-MME by 2.3 percentage points, indicating benefits beyond suspense questions. Table 6 labels the TempCompass gain as +1.8, while the endpoints 74.2 and 72.6 differ by 1.6 percentage points; these conflicting reports are not silently reconciled here. Table 3 also contains arithmetic inconsistencies in reported gains, so cross-benchmark comparisons should prioritize raw endpoints over copying every parenthesized increase. The human study in Table 8, page 14, reports 98.0% Active Seeking, 96.0% Chain-of-Clues, and difficulty 4.6/5, supporting the intended emphasis on connecting multiple clues.
Highlights & Insights¶
- The benchmark makes finding clues part of the challenge rather than only describing an already localized segment. This better captures deciding which evidence matters to a narrative.
- Separating incorrect answers from incorrect explanations exposes cases where a model guesses correctly while misunderstanding the film. Process diagnosis is more actionable than accuracy alone.
- Frame-count and CoT ablations jointly show that a long reasoning trace cannot replace missing evidence. Ensuring that critical events enter the input before improving their integration is an engineering sequence worth testing.
- Cross-domain gains from a small, reasoning-intensive dataset suggest that learning event relationships may transfer. Whether it outperforms equally sized alternative data still requires budget-matched comparisons.
Limitations & Future Work¶
- Suspense, twists, and supernatural narratives provide dense clues but introduce domain bias. Theme inference and fictional-rule understanding do not represent every real-world video task.
- Models use different frame counts, native video interfaces, and modalities, mixing observation budget with reasoning ability in rankings. Comparisons should be repeated under matched input budgets.
- Human plots contain interpreted relationships, so the text upper-bound experiment cannot isolate information lost by visual encoding. A stricter study would supply factual descriptions and causal explanations separately.
- Some model category counts in Table 4 cannot be directly reconciled with the overall results in Table 2, and the main text does not fully explain the selection criteria. This note does not derive full-test error rates from those counts.
- Reinforcement learning transfer is shown for limited configurations without alternative-data controls at equal training budgets. The results support potential transfer but do not eliminate all training-volume or distribution effects.
- The paper states that the benchmark uses Apache-2.0 while source-video copyright remains with the original creators. This does not grant unconditional redistribution or commercial use of every source video.
Related Work & Insights¶
- Compared with Video-MME, MVBench, and TempCompass: these cover more general video capabilities, while Video-Holmes targets locating multiple pieces of evidence independently. It complements broad evaluation rather than replacing it.
- Compared with VCR-Bench: both examine reasoning processes, but this benchmark emphasizes clues not directly supplied by the question and evidence chains across shots. Human difficulty judgments independently support that distinction.
- Compared with Video-R1, VideoChat-R1, and GRPO-CARE: these are training or post-training methods, whereas this paper primarily contributes data and diagnostic protocols. Reusing GRPO-CARE in the auxiliary experiment does not make its optimizer a contribution of this paper.
- Research direction: measure timestamped evidence recall, evidence-link correctness, and final-answer accuracy separately. This is a reader-proposed extension, not a new metric already implemented in the paper.
Rating¶
- Novelty: 4/5. The combination of active evidence seeking and multi-clue narrative inference is well defined, with novelty concentrated in benchmark design.
- Experimental Thoroughness: 4/5. Model comparisons, input interventions, human studies, and transfer are included, but budget controls and numerical consistency remain incomplete.
- Writing Quality: 3/5. Tasks and diagnostic categories are understandable, while some reported gains and statistical scopes need clarification.
- Value: 4/5. Useful for distinguishing missed observations from incorrect clue integration, with evidence for the utility of reasoning-intensive training data.