Skip to content

How Well Do Vision-Language Models Understand Movies?

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/deep-spin/MF2-bench-can-vlms-understand-movies
Area: LLM Reasoning
Keywords: Movie narrative understanding, Vision-language models, Long-context multimodal reasoning, Contrastive claim pairs, Narrative benchmarking

TL;DR

This work introduces MF2 (Movie Facts and Fibs), a fully manually curated benchmark consisting of 53 open-licensed full-length movies and 868 human-authored contrastive fact-fib claim pairs; evaluation across 22 modern VLMs reveals that top models (Gemini 2.5 Pro at 63.8% pairwise accuracy) lag substantially behind humans (84.1%), showing that while subtitles serve as the primary narrative backbone, models struggle to identify and integrate claim-relevant visual cues dispersed across extended multimodal contexts.

Background & Motivation

Recent years have witnessed remarkable progress in vision-language models (VLMs), particularly in expanding context window lengths and performing question answering across long video sequences. However, whether these models can genuinely comprehend the narrative structure of full-length films as humans do remains an open and underexplored challenge. Narrative comprehension provides a rigorous testbed for multimodal long-context intelligence: stories are structured through evolving character arcs, intricate emotional trajectories, and causal interactions that span across tens of minutes or hours. True narrative understanding transcends single-frame spatial perception or atomic action recognition, requiring models to maintain a coherent global mental model across extended, noisy video streams while continuously integrating multimodal evidence.

Existing long-video benchmarks suffer from foundational limitations in both domain content and question synthesis paradigms. On the one hand, prevailing datasets rely heavily on general-purpose video streams such as egocentric daily recordings, instructional tutorials, or lifestyle vlogs; although these videos offer temporal duration, they lack tightly woven plot dynamics and complex narrative causality. On the other hand, prior benchmarks built on cinematic content often face severe copyright barriers that force researchers to restrict access or truncate videos into brief snippets, severely compromising scientific reproducibility. Furthermore, popular pipelines often depend on automated LLM-based question generation, which indiscriminately samples non-narrative perceptual trivia—such as the arbitrary color of an incidental background object in a single frame—rather than testing memory and comprehension of core storyline milestones. Compounding these issues, multiple-choice QA formats frequently introduce distractor artifacts and position biases, distorting the assessment of genuine truth discrimination.

To overcome these structural bottlenecks, this work establishes an open, reproducible, and cognitively grounded benchmark for movie narrative understanding. The authors curate a collection of public-domain feature films and deploy expert human annotators who watch the complete movies to author contrastive claim pairs centered strictly on memorable storyline events. Core idea: construct MF2 (Movie Facts and Fibs), a fully manually curated benchmark of 53 open-licensed full-length movies and 868 contrastive claim pairs (true facts vs. minimally edited fibs), evaluating VLMs under independent claim verification to systematically uncover their bottlenecks in long-context multimodal narrative reasoning.

Method

Overall Architecture

The MF2 construction and evaluation framework follows a three-stage pipeline of movie curation, manual annotation, and rigorous memory-based quality filtering, followed by a multi-modal isolated claim verification protocol. Specifically, the pipeline consists of: (1) collecting full-length open-licensed movies under Public Domain 1.0 accompanied by aligned subtitle tracks; (2) human expert annotation where viewers watch entire movies, identify salient narrative milestones, and formulate contrastive Fact-Fib claim pairs while categorizing their reasoning granularity, comprehension dimensions, and grounding timeframes; (3) a two-stage quality control protocol where new human evaluators verify claims purely from memory without rewatching to filter obscure or ambiguous items; and (4) an evaluation protocol administering claims independently across various input configurations (subtitles only, full-movie visual sampling, multimodal concatenation, and targeted visual spans) under strict pairwise accuracy scoring.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Public Domain Movie Selection & Subtitle Alignment<br/>53 full movies · avg. duration 88.3 min"] --> B["High-Quality Contrastive Claim Pair Construction<br/>Ground-truth Fact vs. Minimal-edit False Fib"]
    B --> C["Multi-Dimensional Reasoning & Temporal Annotation<br/>Single/multi-scene granularity · 4 comprehension axes"]
    C --> D["Two-Stage Memory-Based Quality Control<br/>Disjoint blind human recall · 104 ambiguous pairs pruned"]
    D --> E["Multimodal Ablation & Targeted Visual Probing<br/>Full-movie uniform sampling / subtitles / targeted intervals"]
    E --> F["Independent Claim Evaluation & Pairwise Scoring<br/>Eliminate distractor bias · credit only when both are correct"]

Key Designs

1. Contrastive Claim Pair Formulation: Eliminating Multiple-Choice Biases and Trivial Detail Artifacts

Addressing the noise introduced by LLM-generated questions and distractor biases in multiple-choice benchmarks, MF2 adopts a claim-based contrastive evaluation paradigm inspired by cognitive narrative assessment. Human annotators watch each movie in full and author pairs of statements centered strictly on memorable, central storyline events. Each pair comprises a factually true statement (Fact) and a counterfactual counterpart (Fib) created through minimal lexical edits that flip the truth value. Because claims are presented individually and evaluated on an isolated true/false basis, this design eliminates reliance on synthetic distractors, prevents models from exploiting surface-level option contrast shortcuts, and focuses squarely on narrative truth judgment.

2. Multi-Granularity and Multi-Dimensional Stratification: Decoupling Local Perception from Global Narrative Synthesis

To diagnose whether models fail at localized scene comprehension or at global context integration, MF2 annotates each claim pair across multiple structural axes. At the reasoning granularity level, claims are categorized into single-scene verification (49.4%) and multi-scene/global narrative reasoning (50.6%), with the latter requiring the synthesis of clues distributed across disparate scenes or inferred narrative arcs. Across cognitive comprehension dimensions, claims are labeled under event/entity tracking, temporal perception, character emotion understanding, and causal reasoning. Furthermore, for a 12% single-scene subset, annotators identify the precise visual grounding timeframes, enabling a controlled comparison between full-movie uniform sampling and targeted visual evidence feeding.

3. Two-Stage Memory-Based Quality Filtering: Grounding Benchmark Salience in Human Narrative Recall

To ensure that every claim addresses genuinely salient plot developments rather than ephemeral visual noise, MF2 institutes a two-stage validation protocol. In the first stage, 24 independent human evaluators (who did not author claims for the respective movie) watch the film once and subsequently classify the claims without access to the video, external notes, or rewatching tools, recording their confidence and flagging ambiguous or needle-in-a-haystack items. In the second stage, two senior annotators adjudicate all flagged comments; problematic, poorly phrased, or unmemorable claims are strictly discarded in their entirety rather than rewritten. This process pruned 104 flawed pairs, establishing a curated suite of 868 high-integrity claim pairs.

4. Independent Pairwise Evaluation Protocol and Multimodal Probing: Pinpointing Multimodal Integration Failures

During inference, models receive claims strictly in isolation without access to their counterfactual counterpart, preventing contrastive elimination. Predictions are grouped post-hoc to compute pairwise accuracy, awarding credit if and only if the model correctly labels the Fact as True and the Fib as False (reducing random guessing baseline to 25.0%). To uncover how modalities interact over long sequences, the framework probes models across five input configurations: video frames only, subtitles only, video plus subtitles, Wikipedia plot synopses, and film title/year parametric baselines, complemented by targeted frame extraction from annotated ground-truth segments.

Benchmark & Evaluation Protocol

The benchmark evaluates 2 proprietary frontier models (GPT-4o, Gemini 2.5 Pro) and 20 open-weight vision-language models (including Qwen2.5-VL, Qwen3-VL, InternVL3/3.5 families, Ovis2, Molmo2, Gemma3, GLM-4.5V, and long-video specialists such as LongVILA-R1 and MiMo-VL). Video inputs are downsampled to 1 fps and uniformly sampled to match each model's maximum context length (e.g., 120 frames for Gemini 2.5 Pro, 50 frames for GPT-4o, 64 frames for InternVL3). Prompts are tailored with direct True/False output instructions for open-weight models and brief rationales before final judgments for closed models. All experiments are executed on an infrastructure powered by 2 NVIDIA H200 (140GB) GPUs.

Key Experimental Results

Main Results

The evaluation covers 868 claim pairs across all 53 movies. The table below presents pairwise accuracy and individual standard accuracy across subtitles-only (Subs), uniform full-movie video frames (Video), and combined video plus subtitles (Video+Subs):

Model #Params / Active Pairwise Acc Subs (%) Pairwise Acc Video (%) Pairwise Acc Video+Subs (%) Standard Acc Video+Subs (%)
Random Baseline - 25.0 25.0 25.0 50.0
Human Baseline - – – 84.1 90.5
GPT-4o - (Proprietary) 48.8 19.5 49.8 73.2
Gemini 2.5 Pro - (Proprietary) 58.9 37.9 63.8 79.8
VideoLLaMA3 7B 36.3 21.1 36.1 64.2
Qwen2.5-VL 7B 31.7 24.5 32.8 62.0
LongVILA-R1 7B 9.8 11.5 16.9 56.6
MiMo-VL-7B-RL 7B 36.4 25.8 38.6 65.2
Molmo2 8B 41.0 28.3 43.0 67.7
Aria 25B (MoE 3.9B) 36.2 2.5 33.4 63.2
Gemma3 27B 42.3 31.5 39.7 65.3
Qwen3VL-IT 30B 33.5 19.2 43.8 69.5
Ovis2 34B 47.2 20.7 48.5 71.1
InternVL3.5 38B 47.5 26.6 46.2 70.3
Qwen2.5-VL 72B 37.6 29.7 47.5 69.0
LLaVA-Video 72B 41.7 15.6 41.8 69.1
InternVL3 78B 49.0 22.1 51.3 72.7
GLM-4.5V-FP8 108B 38.4 26.6 31.4 62.4

Ablation & Targeted Visual Evidence Study

To investigate whether poor visual utility stems from uninformative visual signals or context-retrieval failure, the authors evaluated single-scene claims under uniformly sampled full-movie frames versus targeted frames extracted from golden timestamps. Furthermore, alternative text inputs were ablated on Gemini 2.5 Pro:

Experimental Configuration Model Video-Only Pairwise Acc (%) Video+Subs Pairwise Acc (%) Relative Delta over Full-Movie Input
Full Movie vs. Targeted Frames Gemini 2.5 Pro 38.6 → 49.4 59.4 → 74.0 Video: +10.8% / Video+Subs: +14.6%
Full Movie vs. Targeted Frames Gemma3-27B 27.2 → 36.9 42.7 → 49.5 Video: +9.7% / Video+Subs: +6.8%
Full Movie vs. Targeted Frames Qwen3VL-30B 12.6 → 22.3 40.8 → 52.4 Video: +9.7% / Video+Subs: +11.6%
Full Movie vs. Targeted Frames InternVL3.5-38B 29.1 → 41.7 51.5 → 53.4 Video: +12.6% / Video+Subs: +1.9%
Full Movie vs. Targeted Frames InternVL3-78B 27.2 → 40.8 49.5 → 66.0 Video: +13.6% / Video+Subs: +16.5%
Text Ablation (Gemini 2.5 Pro) Wikipedia Synopsis Only – 27.9 (Near random) High-level summary lacks plot grounding
Text Ablation (Gemini 2.5 Pro) Movie Title & Year Only – 45.4 Reflects memorized parametric knowledge
Text Ablation (Gemini 2.5 Pro) Subtitles Only – 58.9 Dialogue delivers primary narrative skeleton

Key Findings

  • Substantial Human-VLM Performance Gap: Even under the best multimodal condition (Video+Subs), the leading proprietary model Gemini 2.5 Pro scores 63.8% in pairwise accuracy, trailing human performance (84.1%) by 20.3% absolute; the best open-weight model InternVL3-78B attains 51.3%, leaving nearly half of the contrastive claim pairs unsolved.
  • Subtitles Provide the Dominant Narrative Signal: Across almost all models, performance with subtitles alone markedly outperforms full-movie video alone (e.g., Gemini 2.5 Pro scores 58.9% on Subs vs. 37.9% on Video; InternVL3-78B scores 49.0% on Subs vs. 22.1% on Video). Adding uniformly sampled video frames to subtitles yields marginal or inconsistent gains for most models, showing that models struggle to extract incremental value from diffuse visual inputs.
  • Visual Signals Are Valuable but Impeded by Long-Context Retrieval: When visual inputs are restricted to targeted frames containing golden scene evidence, model performance improves dramatically (Gemini 2.5 Pro jumps by +14.6% to 74.0%, and InternVL3-78B jumps by +16.5% to 66.0%). This demonstrates that visual information is inherently informative, but models suffer from a fundamental bottleneck in identifying and retrieving relevant frames across long multimodal contexts.
  • Superficial Synopses Fail while Pretrained Knowledge Leaks: Feeding only Wikipedia plot synopses drops Gemini 2.5 Pro to 27.9% (close to the 25.0% random baseline), proving that abstract summaries cannot resolve detailed narrative claims; conversely, providing only the movie title and year yields 45.4%, revealing that large foundation models lean partly on parametric memory for classic films.
  • Cognitive Dimension and Granularity Divergence: Models benefit most from visual information in the emotion understanding dimension (Gemini 2.5 Pro reaches 71%), whereas causal reasoning is weakest under video-only inputs (around 40%) and relies heavily on dialogue clarification; furthermore, while human performance remains stable across single-scene and multi-scene/global reasoning, VLMs degrade significantly when moving from local to global multi-scene synthesis.

Highlights & Insights

  • Cognitively Grounded Narrative Evaluation: By deploying 26 annotators and 24 independent validators who evaluate claims purely from memory without rewatching, the benchmark isolates narrative milestones that form persistent mental models in human recall, avoiding the trivial perceptual distractions common in synthetic QA sets.
  • Pairwise Metric Unmasks Prediction Asymmetries: The analysis reveals that many VLMs exhibit pronounced systematic biases, such as an overwhelming propensity to predict False under video-only conditions (scoring high on Fibs but collapsing on Facts). The pairwise accuracy metric enforces double-verification, suppressing guessing gains to 25% and exposing genuine reasoning capability.
  • Open-Licensed Full-Length Video Release: Sourcing entirely from Public Domain 1.0 movies via the Internet Archive guarantees perpetual data availability, solving the video link rot, takedown issues, and copyright barriers that have historically crippled movie video benchmarks.

Limitations & Future Work

  • Western Cultural Skew: Sourcing from historical public-domain cinema results in an overrepresentation of Western narrative conventions and cultural idioms, leaving model generalization across diverse cultural narrative structures unexamined.
  • Auditory Modality Gap: While human viewers perceive narrative context through concurrent visual, spoken, and auditory/musical cues, evaluated models were restricted to visual frames and subtitle text due to current architectural constraints in scaling audio processing across 90-minute sequences.
  • Absence of Active Memory Consolidation: Current VLMs rely on brute-force uniform frame encoding; future architectures must incorporate hierarchical episodic memory, selective key-event consolidation, and plot-graph retrieval to effectively filter narrative noise in long videos.
  • vs. MovieNet / MovieChat: Early movie understanding benchmarks relied on short isolated clips or token-pruning approximations over brief windows; MF2 assesses full-length 88-minute feature films focusing on macro-level plot causality and character arcs.
  • vs. EgoSchema / Video-MME / InfiniBench: Video-MME and EgoSchema emphasize physical action perception in procedural or egocentric videos; InfiniBench scales duration but utilizes automated LLM question generation that tends to over-index on single-frame trivial details. MF2 provides human-curated contrastive claim pairs centered strictly on memorable storyline developments.
  • vs. NarraBench / One Thousand and One Pairs: Successfully extends the contrastive Fact-Fib claim verification paradigm from long-form literary text into the complex audiovisual domain, establishing a unified methodology for multimodal long-context narrative evaluation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ (Pioneers human-grounded memorable narrative benchmarking using contrastive fact-fib pairs on full-length open movies)
  • Experimental Rigor: ⭐⭐⭐⭐⭐ (Evaluates 22 frontier models across multiple input modalities, targeted vs. uniform frame ablations, and fine-grained cognitive axes)
  • Writing Quality: ⭐⭐⭐⭐⭐ (Clear, coherent narrative progression with precise technical characterizations and rigorous quantitative analyses)
  • Impact & Practical Value: ⭐⭐⭐⭐⭐ (Open-access public domain release provides an enduring, legally unencumbered foundation for long-video reasoning research)