NarrativeTrack: Evaluating Entity-Centric Reasoning for Narrative Understanding¶

Conference: ECCV 2026
Paper: ECCV
Code: https://github.com/apple/ml-NarrativeTrack
Area: VLM Reasoning
Keywords: narrative understanding, entity-centric reasoning, compositional reasoning progression, MLLM benchmark, temporal consistency

TL;DR¶

This paper introduces NARRATIVETRACK, the first benchmark that evaluates narrative understanding in multimodal large language models from an entity-centric angle: a fully automated pipeline (ensemble detection → ReID identity tracking → highlighted contextual recognition) extracts timestamped human entity representations from raw video, then a Compositional Reasoning Progression (CRP) over "entity existence → entity changes → entity ambiguity" programmatically generates 1,006 questions scored by exact match; the strongest model, Gemini-2.5-Pro, reaches only 83.80% while every open-source model stays below 57%, exposing a trade-off between perceptual grounding and temporal integration.

Background & Motivation¶

Video understanding benchmarks have spread widely in recent years: Video-MME, MVBench, NExT-QA and PerceptionTest cover action recognition plus causal and spatio-temporal reasoning, while LVBench and LongVideoBench push durations into the thousands of seconds to probe global context. Yet a middle ground between these two families is missing. Short-clip datasets (tens of seconds on average) contain almost no scene variation, so questions are often answerable from static cues in a single frame; long-video benchmarks do span long durations, but they mostly test coarse-grained summarization and never require the model to maintain the state of one specific entity over a long horizon. Recent work confirms the concern directly: shuffling the frame order leaves performance on many benchmarks essentially unchanged, implying that models rely on language priors and static pixels rather than genuine temporal reasoning. VELOCITI attempts to isolate agent–action binding errors through strict video–language entailment, and the design is clean, but its entities remain continuously visible inside the clip, so it tests within-clip binding rather than cross-scene continuity.

The genuinely hard part is precisely the cross-scene case: a character may vanish midway, reappear several scenes later, having changed clothes and scene, and participate in causally linked events separated by long temporal gaps. Evaluating whether a model can follow this requires time-anchored, entity-level structural annotation — who, when, doing what, where, wearing what — and no existing dataset provides it (Table 1 of the paper compares AVA, TVQA, MovieQA, How2QA, NExT-QA, Video-MME, PerceptionTest, LongVideoBench and LVBench, and none supplies bounding boxes, actions, scenes and outfits together). Annotating unconstrained long videos by hand is not only expensive but also highly inconsistent, which is exactly why this direction has stayed empty.

What changed is that the primitives for building such data now exist: off-the-shelf multi-object detectors produce stable boxes, ReID models cluster the same person across frames into a trajectory, and MLLMs can already describe highlighted clips well enough to be useful. The design choice here is bottom-up — instead of starting from scene-level semantics or story-level summaries, the paper first decomposes a video into entities (restricted to human entities, since people are the primary agents of narrative videos with the richest variation in action and appearance, whereas background objects vary little and would degrade the evaluation into simple identity matching), and reduces narrative understanding to a question that can be programmatically generated and deterministically scored: can a model keep tracking an entity over time and correctly ground its state evolution? Core idea: formalize narrative understanding as an entity-centric compositional reasoning progression — an automated pipeline extracts time-anchored entity trajectories and attributes, then a three-level progression (existence → changes → ambiguity) generates deterministic QA, which makes evaluation scalable while attributing failures precisely to broken temporal continuity, weak attribute grounding, or confusion among similar entities.

Method¶

Overall Architecture¶

Given a raw video \(V\), the pipeline samples it at a fixed frame rate into a timestamped frame sequence. No human annotation is involved: a three-stage entity-centric pipeline first converts the video into structured entity representations — for every timestamp at which an entity appears it records the bounding box at that moment, the action being performed, the surrounding scene, and the entity's outfit; the observations of one entity chained over time form a time-anchored trajectory. On top of this, CRP (Compositional Reasoning Progression) defines three levels of increasing difficulty, a template-based QA generator deterministically fills question templates with entity representations and attaches distractors, and scoring is exact match. The whole chain is serial: input is raw video, output is 1,006 questions with ground-truth answers plus model scores.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw video<br/>sampled at fixed frame rate"] --> B["Ensemble detection and identity tracking"]
    B --> C["Highlighted contextual recognition"]
    C --> E["CRP three-level progression"]
    E --> F["Template QA generation and distractor design"]
    F --> G["Exact-match evaluation"]

Key Designs¶

1. Ensemble detection and identity tracking: turning "a person who persists across shots" into a stable trajectory

Entity-centric reasoning can only be evaluated if the entities themselves are reliable, yet a single detector will always miss or falsely fire on long video — a miss splits one trajectory into two, a false positive merges two people into one, and the error cascades into tracking and attribute annotation. Rather than taking the naive union of Detectron2 and OWLv2 predictions, the authors apply spatial consensus: bounding boxes from the two detectors are paired and their IoU computed, and a pair above 0.5 is judged to refer to the same entity, in which case only the higher-confidence box is kept; otherwise both boxes survive. This recovers boxes a single detector misses without generating duplicate trajectories when both models report the same entity. Evaluated with frame-level matching on AVA, the ensemble raises recall from 0.780 (Detectron2 alone) and 0.801 (OWLv2 alone) to 0.848.

Detection alone does not decide whom to follow. Naively tracking every detected box is both inefficient and error-prone, since many entities appear only briefly and contribute nothing to the narrative. The authors extract ReID embeddings with OSNet-x1.0 and cluster them; each cluster is one entity, ranked by size, and the top four become the main characters — the underlying empirical assumption being that recurring presence correlates with narrative centrality. The choice of four is empirically supported: across 100 sampled videos, central characters typically occupy more than 50% of the frame height or width, and the top-four clusters satisfy this criterion for at least 50% of their instances, making four a stable heuristic for salient characters. To suppress identity drift further, trajectories are refined with face recognition for precise alignment, and ensemble verification across multiple MLLMs with majority voting removes false identities — essentially an unsupervised "consensus among models" that emulates agreement among human annotators. On 1,108 detections, this model-based majority vote agrees with the human expert majority vote in 96.08% of cases, showing that the consensus mechanism does filter out erroneous tracks while preserving coherent identities.

2. Highlighted contextual recognition: attaching action, outfit and scene to every timestamp of a trajectory

Spatial localization alone is not enough: narrative comprehension asks what a person is doing, wearing and where they are, so every timestep of a trajectory is augmented with action, outfit and scene. The critical detail is how annotation is performed. Gemini-2.5-Pro infers these attributes, but the target entity is highlighted in the clip before being fed to the model rather than passing the raw clip. The reason is direct — several people usually share a scene, and without highlighting the model attributes the context to the most visually salient person, contaminating the target entity's representation; highlighting preserves the surrounding context while forcing attention onto the correct individual. Attributes are annotated per segment in which the entity appears and progressively populate that entity's representation, yielding a structured, temporally aligned record. The same Gemini-2.5-Pro is then used in reverse to select source material: based on predicted attributes it identifies videos with significant attribute changes (for the "changes" questions) and videos containing visually similar entities (for the "ambiguity" questions), ensuring that the resulting questions are not trivially answerable.

3. CRP three-level progression: decomposing "narrative understanding" into an attributable diagnostic chain

A single aggregate accuracy says nothing about whether a model lost the thread of time, failed to read a state change, or confused two similar-looking people. CRP splits entity-centric reasoning into three interdependent, increasingly difficult levels. The first is entity existence, which directly tests temporal continuity: an entity appears, disappears and reappears, and the model must judge whether it was only shown at the beginning and never returned. This is the most basic requirement, and failure here means long-range temporal tracking itself has broken down. The second is entity changes, which tests detection and grounding of a target entity's state transitions, subdivided into action changes (temporal grounding of dynamic events), outfit changes (perception of fine-grained appearance transitions) and scene changes (linking the entity to environmental context); this level demands that the model integrate dynamic visual and contextual cues beyond mere continuity, and failure means evolving signals are poorly grounded. The third is entity ambiguity, which introduces visually similar entities and requires joint temporal reasoning and fine-grained perceptual disambiguation; errors here reveal weak compositional reasoning. Chained together, the three levels form a progression from "persisting existence" to "grounded state transitions" to "joint temporal–perceptual disambiguation" that maps onto three distinguishable failure sources — disrupted temporal continuity, weak grounding of evolving attributes, and confusion among similar entities. This is the most valuable part of the design relative to existing benchmarks: the evaluation directly localizes which link a model breaks.

4. Template QA generation and distractor design: a deterministic generation and scoring protocol

Questions are not written by humans but instantiated programmatically from entity representations. Reasoning templates aligned with each CRP dimension are pre-designed, template slots are filled deterministically with attributes from the entity representation, and the ground-truth answer is derived from the same metadata so that question and answer stay semantically consistent; GPT-4o only performs grammatical and fluency refinement and filters out invalid cases (for example, questions generated from representations without a valid state change) — it never generates facts. Three question formats are used: binary, multiple-choice (MC) and ordering. The ordering format requires arranging an entity's attribute transitions chronologically and is the most demanding of the three on temporal compositionality, since it cannot be guessed from recognizing a single event — the entire trajectory's ordering must be right.

On the same set of entity representations, three reasoning directions are generated programmatically by varying the temporal reference point: forward (inferring from the start to the end), backward (reasoning from the end back to the start), and agnostic (a bidirectional form that reasons from a midpoint toward both ends). This design lets the benchmark measure a model's temporal directional bias in isolation. Distractors come in two kinds. Real distractors are sampled from other entities within the same video clip (intra-clip negatives); since these entities co-exist temporally and spatially with the target, the model must actually track and separate the target, which tests fine-grained discrimination in realistic multi-entity settings. Synthetic distractors are sampled from entities in different videos (cross-clip negatives) and do not appear in the queried clip at all; they test whether a model accepts an entity simply because it is semantically plausible, i.e. hallucination suppression and presence verification.

Quality control proceeds in two steps. First, 100 sampled items are reviewed independently by three annotators (valid/invalid); 70% are unanimously judged valid with substantial inter-annotator agreement (Fleiss' κ = 0.767), and most invalid cases stem from overly obvious synthetic distractors with implausible attribute assignments rather than from errors in entity tracking or contextual grounding, consistent with the pipeline's own 96.08% agreement. Second, human annotators verify the attributes inferred by Gemini against the video evidence and refine the retained pairs along three axes: correcting inconsistent ground-truth answers, revising ambiguous or weakly grounded attributes in the questions, and replacing trivial synthetic distractors with visually confusable alternatives. The result is 1,006 QA pairs drawn from 406 video clips, covering diverse genres (documentary, news, TV drama) with durations up to 659 seconds; the five sub-dimensions are balanced (entity existence 200, action changes 222, outfit changes 192, scene changes 191, entity ambiguity 201), and the distribution of ground-truth answers across candidates is balanced after verification. Re-evaluated by three annotators, the cleaned benchmark yields 96% average human accuracy, confirming that the questions are clear and reliable.

The evaluation protocol deliberately avoids LLM-based judging and open-ended generation: every question is binary, multiple-choice or option-ordering, and model predictions are scored by exact match against ground truth, eliminating ambiguity from phrasing, lexical variation and subjective assessment.

A Worked Example¶

Take the video in Figure 4 of the paper: the target entity appears in four segments. In segment 1 she is smiling, wearing a bare-shoulder top, in a bedroom; in segment 2 she is singing on a stage, now in a yellow vest and white shirt; in segment 3 she is still on the stage in the same outfit, now handing over a microphone; in segment 4 she is in a dimly lit room looking down and to the side, wearing a dark collared jacket on top. Once these four observations form a trajectory, the templates instantiate five questions spanning the CRP levels at once:

existence (binary) — "Is the person with bare shoulders, smiling in the bedroom, only seen at the beginning and then gone?"; action changes (ordering) — "Identify the order of actions made by the person with bare shoulders, smiling in the bedroom at the beginning: (a) handing over the microphone (b) singing on the stage (c) looking down and to the side", where the chronological answer following the trajectory is (b) → (a) → (c); outfit changes (binary) — "Is the person with bare shoulders, smiling in the bedroom at the beginning, later seen wearing a dark collared jacket?" (true, from segment 4); scene changes (multiple-choice) — "In which scene does the person with bare shoulders, smiling in the bedroom at the beginning, appear later? (a) stage (b) kitchen (c) bathroom" (a, from segments 2 and 3); entity ambiguity (binary) — "After appearing with bare shoulders and smiling in the bedroom at the beginning, is the same person later shown cooking in the kitchen while wearing a beige shirt and an apron?" (no — this is a real distractor taken from another entity in the same video). One representation of one entity is reused into five questions at different granularities, which is exactly why template-based generation scales and stays attributable.

Key Experimental Results¶

Main Results¶

Source videos come from AVA, Video-MME and LVBench; the authors filter out clips that are too dark or low quality, as well as videos with only a single character and no meaningful attribute changes. In total 13 open-source and 7 proprietary MLLMs are evaluated. Open-source models split into general-purpose (OGP-MLLM, image-text alignment emphasizing visual grounding) and video-specialized (OVS-MLLM, video-text alignment emphasizing temporal modeling) families. Open-source models use 20 frames uniformly (the optimum found in the frame-density analysis of Section 5), while proprietary models use 128 frames by default to exploit their larger visual context; GPT-4o is additionally evaluated at 20 frames for fair comparison.

Model	Type	Frames	Avg. accuracy
Gemini-2.5-Pro	Proprietary	128	83.80
Gemini-3.1-Pro	Proprietary	128	83.40
Gemini-3-Flash	Proprietary	128	79.32
GPT-4.1	Proprietary	128	74.85
Gemini-2.5-Flash	Proprietary	128	74.25
GPT-4o	Proprietary	128	72.27
GPT-4o	Proprietary	20	70.97
Gemini-2.0-Flash	Proprietary	128	60.24
Qwen-2.5-VL-32B	OGP	20	56.96
InternVL3-38B	OGP	20	55.47
Video-LLaMA2-72B	OVS	20	49.70
Qwen-2.5-VL-7B	OGP	20	48.81
InternVL3-8B	OGP	20	48.81
VILA-8B	OVS	20	45.33
Video-LLaMA2-7B	OVS	20	43.04
mPLUG-Owl3-7B	OVS	20	42.94
LLaVA-NeXT-Video-7B	OVS	20	42.94
VideoChat2-7B	OVS	20	42.55
LLaVA-NeXT-Video-34B	OVS	20	39.36
LLaVA-Video-7B	OVS	20	34.39
Video-LLaVA-7B	OVS	20	33.00

Ordering is the hardest of the three formats, and the gap between open-source and proprietary models is widest there:

Model	Action changes (ordering)	Outfit changes (ordering)	Scene changes (ordering)
Gemini-2.5-Pro	60.98	83.33	91.30
Gemini-3-Flash	63.42	94.44	95.65
Gemini-3.1-Pro	48.78	88.89	95.65
GPT-4.1	46.34	66.67	73.91
Qwen-2.5-VL-7B	17.07	22.22	34.78
InternVL3-8B	29.27	22.22	39.13
Video-LLaMA2-72B	19.51	11.11	43.48
LLaVA-Video-7B	17.07	5.56	21.74
LLaVA-NeXT-Video-7B	2.44	0.00	13.04

Ablation Study¶

The benchmark has no modules to remove, so the paper substitutes a set of controlled analyses that verify what models actually rely on when answering:

Analysis setting	Compared on	Key numbers	Note
Removing visual input	GPT-4o	Accuracy drops 30.52%, approaching the random baseline	Rules out solving the benchmark from language priors alone
Reversing frame order	Entity-change ordering questions	51.2% → 6.1%	Performance collapses when temporal order is destroyed, confirming genuine temporal reasoning
Frame sampling density k	Open-source MLLMs	k ∈ {8,12,16,20,24,28,30,40,128}; peaks at k=20 and degrades beyond	The bottleneck is not temporal coverage; Video-LLaMA2 (trained on 8 frames) is least stable at 128
Real vs synthetic distractors	OGP-MLLMs	0.44% higher on synthetic	Relies on localized visual cues and rejects irrelevant attributes
Real vs synthetic distractors	OVS-MLLMs	Up to 9.69% drop on synthetic	Stronger temporal modeling comes with stronger hallucination
Real vs synthetic distractors	Proprietary MLLMs	Largest real–synthetic gap, up to 20.03%	Even the strongest models struggle to suppress contextually irrelevant generation
Controlled analysis (conditioned on correct existence predictions)	OVS-MLLMs	−11.5% real, −14.9% synthetic	Rules out "merely a detection failure"; the temporal–perceptual trade-off is real
Forward vs backward reasoning	OGP / OVS / proprietary	Gaps of 20.65% / 9.96% / 17.55%	Temporal directional bias; agnostic (bidirectional) conditions score lowest
Entity continuity types	All three families	OGP best on disappear; OVS and proprietary best on reappear	The former leans on static cues, the latter integrates time better but over-predicts reappearances

Key Findings¶

Perceptual grounding and temporal integration trade off against each other. Qwen-2.5-VL-32B (56.96%), an open-source general-purpose model, surpasses the best open-source video-specialized model Video-LLaMA2-72B (49.70%). The former recognizes static visual cues accurately but cannot hold temporal consistency; the latter tracks temporal continuity more reliably but hallucinates frequently under entity ambiguity or appearance changes. The authors conclude that narrative understanding is a product of both, and strength in either dimension alone is insufficient.
Temporal directional bias is an independent and severe weakness. Forward reasoning consistently beats backward reasoning, with gaps of 20.65%, 9.96% and 17.55% for OGP, OVS and proprietary models respectively — closely paralleling the Reversal Curse in LLMs, where models trained on "A is B" fail to learn "B is A". Models propagate entity states forward along the timeline but cannot invert the mapping, and the bidirectional form that must infer from a midpoint toward both ends scores lowest. The authors put it vividly: models can extend a narrative but cannot rewind it.
More frames do not mean stronger entity reasoning. Open-source accuracy peaks at 20 frames and then declines as frame count grows, the opposite of the monotonic gains reported for long-video reasoning. Extra frames add redundant information while the ability to maintain temporal coherence and perceptual grounding does not improve in step. Model size is no guarantee either: InternVL3-38B beats 8B by 6.66% and Video-LLaMA2-72B beats 7B by 5.66%, yet LLaVA-NeXT-Video-34B (39.36%) falls below its own 7B version (42.94%).
Synthetic distractors are not easy for most models. Intuitively, an entity from another video should be easier to reject, yet video-specialized models drop by up to 9.69% on synthetic distractors and proprietary models show a real–synthetic gap of up to 20.03%, indicating they are pulled along by textual or global priors instead of performing fine-grained visual discrimination. General-purpose models, by contrast, behave almost identically across the two distractor types.
Even the most basic existence dimension hides a large gap. Most open-source models still reach 40–70% on binary existence questions, but the numbers collapse on multiple-choice — LLaVA-Video-7B falls from 67.00% to 2.00% and Qwen-2.5-VL-7B from 57.00% to 36.00%, suggesting that "was this entity present" judgments lean heavily on the guessing space that the format provides rather than on stable tracking ability.
Failures cascade across levels. In the qualitative analysis, at the existence level models overestimate continuity, judging an entity that appears only briefly at the start as persisting throughout; at the change level they produce semantically plausible but visually ungrounded answers (reciting stage-plausible action sequences that do not match what happened); the ambiguity level is worst, where even strong proprietary models confuse identities or assert an entity's presence without evidence.

Highlights & Insights¶

Reducing narrative understanding to a programmatically generatable and scorable entity-tracking problem is the key translation of this paper. It turns what used to require human annotation and could not scale into a pipeline of detection plus ReID plus MLLM attribute labeling, with ground truth derived directly from metadata and therefore self-consistent. The general recipe — formalize the task as a structured representation, then generate questions from that representation — transfers to any evaluation with a well-defined tracking target.
The three CRP levels are a failure attributor, not merely a difficulty ladder. Most benchmarks tell you a score; CRP tells you whether the model lost the entity over time, failed to ground an attribute, or was fooled by a similar-looking person. Layered with forward/backward/bidirectional reasoning directions and real/synthetic distractors, one dataset yields several orthogonal diagnostic axes — a very cost-effective "one dataset, many conclusions" design.
Replacing human agreement with model consensus. Trajectory verification is fully unsupervised, using multi-MLLM majority voting to emulate annotator consensus and reaching 96.08% agreement with the human majority vote. This scales far better than labeling a subset by hand and generalizing, and it is what lets the pipeline legitimately claim to be fully automated.
Highlighting the target before querying an MLLM is a practical detail worth stealing: inside multi-entity video, attribute attribution is easily misassigned to the visually dominant person, and a single visual prompt fixes most of it. Any work on entity-level attribute extraction can reuse this directly.
The most striking conclusion is the analogy between temporal directional bias and the Reversal Curse. It suggests that "models can read a narrative forward but not backward" may not be a data problem at all, but a structural limitation of the left-to-right autoregressive decoding paradigm inherited from LLM backbones — pointing toward bidirectional temporal modeling or contrastive reversal objectives as new training targets.

Limitations & Future Work¶

The authors explicitly acknowledge that the benchmark currently covers human entities only. The rationale is that people vary richly in action and appearance and are the most representative subjects, whereas non-human entities and background objects vary little, which would degrade evaluation into simple identity matching and introduce confounding. The cost is that person–object interaction, animal behavior, and narratives built on objects being passed or modified are excluded entirely.
"Main characters = the four largest clusters by frequency" is a heuristic supported only by 100 sampled videos. Secondary entities that are narratively critical (say, a key figure appearing once in a crucial scene) are dropped outright — and those may be exactly the hardest objects of temporal reasoning.
Attribute annotation depends on Gemini-2.5-Pro, a proprietary model, and only sampled human verification and correction was performed; no error-rate estimate over the full attribute set is given. Whether the annotations and the resulting question distribution remain stable under a different MLLM or a different model version is not answered.
The quality review itself leaves room: only 70% of the 100 sampled items were unanimously judged valid (κ = 0.767 is "substantial" rather than "almost perfect" agreement), and the remaining 30% are contested, mainly because synthetic distractors were too trivial; the substitution was performed but the post-substitution distribution is not quantified.
Average clip length is 55.3 seconds with a maximum of 659 seconds, still short of genuine feature-length narrative, and the protocol relies on exact match without open-ended generation, so the ability to produce a coherent narrative cannot be assessed. The collapse on existence multiple-choice (67.00% → 2.00%) also hints that some items are affected by the format's guessing space; the paper only reports balancing the answer distribution and does not analyze yes/no bias further.
Frame budgets are asymmetric (20 frames for open-source, 128 for proprietary), so cross-family comparisons are trends rather than controlled rankings; the paper itself states that analyses are read as within-category trends.
Concrete improvements: extend the entity definition from humans to objects and animals (deciding inclusion by the amplitude of attribute change rather than by entity category); replace the fixed "top four characters" with narrative-contribution-based dynamic selection; use multi-model cross-validation plus full-set human spot checks in attribute annotation to quantify labeling noise; and introduce bidirectional temporal modeling or reversal-contrastive objectives aimed specifically at the forward/backward asymmetry.

vs Video-MME / NExT-QA / MVBench / PerceptionTest: these benchmarks are dominated by short clips with little scene variation, and many questions are answerable from single-frame cues or language priors (shuffling frame order barely changes performance). NARRATIVETRACK questions require tracking an entity across shots: reversing frame order drops entity-change ordering from 51.2% to 6.1% and removing visual input costs 30.52%, evidence that temporal reasoning is genuinely required.
vs LVBench / LongVideoBench / the long-video portion of Video-MME: these span longer durations but test coarse-grained global context rather than maintaining a specific entity's state across scenes. This paper inverts the emphasis and only tests fine-grained entity-level continuity, which makes the diagnosis sharper at the cost of a narrower task family.
vs VELOCITI: VELOCITI isolates agent–action binding errors through strict entailment with carefully constructed positive and negative captions, but its entities stay continuously visible within the clip, so reasoning is confined to within-clip binding. This paper pushes to cross-scene, cross-gap settings where entities can disappear and return, change outfits, and be confused with similar entities, testing long-range identity and state persistence.
vs MovieQA / TVQA: movie and TV QA also target narrative, but rely on manual annotation (TVQA averages only 11.2-second clips with 15,253 questions; MovieQA has 2,144), provide no entity-level structured attributes such as bounding boxes, actions, scenes and outfits, and cannot attribute a failure to a specific link of entity tracking; this paper's automated pipeline differs on both scale and diagnostic granularity.
vs general MLLM evaluation: splitting open-source models into OGP and OVS by training objective before reading trends is itself worth borrowing — it moves the "why is a model strong or weak" question away from parameter-count narratives and toward the correspondence between training objectives and capability structure.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First benchmark to define narrative understanding bottom-up from an entity-centric perspective; the three-level CRP and the forward/backward/agnostic reasoning split are new diagnostic tools.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ 20 models (13 open-source + 7 proprietary), three controlled analysis axes (reasoning direction, distractor type, frame density) plus conditioned controls, and the pipeline itself is validated with AVA recall and human agreement.
Writing Quality: ⭐⭐⭐⭐ The argument chain is clear and the failure attribution is concrete, though some figures are garbled in the cached text and appendix formula details are missing; a few numbers need checking against the original.
Value: ⭐⭐⭐⭐⭐ It quantifies the structural defect that "models can read a narrative forward but not backward" and provides a scalable evaluation, with direct guidance for future entity-centric training objectives and bidirectional temporal modeling.