A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation

Conference: ECCV2026
Paper: Official ECCV page
PDF: Full paper
Area: Multi-Agent Systems
Keywords: Cinematic video compilation, hierarchical narrative memory, iterative narrative planning, multi-agent collaboration, instruction following

TL;DR

CineBench evaluates instruction-driven cinematic compilation, while the training-free CineAgents system analyzes source footage before planning and executing an edit, achieving a shot-level F1 of 64.13, temporal correctness of 52.09%, and adversarial rejection of 87.23% on this benchmark.

Background & Motivation

Automatic compilation is neither the generation of new footage nor simply the retrieval of relevant moments from a long video. A user may ask for a character-centered edit, an outcome-first narrative followed by its causes, or a compilation spanning several sources, with titles, music, and transitions. Existing approaches for casual footage typically retrieve matching shots and then rank or arrange them; cinematic methods often optimize a predefined task such as trailer generation or highlight detection. Neither setting adequately tests open-ended cinematic editing intentions, motivating a benchmark with complex instructions and professional reference compilations.

A single movie shot rarely explains who is addressing whom or why the current event matters. Independent captions discard dialogue attribution, character relationships, and links across scenes, while dumping all captions into context introduces redundancy and narrative errors. Even correctly retrieving the named characters and objects does not guarantee a coherent edit: connecting events may be missing, or the source chronology may conflict with a requested flashback. The paper calls these problems contextual collapse and temporal fragmentation, respectively, separating the need for better source representations from the need for better narrative decisions.

CineBench supplies editor-created instructions and reference outputs, and CineAgents separates creative proposals from verification against the source material. Rather than immediately predicting a list of shot IDs, the system builds reusable narrative memory and progressively grounds an editing blueprint in actual footage. Core idea: replace retrieve-and-rank with design-and-compose, using character and dialogue evidence to build narrative memory and a director–orchestrator verification loop to constrain the final edit.

Method

Overall Architecture

The inputs are a text instruction and a collection of existing cinematic videos; the output selects, rearranges, and edits shots from those sources. The system does not train a video generator or synthesize the main story footage: its creative decisions concern shot selection, narrative order, and editing operations. A manager agent interprets the instruction and coordinates four specialist roles: script agent, director agent, orchestrator agent, and editor agent.

The script agent performs identity and dialogue alignment, then builds hierarchical narrative memory as a reusable analysis of the source. The director proposes a blueprint, while the orchestrator checks whether the source supports it; iterative narrative planning progressively determines the required shots. Supervised tool execution then turns the confirmed script into a video, with shared history supporting monitoring and restoration of previous editing states.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Existing cinematic footage"] --> B["Identity and dialogue alignment"]
    B --> C["Hierarchical narrative memory"]
    C --> D["Iterative narrative planning"]
    U["User editing instruction"] --> D
    D -->|"Insufficient evidence: revise blueprint"| D
    D --> E["Supervised tool execution"]
    E --> F["Final compiled video"]

Key Designs

1. Identity and dialogue alignment: establish who appears and who speaks

Script reverse-engineering begins with identity anchors rather than asking a captioning model to rediscover character names in every shot. The system builds a character information dataset from metadata such as Wikipedia, using InsightFace for frontal views and SOLIDER representations for non-frontal views. Detections with similar appearance are grouped into temporal trajectories, along which anchor identities are propagated. This provides continuity through difficult poses or occlusions; the contribution is the organization of existing perception outputs into consistent film-level references, not a new recognition network.

Dialogue requires a separate association step because an on-screen character need not be speaking, and an off-screen speaker may not be visible. OCR extracts on-screen subtitles, while lip activity detection using InsightFace keypoints and a ResNet classifier establishes confirmed audiovisual speaker anchors. WeSpeaker extracts audio features from these anchors, and K-means clustering helps bind a voiceprint to each character, supporting character–dialogue associations. This dialogue character matching component, abbreviated DCM in the ablation, lets later summaries distinguish an utterance from another character's response. Its effectiveness still depends on subtitles, metadata, and reliable perception; it is not an unconditional solution to identity recognition in arbitrary footage.

2. Hierarchical narrative memory: preserve detail while exposing story context

AutoShot first segments the source into shots, after which the script agent summarizes visual information, character identities, and dialogue. Each summary also receives a buffer containing multimodal information from the preceding 10 shots, reducing isolated interpretations of the current image. The buffer helps interpret an action as part of an unfolding event rather than repeatedly describing appearances and locations. BaSSL then groups shots into coherent events; these events support story abstraction and character profile generation. The resulting memory retains shot, event, story, and character levels instead of replacing all low-level evidence with a single movie synopsis.

This hierarchy gives abstract instructions an appropriate entry point: the system can first locate a story or character relationship, then descend to events and executable shot choices. Keeping shot IDs and detailed summaries also prevents a correct high-level plan from becoming impossible to implement. Unlike simply extending the context window, the representation organizes access to information rather than requiring one prediction over a flat transcript. The HNM ablation removes this hierarchy and leaves only shot-level information. The paper describes the construction process but does not fully specify event-merging thresholds or the complete character-profile schema, so these should not be treated as established implementation details.

3. Iterative narrative planning: accept creative proposals only when the source supports them

The director turns the instruction into a story blueprint organized around narrative stages such as an opening, development, and climax. The orchestrator acts as an evidence checker, asking whether the relevant character belongs to the source, whether the proposed event exists, and whether a stage can be realized using actual footage. Verification starts at the story and character levels; once the agents agree, confirmed context is incorporated into the blueprint and the process descends toward events and shots. When evidence is missing, the orchestrator requests a revision instead of filling the gap with loosely related material.

The cycle continues until the requirements are grounded in precise shots, producing a compiled script with explicit shot IDs. Ordering is therefore part of the blueprint rather than a cosmetic operation after retrieval: a requested flashback should follow the intended narrative stages, not blindly preserve source chronology. The same evidence requirement helps identify factually impossible instructions, such as requesting a character from a film in which that character never appears. However, the paper does not fully specify rejection thresholds, iteration budgets, termination caps, or failure fallback procedures. The supported mechanism is a proposal–validation–revision loop grounded in source memory, not a formally convergent search algorithm.

4. Supervised tool execution: realize and revise an approved edit

After the manager checks the workflow and validates the compiled script against the instruction, the editor assembles shots and invokes external tools. The paper lists four operation categories: background music, text summaries or titles, video covers, and transition effects. The editor selects these operations using the instruction and final script; they implement the requested presentation rather than replacing narrative understanding. Consequently, successful tool execution does not establish narrative coherence, and execution success must be measured separately from editing quality.

Agents communicate through structured messages in a shared cinematic compilation workspace, recording actions and responses in shared history. This history supplies dynamic context and acts as a recoverable version log, allowing a later modification to reload an existing compilation state without repeating the foundational analysis. The manager can inspect intermediate outputs and intervene when something goes wrong. Execution history and narrative memory are distinct: the former records task state, while the latter stores knowledge about the source footage, enabling different kinds of reuse.

A Worked Example

Figure 2 includes a Green Book instruction that places a thanking scene before earlier events involving help with letter writing and the recipient's reaction, with a cover and editing effects. Identity and dialogue alignment associates the relevant speech with the correct characters, while hierarchical narrative memory links the thanking and letter-writing events to their source contexts. The director proposes an outcome-first blueprint, and the orchestrator verifies its events and shots instead of treating the original chronology as the requested order. Only the confirmed shot script proceeds to the editor for presentation effects; this is an explanation of the mechanism, not a reported turn-by-turn execution trace.

Loss & Training

CineAgents is training-free and uses Gemini-2.5-pro to organize agent reasoning; it introduces no new end-to-end training objective. Experiments use a server with four NVIDIA A100 GPUs, but this should not be read as Gemini itself running on those local GPUs, since the paper also reports API costs. Section 6.3 gives an average of approximately one hour and US\(14 in API fees to construct narrative memory for a film or television source, followed by about six minutes and US\)0.62 per instruction. Because the memory is reusable, cold-start analysis and subsequent instruction execution must be distinguished: six minutes is not the reported end-to-end latency for an unseen film.

Key Experimental Results

Main Results

CineBench contains more than 500 instruction–compilation pairs from over 70 English- and Chinese-language films and series released between 1930 and 2020. Instructions cover source selection, target content, temporal requirements, and editing operations; five professional editors participate in annotation. Agreement is checked on a random 20% subset, with Cohen's Kappa of 0.61, and 30% of the samples contain factually inconsistent negative instructions. The paper does not clearly specify training, validation, and test partitions; the results below are the reported CineBench evaluation, not an inferred independent test split.

Method	Shot F1 ↑	TCS ↑	ARR ↑	CQ ↑
Gemini-2.5-pro	41.59	37.41%	23.85%	8.67
MetaGPT	55.73	46.21%	52.31%	8.85
LAVE	44.60	35.37%	40.28%	8.53
CineAgents	64.13	52.09%	87.23%	9.01

Source: the comparison portion of Table 1; metric definitions are in Section 3.3, and higher is better throughout. F1 measures shot selection against professional reference annotations, retaining the paper's reported scale; cross-example aggregation is insufficiently specified, so recomputing it from the displayed average precision and recall would be inappropriate. TCS measures the duration of correctly ordered shot sequences relative to the generated video's total duration, while ARR measures correct rejection of factually impossible instructions. CQ is a VLM-assigned overall compilation-quality score on a 1–10 scale, not a success rate.

This is not a uniform comparison in which every baseline independently processes raw videos: the VLLM baselines and MetaGPT receive scripts produced by CineAgents. The VLLMs also use a two-stage chain-of-thought prompt that retrieves and then reorders shots, focusing the comparison on compilation ability. The improvement over Gemini therefore supports the structured workflow, but cannot be attributed entirely to better visual parsing; an equal-inference-budget isolation study is also absent.

Ablation Study

CineBench configuration	Shot F1 ↑	ARR ↑	ESR ↑
Full CineAgents	64.13	87.23%	92.76%
Without dialogue character matching, DCM	61.17	85.34%	89.91%
Without hierarchical narrative memory, HNM	58.04	72.96%	84.60%
Retrieve-and-rank replaces iterative narrative planning, INP	59.52	78.81%	85.07%

Source: the ablation portion and full-system row of Table 1, with variant definitions in Section 5.3; ESR is the fraction of instructions processed without system errors. Removing HNM decreases F1 by 6.09 and ARR by 14.27 percentage points, the largest decreases among these three ablations. Replacing INP decreases F1 by 4.61 and ARR by 8.42 percentage points, linking evidence verification to recognizing impossible requests as well as organizing footage. These are differences between reported values, not statistical-significance claims: corresponding error bars and confidence intervals are not provided.

Key Findings

• Relative to MetaGPT, CineAgents improves F1 by 8.40 and TCS by 5.88 percentage points, indicating gains in both selection and ordering rather than only aesthetic ratings.
• The HNM ablation degrades more than the INP ablation, suggesting that reliable source knowledge is a prerequisite for planning; this does not establish that memory is universally more important than planning.
• Each human-preference study samples 50 examples and recruits 25 volunteers to assess prompt alignment, sequence alignment, narrative coherence, and overall quality; the animated extension additionally covers 10 films and 50 instructions.
• The text reports leading preference scores, but the local extraction does not preserve Figure 5's numerical bar values, so no unverifiable preference percentages are reproduced here.

Highlights & Insights

• Narrative representation matters more than merely adding agent roles. Identity, dialogue attribution, and event structure make shared information verifiable; otherwise several agents may all reason from the same inaccurate caption.
• Creative freedom remains bounded by existing evidence. The director proposes a story and the orchestrator grounds it, a useful distinction for media reorganization that must not invent source events.
• Source memory and execution history support separate forms of reuse. One amortizes film analysis, while the other enables revisions to an existing edit, explaining the workflow's practical appeal.

Limitations & Future Work

• Author-stated limitation: off-the-shelf character detection and tracking can fail in complex cinematic scenes, cascading into compilation errors; stronger perceptual modules are left for future work.
• Reader assessment: subtitles, character metadata, and voiceprint anchors impose meaningful input assumptions; robustness to missing subtitles, dubbing changes, and unusual character appearances is not separately quantified.
• Reader assessment: several quality metrics use a VLM while the system itself depends on Gemini; the human study is useful but does not eliminate evaluator bias or the subjectivity of reference compilations.
• Reproducibility boundary: the available full paper has no appendix and does not provide complete prompts, iteration budgets, clear data splits, or uncertainty statistics; no confirmed project code link was found.
• Usage boundary: the authors restrict the cinematic materials to non-commercial academic research; benchmark results do not authorize redistribution or commercial reuse of the underlying footage.

Related Work & Insights

• Versus LAVE: LAVE assists editing of casual videos, whereas this work emphasizes cinematic characters, event dependencies, and nonlinear narrative instructions that local content relevance alone cannot resolve.
• Versus MetaGPT: generic multi-agent organization is supplemented here by cinema-specific memory and coarse-to-fine evidence verification; the comparison suggests that collaboration does not substitute for an appropriate domain representation.
• Versus FilmAgent / MovieAgent: those systems concern new cinematic content generation, while CineAgents selects and rearranges existing shots, making source-constrained narrative correctness rather than pixel synthesis its central problem.

Rating

• Novelty: 4/5. Combines an open-ended cinematic compilation benchmark with an evidence-constrained multi-agent workflow; novelty is chiefly in task and system design.
• Experimental Thoroughness: 3/5. Includes comparisons, real ablations, and human preferences, but splits, budget controls, and uncertainty reporting remain limited.
• Writing Quality: 3/5. The problem and role separation are clear, while implementation and evaluation details do not support full reproduction.
• Value: 4/5. Relevant to long-video source management and revisable editing workflows, subject to perception errors, costs, and source licensing.