OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video¶

Conference: ECCV 2026
Paper: ECCV Paper Page
Project: OmniScript
Code: https://github.com/TencentARC/OmniScript
Area: Video Understanding
Keywords: Audio-Visual Script Generation, Long Video Understanding, Character Memory, Chain-of-Thought, Segmented Reward

TL;DR¶

OmniScript transcribes cinematic videos into temporally grounded hierarchical scripts, combining character-memory-assisted data construction, deep audio-visual fusion, plot reasoning, and event-level segmented rewards to achieve 37.7 event Overall F1 and 69.3 [email protected] with an 8B model on 5-minute videos, without uniformly surpassing proprietary models.

Background & Motivation¶

Cinematic understanding requires more than summarizing what happens: a recurring character may change clothes or continue speaking off-screen, while music and expressions can alter the meaning of dialogue. Question answering or short-clip captioning can omit these details, but script transcription must identify who does or says what, and when, while separating actions, expressions, dialogue, and sounds into searchable fields. Role-aware narration such as Movie101 and dense video captioning provide foundations, but generally do not jointly address this fine-grained field decomposition and cross-scene narrative continuity.

The bottleneck is not simply context-window size. Independent segment annotation can assign different identities to the same person; directly generating long JSON outputs can disconnect character relationships, event order, and local details. Evaluation is equally difficult: global semantic similarity may reward a broadly correct summary while overlooking a missing line of dialogue or a mistimed action. The paper also notes that describing approximately 2 minutes of video can require approximately 4,000 output tokens, making detail density itself a generation burden.

The paper designs data, modeling, and evaluation around the script structure rather than merely adding a prompt to a general-purpose model. Core Idea: establish consistent plot supervision through character memory, teach an audio-visual model to reason about plot and character relationships before decoding event records, and constrain omissions and mismatches with locally aligned event rewards.

Method¶

Overall Architecture¶

The input is a movie or television clip with its original audio track, and the output is a JSON-style hierarchical script. The top level contains the title, duration, and character list; scenes describe location, environment, and time attributes; events specify timestamps, characters, and at least one content field: dialogue, action, expression, or audio cue. An event can also belong to the environment, so every sound need not be assigned to a person.

Two paths must be distinguished. Offline data construction maintains character-profile memory across segments, whereas the deployed model processes synchronized audio-visual input through AV-DeepStack fusion and plot reasoning to generate a script. Segmented rewards are used during post-training, not by an external evaluator required at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Raw["Raw cinematic video"] --> Memory["Character-Memory<br/>Annotation"]
    Raw --> Fusion["Deep Audio-Visual<br/>Fusion"]
    Memory -.->|Offline supervision| Reason["Plot-Guided<br/>Decoding"]
    Fusion --> Reason
    Reason --> Script["Scene and event script"]
    Script -.->|Training predictions| Reward["Event-Segmented<br/>Reward"]
    Human["Human-annotated script"] -.-> Reward
    Reward -.->|GRPO post-training| Reason

The connections distinguish inference data flow from training supervision. CPM belongs to the teacher annotation pipeline; the paper does not establish that the 8B inference model maintains the same cross-segment profile manager. Memory in the data engine should therefore not be presented as a demonstrated deployment capability.

Key Designs¶

1. Character-Memory Annotation: resolve cross-segment identity before producing event-level supervision

The authors use PySceneDetect and scene boundaries to extract approximately 45K semantically coherent segments, typically shorter than 5 minutes, from over 10K raw cinematic videos. Gemini-2.5-Pro reads the current audio-visual input together with historical information in a Character Profile Manager (CPM), generates character-centric plot reasoning, and updates evolving attributes such as clothing. This makes it possible to associate a person initially described by clothing with a named character appearing later, instead of attempting to repair identities only after concatenating annotations.

CPM also uses lazy naming: uncertain identities retain provisional IDs until explicit naming in later dialogue allows retroactive resolution and duplicate merging. Gemini then combines the plot description with the original audio-visual input to generate detailed scripts. DeepSeek retrospectively derives intermediate plot and relationship reasoning from those scripts, forming Video → Thinking → Script supervision. These are teacher-constructed reasoning trajectories, not observations of a model's actual internal reasoning process.

2. Deep Audio-Visual Fusion: repeatedly expose language reasoning to synchronized sound and vision

The model uses Qwen3VL-8B as its backbone and initializes its audio encoder from Whisper large-v3. Visual frames and raw audio become temporally indexed tokens, with visual and acoustic representations paired within corresponding temporal units. This prevents visible actions and off-screen speech from being treated as unrelated temporal context. Audio is not included merely to recover subtitle content: narration, environmental sounds, and background music are also part of the task.

Extending Qwen3-VL's DeepStack approach, AV-DeepStack injects paired audio-visual features into multiple Transformer layers through residual multimodal adapters rather than concatenating them only at the input. Later semantic processing can repeatedly access perceptual evidence, distinguishing the design from directly fine-tuning a vision-only backbone for script generation. The main text does not specify the exact injection-layer configuration, sampling rates, or token-compression details, so these implementation hyperparameters cannot be reconstructed from it.

3. Plot-Guided Decoding: establish relationships and plot progression before filling local fields

The model first produces a chain-of-thought (CoT) trace containing a plot summary and character-relationship state, then generates scene context and ordered events. This intermediate state addresses the dependence between identity attribution and local details: without first resolving who is speaking to whom, a lexically correct transcription can still assign a line to the wrong character.

Decoding proceeds from scenes to events, separating dialogue, actions, expressions, and sounds into distinct fields. Character actions and environmental audio can be represented as different records at the same time instead of being compressed into one holistic sentence. The dedicated plot CoT helps in the ablation, but this does not imply that enabling a generic model's Thinking mode necessarily helps; several general-purpose Thinking variants in the main table perform worse than their non-thinking counterparts.

4. Event-Segmented Reward: evaluate locally aligned details instead of only the overall narrative

Evaluation begins with text-content-guided event alignment. Under temporal-proximity and non-overlapping-interval constraints, dynamic programming solves an order-preserving weighted interval scheduling problem. This separates whether two events describe the same content from whether their timestamps are accurate, avoiding the complete rejection of semantically correct but slightly shifted predictions that can occur with purely temporal matching. An LLM also helps resolve character aliases, followed by a semantically constrained bipartite mapping that prevents inconsistent identity assignments.

After alignment, an LLM scores semantic fields such as actions and expressions; normalized Levenshtein distance measures dialogue differences; character identities are compared exactly after mapping. Field scoring accounts for both predicted and ground-truth counts, so producing more content does not improve recall for free. GRPO uses these localized field evaluations as segmented rewards rather than assigning a single global semantic reward to the entire script. The main text does not disclose the full reward weights, so no complete reward-combination formula can be inferred.

Following Section 4.1, let \(S_g\) denote field similarity for an aligned group and let \(N_{\mathrm{pred}}\) and \(N_{\mathrm{gt}}\) denote predicted and ground-truth field counts. Soft-count precision, recall, and F1 are:

\[ P_{\mathrm{field}}=\frac{\sum_g S_g}{N_{\mathrm{pred}}},\qquad R_{\mathrm{field}}=\frac{\sum_g S_g}{N_{\mathrm{gt}}},\qquad F1_{\mathrm{field}}=\frac{2P_{\mathrm{field}}R_{\mathrm{field}}}{P_{\mathrm{field}}+R_{\mathrm{field}}}. \]

Overall is the mean of field-level F1 scores. Temporal evaluation separately counts aligned groups whose tIoU reaches a threshold and computes the harmonic mean of temporal precision and recall. Consequently, [email protected] in the tables is temporal F1 at that threshold, not mean IoU. The protocol defines thresholds of 0.1, 0.3, 0.5, 0.7, and 0.9, but the main tables only show 0.1; they do not establish that precise boundary localization is solved.

A Worked Example¶

In Figure 1, Dai Xi wipes a staircase handrail at home, stops, speaks, and receives a response from a holographic AI. The model needs to establish their identities and interaction before recording her action, expression, dialogue, and the light background music in the relevant events, instead of merely summarizing that she is talking with an AI at home.

The figure shows a Dai Xi event at 00:04 and a holographic AI event at 00:08. The character switch requires identity understanding; the AI's synthesized electronic voice and voiceover type require acoustic evidence; the handrail action requires visual evidence. This illustrates output structure and evidence allocation, not an additional success-rate experiment. Character memory supplies identity history during cross-segment annotation only.

Loss & Training¶

The steps actually listed in the main text can be understood as three foundational training stages followed by RL refinement:

Modality alignment: approximately 1M bilingual Chinese/English cinematic samples with timestamped ASR supervision train the new audio projector while Whisper, ViT, and the LLM remain frozen. Random video-frame masking discourages reliance on vision alone.
Multimodal pretraining: approximately 2.4M in-domain videos support full fine-tuning of core components through ASR with/without timestamps, summarization, dense captioning, and temporal grounding. Random frame masking is retained.
SFT: approximately 45K videos, comprising 21K horizontal movie/TV clips and 24K vertical short-drama clips, supervise plot CoT and structured scripts, with random subtitle masking.
RL: GRPO with segmented rewards is applied to a small, high-quality human-annotated script set. The main text does not give its exact size or the complete optimization hyperparameters.

Section 5.2 calls the recipe four-stage while also saying it is followed by RL; Section 6.1 calls it three-stage. This note follows the four enumerated operations rather than inventing a fifth stage. Section 5.2 specifically identifies the audio projector as trainable during alignment, whereas Section 6.1 uses the broader term modality projectors; the exact implementation still requires code verification.

Key Experimental Results¶

Main Results¶

The benchmark contains 10 full cinematic works totaling 19.9 hours, approximately 1.4K scenes, and over 16.8K events, with an average density of 14.1 events per minute. Annotators refine automatically generated drafts, and senior experts review each script. Duration subsets contain 200 clips of 5 minutes, 100 of 10 minutes, 50 of 15 minutes, 40 of 20 minutes, 30 of 25 minutes, and 25 of 30 minutes. They derive from the same source works and should not be treated as a large collection of independent films.

The following selection from Tables 1 and 2 reports 5-minute results on the paper's percentage scale. Overall is the mean field F1 at the corresponding hierarchy level; both temporal columns are [email protected].

Model	Parameters	Event Overall	Dialogue F1	Event Temporal F1	Scene Overall	Scene Temporal F1
Qwen3VL-8B	8B	27.7	49.6	47.6	40.9	60.6
Qwen3VL-235B-A22B	235B / 22B activated	33.0	58.6	62.0	53.4	72.8
Gemini-3-pro	Not reported	38.9	68.8	64.4	57.0	75.3
Gemini-2.5-flash	Not reported	43.6	75.5	74.3	52.3	69.6
Seed-2.0-pro	Not reported	40.9	68.1	67.1	57.2	75.5
OmniScript	8B	37.7	72.2	69.3	52.4	74.6

Relative to Qwen3VL-235B-A22B, OmniScript improves event Overall by 4.7 percentage points and event temporal F1 by 7.3 points. However, its scene Overall is 52.4 versus 53.4, a 1.0-point deficit. Its dialogue and event temporal scores exceed Gemini-3-pro, but event Overall remains 1.2 points lower; the selective advantages should not be presented as uniform superiority.

Ablation Study¶

Table 3 separates plot CoT from reward design. All five configurations are retained here to avoid conflating the effects of adding RL and replacing global rewards with segmented rewards.

Training Config	CoT	Reward	Dialogue F1	Event Overall	[email protected]
SFT	No	None	68.2	35.3	66.6
SFT	Yes	None	71.0	37.0	68.9
SFT+RL	No	Segmented	70.9	37.0	69.0
SFT+RL	Yes	Global	69.0	37.0	68.7
SFT+RL	Yes	Segmented	72.2	37.7	69.3

CoT alone raises Overall from 35.3 to 37.0, a 1.7-point gain. With CoT enabled in both RL configurations, segmented rather than global rewards increase Overall from 37.0 to 37.7 and Dialogue F1 from 69.0 to 72.2. Not every field improves: Table 3 reports audio scores of 12.3 with global rewards and 11.6 with segmented rewards.

Table 4 masks subtitles at inference time to directly test whether dialogue recognition exploits visible text as a shortcut.

Model	Subtitles	Dialogue F1	Event Overall	[email protected]
Qwen3VL-235B-A22B	Visible	58.6	33.0	62.0
Qwen3VL-235B-A22B	Masked	7.7	18.6	45.1
Gemini-3-pro	Visible	68.8	38.9	64.4
Gemini-3-pro	Masked	60.9	36.6	60.3
OmniScript	Visible	72.2	37.7	69.3
OmniScript	Masked	63.8	34.4	67.0

Key Findings¶

Subtitle masking reduces OmniScript Dialogue F1 by 8.4 points, compared with 50.9 points for Qwen3VL-235B-A22B. This supports speech perception beyond subtitles, not complete independence from subtitles.
In the SFT comparison of Table 5, adding audio and its associated pretraining raises Dialogue F1 from 52.0 to 68.2 and Overall from 31.1 to 35.3, but [email protected] decreases from 68.2 to 66.6. This comparison neither isolates the contribution of layer-wise AV-DeepStack injection nor shows that audio improves every metric.
Figure 4 shows degradation as videos lengthen for all models; the authors report that OmniScript surpasses Seed-2.0-pro on the 30-minute subset. The cached text does not provide precise curve values, so only the trend is recorded here.
The final model's event audio F1 is only 11.6, with expression F1 at 31.9, indicating that dialogue progress is substantially ahead of mature understanding of subtle sounds and affect.

Highlights & Insights¶

Output structure is itself a supervision design. Separating identity, time, actions, dialogue, expressions, and sounds prevents fluent summaries from concealing missing fields and makes errors easier to locate.
Lazy naming suits narratives with delayed identity revelation. Keeping provisional identities and retroactively merging them when dialogue provides evidence better matches cinematic information flow than forcing an immediate definite name for every segment.
Segmented rewards align evaluation granularity with output granularity. Their transferable value is localized error feedback for long structured generation, rather than treating overall resemblance to a reference as sufficient.

Limitations & Future Work¶

The authors acknowledge difficulties with subtle attributes and temporal boundaries. Main tables emphasize the permissive [email protected] threshold; stricter thresholds and short-event performance should also be reported.
Dense annotation does not overcome the benchmark's restriction to 10 works. The supplied main text does not provide sufficient train/test work-level deduplication details, language-stratified results, or statistical intervals, leaving broader generalization to be verified.
Semantic alignment and field scoring rely on an LLM, so the rewards are not purely deterministic verification. Evaluator bias, identity-mapping errors, and teacher-generated style may influence scores.
CPM primarily operates during data construction. Additional experiments are needed to establish identity-state maintenance over longer continuous videos; 30-minute evaluation is not equivalent to validated full-movie transcription.
Fewer parameters do not establish low end-to-end cost. The paper does not report training cost, throughput, memory use, or long-script generation latency, while CoT adds output tokens; practical efficiency remains to be measured.

vs Movie101 / Movie101v2: Role-aware movie narration emphasizes readable descriptions. OmniScript additionally requires explicit scenes, events, multimodal fields, and temporal grounding, moving evaluation beyond descriptive fluency.
vs TimeChat-Captioner: The concurrent work provides timestamped scene-level audio-visual descriptions. OmniScript emphasizes atomic event fields to reduce ambiguity from mixing actions, intentions, and dialogue in coarse summaries.
vs Qwen3-VL / Qwen3-Omni: The former lacks raw audio input, while audio support in the latter does not automatically confer script-schema adherence. The evidence supports the importance of domain data and post-training, not a general claim that omni-modal models are weaker than vision-only models.
Research direction: Compare deep fusion with input-only fusion under identical data and audio budgets, jointly reporting strict temporal thresholds, identity consistency, and generation cost per minute of video to better isolate the sources of improvement.

Rating¶

Novelty: 4/5. Hierarchical V2S, character-memory annotation, and localized rewards form an integrated contribution, while the modeling components build on existing backbones and GRPO.
Experimental Thoroughness: 3/5. Duration comparisons, CoT/RL ablations, and subtitle experiments are useful, but source-work diversity, architectural ablations, and efficiency measurements remain limited.
Writing Quality: 4/5. The output schema and experimental questions are clear, while training-stage terminology and reproduction details need clarification.
Value: 4/5. The work offers actionable field definitions and evaluation for cinematic transcription, with deployment still constrained by fine-grained accuracy and generation cost.