Skip to content

From Visual Cues to Spoken Narration: Rethinking Audio Description Generation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/multimodal-ai-lab/Cue2Narrate
Area: audio_speech
Keywords: Audio Description, Temporal Window Localization, Multimodal Video Understanding, Description Ranking Loss, Long-form Video Benchmark

TL;DR

This paper re-evaluates automatic Audio Description (AD) by showing that the visual cue window (what to describe) and the spoken narration window (when to voice it) routinely exhibit temporal offsets, introducing Cue2Narrateโ€”a two-stage framework pairing an audio-visual dual-head localizer with a window-conditioned VLM trained via a Description Ranking Lossโ€”and releasing the LongLSMDC long-form benchmark.

Background & Motivation

Audio Description (AD) serves as a vital assistive medium that provides spoken narration of essential on-screen visual occurrences during dialogue gaps, granting blind and visually impaired audiences access to film and television. In professional practice, human describers continuously make two coupled yet distinct temporal decisions: deciding what visual action or scene change warrants narration, and determining when that description should be voiced within an unvoiced dialogue gap so as not to overlap with character speech. Professional AD production is notoriously labor-intensive and costly (running between $15 and $75 per finished minute), restricting accessibility coverage to only a tiny fraction of released media and underscoring the urgent need for scalable automated solutions.

However, existing literature significantly trivializes this challenge. Prior methodologies almost universally operate under an oracle-window assumption on pre-segmented short video clips, effectively reducing the AD problem to generic video captioning. In doing so, these approaches bypass the core challenge of detecting multiple narration intervals in continuous, untrimmed long-form video. Furthermore, existing benchmarks have reinforced a false dichotomy by offering only single-window annotations: datasets such as CMD-AD and TV-AD supply spoken-slot intervals aligned to dialogue pauses without visual cue boundaries, whereas LSMDC and MAD-Eval supply visual event boundaries without the paired spoken delivery slots. In cinema, narrative events and the dialogue silences available to describe them rarely coincide perfectly; in fact, descriptions are voiced slightly before, during, or after the visible action. Generating descriptions directly from frames extracted at the spoken pause inevitably causes semantic misalignment and narrative distortion.

To resolve this limitation, automatic AD must be formulated to simultaneously localize where the visual evidence resides and where the resulting spoken narration belongs in continuous video. Core idea: reframe audio description as joint localization of temporally offset visual cue and spoken narration windows paired with concise narration generation, using an audio-visual dual-head localizer to independently isolate dialogue gaps and visual events, and conditioning a VLM equipped with a Description Ranking Loss to suppress verbose captioning priors in favor of compact narrative descriptions.

Method

Overall Architecture

Cue2Narrate factorizes the untrimmed audio description problem into a two-stage pipeline. Stage 1 consists of the Audio-Visual Dual-Head (AVDH) Localizer, which ingests raw long-form video and its accompanying dialogue audio track to jointly detect multiple narrative opportunities, predicting paired temporal windows for each instance: a visual cue window \(\hat{y}^{\text{vn}}_i\) and a spoken narration window \(\hat{y}^{\text{sp}}_i\). Stage 2 comprises the Window-Conditioned AD Generator (WC-AD Gen), which extracts video frames strictly from the predicted visual cue window \(\hat{y}^{\text{vn}}_i\) and feeds them alongside structured instructions and character bank references into a LoRA-adapted vision-language backbone. Trained with a Description Ranking Loss (DRL) that penalizes verbose vanilla captions in favor of concise ground-truth AD style, the generator outputs a single-sentence description \(\hat{d}_i\) that is voiced inside the predicted spoken narration window \(\hat{y}^{\text{sp}}_i\).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    IN["Untrimmed Long-form Clip Input<br/>Video Stream V + Audio Track A"] --> S1
    subgraph S1["Stage 1: Audio-Visual Dual-Head Localizer (AVDH)"]
        direction TB
        FPN["Temporal Pyramid & Additive Fusion<br/>VideoMAE Visual Features + Log-Mel Audio Features"] --> H_VN["Visual Cue Head<br/>Detects on-screen actions & scene cues"]
        FPN --> H_SP["Spoken Narration Head<br/>Aligns with dialogue silence gaps"]
        H_VN --> W_VN["Paired Temporal Windows<br/>Visual cue window y_vn"]
        H_SP --> W_SP["Paired Temporal Windows<br/>Spoken narration window y_sp"]
    end
    W_VN --> S2
    subgraph S2["Stage 2: Window-Conditioned Generator (WC-AD Gen)"]
        direction TB
        PROMPT["Structured Prompt Construction<br/>Sampled frames + Task instruction + Character Bank"] --> VLM["LoRA-adapted Vision-Language Model<br/>Qwen-VL generates single-sentence AD"]
        DRL["Description Ranking Loss (DRL)<br/>Suppresses verbose caption prior via contrast"] -.->|Training objective| VLM
    end
    W_SP --> OUT["Final Audio Description Delivery<br/>Insert narration d_i into dialogue gap y_sp"]
    VLM --> OUT

Key Designs

1. Audio-Visual Dual-Head Localizer: element-wise additive pyramid fusion decoupling temporal offsets

Dialogue pauses reveal acoustic availability for inserting speech but cannot ascertain whether an accompanying visual occurrence is narratively salient; conversely, visually compelling actions often unfold across continuous dialogue, precluding immediate vocal insertion. To reconcile these complementary dynamics, the AVDH Localizer processes 768-dimensional visual tokens from a frozen VideoMAE backbone and 130-dimensional log-mel spectrogram audio features resampled to the visual timeline. The visual pathway passes through lightweight temporal self-attention (LMSA) and a ConvTransformer backbone to yield a 4-level temporal pyramid \(\{F_v^\ell\}_{\ell=1}^L\) with strides spanning \(T, T/2, T/4, T/8\). The audio stream is projected to dimension 256, refined with self-attention, and average-pooled to match each pyramid level \(\{F_a^\ell\}_{\ell=1}^L\). The modalities are unified via pyramid-level element-wise addition: $\(F^\ell = F_v^\ell + F_a^\ell\)$ Compared to channel concatenation or symmetric averaging, additive fusion preserves modality-specific peak activations without diluting subtle acoustic silence boundaries. Two independent detection headsโ€”a Visual Head and a Spoken Headโ€”operate over this shared pyramid. Each head incorporates a Sigmoid Focal classification branch and a regression branch predicting non-negative boundary offsets \((\delta_l, \delta_r)\) relative to temporal anchor \(t\), yielding predicted intervals \([t - \delta_l \cdot s_\ell, t + \delta_r \cdot s_\ell]\). The spoken head additionally incorporates a learned temporal-position prior over normalized clip time to capture macro-level narration rhythms. By decoupling the prediction into dual heads refined via Soft-NMS, the architecture naturally models arbitrary temporal offsets between event visibility and verbal delivery.

2. Window-Conditioned AD Generation with Description Ranking Loss: suppressing captioning priors for concise narration

Standard pre-trained VLMs (such as Qwen2.5-VL-7B and Qwen3-VL-8B) exhibit an inherent bias toward outputting exhaustive, paragraph-length scene descriptions. Professional audio descriptions, however, must adhere to a strict register: delivering concise, single-sentence depictions of salient movements and character gestures that fit into brief speech pauses without overshadowing background audio. Standard supervised cross-entropy fine-tuning (\(L_{\text{sft}}\)) is insufficient to overwrite this ingrained verbosity prior given the modest size of movie AD datasets. To address this, the authors introduce the Description Ranking Loss (DRL). An offline negative caption \(y^-_i\) is gathered by querying the frozen VLM on visual frames \(V_i\) with an unconstrained captioning prompt. During Stage 2 fine-tuning, the model computes token-normalized log-likelihoods for both the concise reference AD \(y^+_i\) and the verbose negative \(y^-_i\): $\(s^+_i = \frac{1}{|y^+_i|} \sum_{t} \log P(y^+_{i,t} \mid y^+_{i,<t}, V_i, P_i), \quad s^-_i = \frac{1}{|y^-_i|} \sum_{t} \log P(y^-_{i,t} \mid y^-_{i,<t}, V_i, P_i)\)$ $\(L_{\text{drl}} = -\log \sigma \left( \frac{s^+_i - s^-_i}{\tau} \right)\)$ where \(\tau=0.5\) serves as the temperature scaling factor. DRL directly penalizes the model when it assigns high probability mass to verbose, scene-summarizing structures, explicitly conditioning the LoRA adapters to favor the brisk, action-focused phrasing demanded by real-world AD standards.

Loss / Training Strategy

The Stage 1 AVDH Localizer is trained end-to-end via a multi-task formulation: $\(L_{\text{stage1}} = L_{\text{vn}}^{\text{cls}} + L_{\text{vn}}^{\text{reg}} + \lambda_{\text{sp}} (L_{\text{sp}}^{\text{cls}} + L_{\text{sp}}^{\text{reg}})\)$ where \(\lambda_{\text{sp}} = 1\). The classification components utilize Sigmoid Focal Loss, while the regression components employ 1D center-based DIoU loss. LongLSMDC provides three categories of training instances: 37,302 paired annotations, 21,295 visual-only annotations, and 20,197 spoken-only annotations. Training proceeds asymmetrically: single-modality annotations update only their corresponding head, whereas paired annotations supervise both heads simultaneously, providing the structural constraint that prevents the heads from collapsing onto a single shared window.

Stage 2 optimizes the LoRA adapters (\(r=16, \alpha=32\), dropout \(0.05\)) on language model projections and multimodal connectors using: $\(L_{\text{stage2}} = L_{\text{sft}} + \lambda L_{\text{drl}}\)$ with weighting parameter \(\lambda = 0.2\). The vision encoder remains frozen throughout, ensuring efficient adaptation while sampling 32 frames during training and 64 frames at inference.

Key Experimental Results

Main Results

Evaluations are conducted on LongLSMDC, featuring untrimmed clips of up to 8 minutes (\(\sim 6.5\) minutes on average). Localization in Stage 1 is assessed using standard Temporal Action Localization (TAL) average mAP across temporal IoU thresholds \(\{0.2, 0.3, 0.4, 0.5, 0.6\}\), alongside zero-shot (ZS) and fine-tuned (FT) transfer to CMD-AD and MAD-Eval.

Method Modality
Video / Audio
LongLSMDC Visual mAP LongLSMDC Spoken mAP LongLSMDC Avg. mAP CMD-AD Spoken mAP (FT / ZS) MAD-Eval Visual mAP (ZS)
Spectrogram Baseline โœ— / โœ“ 1.4 1.3 1.4 1.7 / 1.7 23.1
Gemini 2.5 Pro (Prompted) โœ“ / โœ“ 7.1 4.4 5.8 โ€“ / โ€“ โ€“
ActionFormer (Video-only) โœ“ / โœ— 32.0 28.5 30.3 โ€“ / โ€“ 63.7
Audio-ActionFormer (Audio-only) โœ— / โœ“ 35.2 39.1 37.1 39.2 / 25.0 โ€“
ActionFormer Dual-Head Baseline โœ“ / โœ“ 39.0 38.1 38.5 โ€“ / โ€“ โ€“
Cue2Narrate (Ours) โœ“ / โœ“ 42.2 42.0 42.1 41.0 / 26.5 70.0

End-to-end generation quality is benchmarked under predicted temporal windows (S1โ†’S2, matched to ground truth at [email protected]) as well as under oracle ground-truth spoken windows (GTโ†’S2), evaluated by CIDEr (C), Action Score (AS), and Recall@1/5 (R@1/5).

Evaluation Setting Base Model & Variant LongLSMDC CIDEr (C) LongLSMDC Action Score (AS) LongLSMDC R@1/5 CMD-AD (Zero-Shot) CIDEr (C) CMD-AD Action Score (AS)
Ground-Truth Windows (GTโ†’S2) UniAD (MAD-pretrained checkpoint) 10.1 โ€“ โ€“ 21.8 โ€“
AutoAD-Zero (Training-free) 14.2 โ€“ 31.3 17.7 25.5
Shot-by-Shot (Training-free) 16.9 25.3 28.4 28.4 33.0
Qwen2.5-VL-7B (SFT baseline) 30.1 41.7 43.8 20.5 35.4
+ Cue2Narrate (+DRL) 32.5 40.9 43.3 22.3 35.0
Qwen3-VL-8B (SFT baseline) 36.0 42.0 46.8 23.8 35.1
+ Cue2Narrate (+DRL) 37.3 42.6 47.0 25.1 36.0
Predicted Windows (S1โ†’S2) AutoAD-Zero (Training-free) 5.1 20.7 23.4 10.1 30.9
Qwen2.5-VL-7B (SFT baseline) 21.7 38.8 37.3 7.5 23.4
+ Cue2Narrate (+DRL) 22.2 38.3 37.0 9.2 25.1
Qwen3-VL-8B (SFT baseline) 23.8 39.4 39.3 8.8 24.8
+ Cue2Narrate (+DRL) 25.2 40.2 40.1 11.5 27.8

Ablation Study

Ablation experiments investigate audio-visual fusion techniques, cross-head degradation, and the contribution of DRL on LongLSMDC.

Configuration / Variant Visual mAP Spoken mAP Cross-head Spoken mAP Note / Finding
Cue2Narrate (Element-wise Addition) 42.2 42.0 37.2 Optimal fusion preserving modality-specific feature peaks
Concatenation Fusion (Stack / Concat) 40.5 40.2 35.8 Forces early joint representation, diluting sparse silence cues (-1.7 mAP)
Averaging Fusion (Average) 40.7 40.1 35.5 Dampens both streams symmetrically, trailing addition by 1.5โ€“2.0 mAP
Single-Head Shared Window 37.5 37.1 โ€“ Cannot resolve temporal offsets present in ~88% of paired data
Without DRL (SFT Only, Qwen3-VL) โ€“ โ€“ โ€“ CIDEr under GT windows drops from 37.3 to 36.0 (-1.3)
Closed-source VLM (Gemini 2.5 Pro) 7.1 4.4 โ€“ Prompted timestamps degrade severely as tIoU threshold tightens

Furthermore, coverage analysis of predicted spoken windows reveals that under [email protected] greedy matching, Cue2Narrate achieves 27.5% precision, 94.7% recall, and 42.7% F1 (leaving only 5.3% of true narration opportunities missed). Under stricter [email protected], recall remains robust at 73.0% (32.9% F1), demonstrating dependable event capture in long sequences.

Key Findings

  • Acoustic signals drive slot availability, while visual cues establish narrative salience: Audio-only localization reaches 39.1 spoken mAP compared to 28.5 for video-only models, showing that acoustic silence is the premier indicator of spoken insertion opportunities. However, without video signals, audio models fail to identify which silences contain narratable actions; fusing visual features raises spoken mAP to 42.0 and boosts visual mAP from 1.3 to 42.2.
  • Cross-head evaluation validates temporal offset modeling: Evaluating the spoken head's predictions against visual ground truth results in a sharp 4.8 mAP drop (42.0 vs 37.2), matching the structural fact that 88% of paired training examples exhibit temporal offsets. Single-head architectures inevitably collapse this temporal displacement.
  • DRL aligns model phrasing with professional audio description register: Prompting Gemini 2.5 Pro achieves a high LLM-Eval score (2.80) but a meager CIDEr of 17.3, reflecting verbose, unfocused prose. Integrating DRL penalizes uninformative conversational padding, yielding consistent +1.3 to +1.4 CIDEr improvements across backbones and settings.

Highlights & Insights

  • Pioneering paired-window formulation for continuous AD: Formulates the separation between the visible event (visual cue window) and spoken insertion gap (spoken narration window), establishing the realistic LongLSMDC benchmark and retiring the artificial oracle-window assumption.
  • Lightweight multi-scale additive feature fusion: Implements element-wise addition across a 4-level temporal pyramid with only 6.64M parameters, outperforming parameter-heavy concatenation and cross-attention baselines while maintaining clean modality separation.
  • Self-contrasted offline negative ranking: Leverages unguided VLM-generated captions as negative samples within a simple pairwise ranking loss, aligning the generative distribution with concise accessibility standards without expensive expert annotations.

Limitations & Future Work

  • Independent two-stage execution: Stage 1 temporal localization and Stage 2 text generation operate independently; end-to-end joint training with text-guided feedback could refine temporal boundaries.
  • Isolated sentence generation lacking cross-event context: Descriptions are produced one clip at a time without conditioning on adjacent dialogue or previously generated narrations, occasionally resulting in redundant character referents or abrupt tone shifts.
  • Face matching failures in character banks: CharBank identification depends on facial recognition; characters shot from behind, positioned off-screen, or obscured in shadow cannot be matched, causing descriptions to revert to generic nouns.
  • vs AutoAD Series (AutoAD / AutoAD-II / AutoAD-III): Earlier AutoAD iterations relied on pre-segmented clips or basic binary classification over dialogue pauses, addressing single-window formulations; Cue2Narrate handles untrimmed 8-minute clips and predicts paired, temporally offset windows.
  • vs CA3D: CA3D restricts prediction to a single isolated event per pre-cut clip; Cue2Narrate manages dense, multi-segment localization in long-form video, achieving 90.0% top-1 F1 on MAD-Eval (surpassing CA3D's 65.3%).
  • vs Shot-by-Shot: Shot-by-Shot relies on training-free shot-boundary heuristics; Cue2Narrate's learned dual-head localizer and DRL-aligned generator outperform Shot-by-Shot in blind human evaluation across key event coverage, temporal coherence, and narrative clarity with a 65% to 35% win rate.

Rating

  • Novelty: โญโญโญโญโญ Formulates the temporal offset between visual cue and spoken delivery windows in long-form video.
  • Experimental Thoroughness: โญโญโญโญโญ Thoroughly evaluated across localization, generation, cross-dataset transfer, and human preference studies.
  • Writing Quality: โญโญโญโญโญ Rigorously articulated problem definition with comprehensive appendices and architectural clarity.
  • Value: โญโญโญโญโญ Sets an essential benchmark and practical foundation for automated media accessibility and fine-grained video understanding.