Skip to content

AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Omni-modal Video Captioning, Audio-Visual Synergy, Decoupled-then-Fused Data Engine, Reinforcement Learning GRPO, Event-based Recall Benchmark

TL;DR

To tackle modality isolation and speech-centric bias in omni-modal video captioning, AVSCap constructs a 130K tri-modal corpus via a decoupled-then-fused pipeline, trains a 7B captioner with hybrid-reward GRPO, and introduces AVSCapBench with fine-grained atomic event recall.

Background & Motivation

The rapid advancement of large multimodal models (LMMs) is driving video understanding beyond traditional visual-centric perception toward comprehensive omni-modal joint reasoning. Within this evolving landscape, omni-modal video captioning stands as a critical benchmark for evaluating holistic scene comprehension. A high-quality omni-modal caption cannot simply concatenate visual descriptions with automatic speech recognition (ASR) transcripts; it must simultaneously recognize physical movements, environmental dynamics, human dialogue, sound effects (SFX), and background musical scores, articulating how these heterogeneous streams co-evolve across time.

However, existing state-of-the-art multimodal models suffer from an "illusion of integration," crippled by two systemic bottlenecks. The first is modality isolation: current architectures treat audio and video streams as weakly coupled parallel tracks. Even when a generated caption mentions both a visual action and a concurrent sound, it routinely omits explicit temporal or causal conjunctions (such as "as," "while," or "accompanied by"), resulting in descriptions that are superficially dense yet relational-poor. The second bottleneck is speech-centric bias: because acoustic pre-training is overwhelmingly dominated by speech data, models heavily prioritize verbal transcription while severely ignoring non-speech sounds, such as physical collisions, ambient noise, and the emotional nuances of background music. Compounding these weaknesses, standard evaluation protocols rely on n-gram text overlap (e.g., BLEU, CIDEr) or unimodal recognition benchmarks, which fail to penalize disjointed cross-modal reasoning.

To overcome these fundamental deficiencies, AVSCap centers omni-modal video captioning on explicit cross-modal event binding. Instead of relying on unguided end-to-end fitting, the framework decouples unimodal evidence collection before structured fusion to generate grounded training data, complemented by sample-efficient reinforcement learning that optimizes acoustic coverage and temporal binding. Core idea: construct an orchestrated 130K tri-modal corpus via a "decoupled anchoring-structured fusion-three-stage verification" engine, and train AVSCap-7B with Group Relative Policy Optimization (GRPO) using hybrid rewards across length bounds, regex speech recall, and LLM-adjudicated cross-modal synergy.

Method

Overall Architecture

The AVSCap framework establishes an integrated system encompassing data curation, two-stage policy training, and benchmark evaluation. At the data construction level, it introduces a decoupled-then-fused data engine that independently extracts unimodal visual and acoustic evidence before synthesizing unified captions with explicit temporal connectors, filtered through a three-stage verification pipeline to yield AVSCap-130K. At the model level, built upon Qwen2.5-Omni-7B, Stage 1 supervised fine-tuning (SFT) equips the backbone with baseline multimodal description capabilities augmented by 10K non-speech audio samples, while Stage 2 leverages Group Relative Policy Optimization (GRPO) on a small set of 2,000 videos to optimize acoustic completeness and cross-modal binding without requiring vast additional supervision.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Video & Audio Stream Separation"] --> B["Decoupled Unimodal Anchoring & Temporal Placeholders"]
    B --> C["Cross-Modal Fusion & Temporal Conjunction Orchestration"]
    C --> D["Three-Stage Automated Verification & Filtering"]
    D --> E["Two-Stage Multi-Grained SFT & Audio Augmentation"]
    E --> F["Hybrid-Reward GRPO Policy Optimization"]
    F --> G["AVSCap-7B Synergistic Caption Generation"]

Key Designs

1. Decoupled Unimodal Anchoring & Temporal Placeholders: Eliminating Cross-Modal Interference When multimodal foundation models directly process concurrent audio and video streams, dominant modalities often suppress subtle signals, leading to hallucinations or omissions. AVSCap resolves this by employing Gemini-3-Flash in an isolated manner for unimodal perception. On the visual side, the parser extracts scene layouts, characters, physical actions, object interactions, camera trajectories, and on-screen OCR text. Crucially, when an action indicates an auditory accompaniment, the parser inserts an empty [AUDIO] placeholder without describing any sound, entirely preventing premature acoustic hallucination while fixing the temporal anchor. On the audio side, the parser independently categorizes the acoustic stream into Speech (verbatim transcription), sound effects (SFX, following AudioCaps formatting for physical texture and intensity), and background music (Music, following MusicCaps formatting for rhythm, instrumentation, and mood), assigning each event an ordered, unique tag ID (e.g., (Speech-1), (SFX-1), (Music-1)).

2. Cross-Modal Fusion & Temporal Conjunction Orchestration: Explicitly Binding Cross-Modal Relations Conventional models frequently produce disjointed statements where visual events and auditory cues are described in separate sentences without causal linkage. To bridge this divide, the fusion module uses Gemini-3-Flash to systematically align each structured audio event with its corresponding visual [AUDIO] anchor. In synthesizing the omni-modal caption, the prompt strictly enforces the inclusion of temporal and associative conjunctions (such as "as", "while", "simultaneously", "accompanied by", and "in sync with"). All original audio tag IDs are preserved inline, maintaining verifiable traceability for downstream data filtering and policy rewards.

3. Three-Stage Automated Verification & Filtering: Multi-Tiered Auditing against Hallucinations To prevent errors introduced during automated fusion, AVSCap deploys a rigorous three-stage verification pipeline combining deterministic matching with semantic auditing. Stage 1 executes Audio Tag Preservation, using regex checks to ensure that every source audio tag ID appears exactly once and that category counts match the ground truth; instances with missing, duplicated, or misordered tags are discarded. Stage 2 performs a Tag-Event Consistency Check with Qwen3-32B to verify that each tagged sentence faithfully retains the semantic fidelity of the original acoustic description. Stage 3 conducts Local Synergy Verification with Qwen3-32B to confirm that each tag-anchored sentence explicitly contains both the audio event and its associated visual trigger bound by legitimate temporal conjunctions. Captions failing any stage are eliminated, curating the pristine AVSCap-130K training corpus from 40K raw video clips.

4. Hybrid-Reward GRPO Policy Optimization: Direct Optimization of Completeness and Synergy Relying solely on supervised fine-tuning often yields repetitive generations or superficial template memorization, while scaling SFT data offers diminishing returns. AVSCap implements Group Relative Policy Optimization (GRPO) on an optimization set of 2,000 non-overlapping videos using a tri-part hybrid reward \(R_{\text{total}} = R_{\text{len}} + R_{\text{sp}} + R_{\text{syn}}\). First, a length regularization reward \(R_{\text{len}} = \mathbb{I}(\tau_{\min} \le L_{\text{gen}} \le \tau_{\max})\) enforces generation within \([\tau_{\min}=200, \tau_{\max}=2048]\) to eliminate degenerate short captions and repetition loops. Second, a regex-anchored Longest Common Subsequence (LCS) speech recall reward \(R_{\text{sp}}\) provides fully deterministic grading of speech preservation without costly judge LLM calls: $\(R_{\text{sp}} = \frac{1}{M} \sum_{i=1}^{M} \frac{\text{LCS}(s_{\text{gen}}^i, s_{\text{ref}}^i)}{|s_{\text{ref}}^i|}\)$ where \(M\) denotes reference speech segments, and \(s_{\text{gen}}^i\) and \(s_{\text{ref}}^i\) represent punctuation-stripped generated and reference dialogues. Third, cross-modal synergy recall \(R_{\text{syn}} = \frac{|E_{\text{covered}}|}{|E_{\text{syn}}|}\) decomposes reference text into atomic synergy events and utilizes GPT-5 to verify whether the candidate caption captures the temporal binding. This targeted reward formulation enables AVSCap-7B to dramatically boost non-speech acoustic coverage and cross-modal binding with exceptional sample efficiency.

Benchmark & Evaluation Protocol

To address the inadequacy of traditional metrics in evaluating relational binding, the paper introduces AVSCapBench, comprising 1,226 human-annotated video clips (spanning durations from 30 to 120 seconds across YouTube, TikTok, and Video-MME). The benchmark adopts an atomic event-based matching protocol where GPT-5 breaks ground-truth captions into visual events \(E_{\text{visual}}\), audio events \(E_{\text{audio}}\) (Speech, SFX, Music), and synergistic events \(E_{\text{synergy}}\). During evaluation, Gemini-3.1-Pro assesses semantic recall for each category: $\(\text{Recall}_{\text{type}} = \frac{1}{|E_{\text{type}}|} \sum_{e_i \in E_{\text{type}}} h(e_i, \hat{C})\)$ where \(h(e_i, \hat{C}) \in \{0, 1\}\) reflects coverage of atomic event \(e_i\) in candidate caption \(\hat{C}\), with the final benchmark score computed as the event-count-weighted average across all three event types.

Key Experimental Results

Main Results

On AVSCapBench, AVSCap-7B achieved the highest overall score among all evaluated open-source models, outperforming open-source baselines by a wide margin in Synergy Recall and non-speech audio understanding (SFX and Music), approaching proprietary commercial systems.

Model Category Model Name Visual Audio: Speech Audio: Music Audio: SFX Audio Overall Synergy Total
Closed-Source Gemini-3-Pro 60.43 79.81 39.52 27.77 71.29 48.88 60.97
Closed-Source Gemini-3-Flash 58.14 79.78 39.46 32.34 72.65 48.94 60.54
Open-Source AVoCaDO-7B 50.59 70.42 38.71 19.25 61.07 29.13 49.31
Open-Source ASID-Captioner-7B 47.42 68.73 30.50 17.91 59.02 24.84 45.94
Open-Source Qwen3-Omni-30B-A3B 41.85 49.08 9.34 8.68 39.17 16.19 35.29
Open-Source video-SALMONN-2-7B 39.05 46.76 13.76 8.71 36.52 12.43 32.02
Open-Source UGC-VideoCaptioner-3B 33.24 21.30 22.00 11.48 20.77 10.43 24.24
Open-Source Qwen2.5-Omni-7B (Base) 34.78 13.92 4.02 7.22 13.71 7.00 21.53
Proposed AVSCap-7B (Ours) 59.33 69.45 40.36 30.82 64.30 57.70 60.44

Furthermore, evaluation on external benchmarks confirms robust cross-domain transferability on UGC-VideoCap and caption-guided QA on Daily-Omni:

Model UGC-VideoCap Aud. โ†‘ UGC-VideoCap Vis. โ†‘ UGC-VideoCap Det. โ†‘ UGC-VideoCap Avg. โ†‘ Daily-Omni QA Acc. โ†‘
Gemini-3-Pro (Commercial) 80.4 84.7 80.6 81.9 66.4
Qwen2.5-Omni-7B 46.9 58.4 60.0 57.7 17.0
video-SALMONN-2-7B 61.8 71.4 68.5 67.2 29.9
AVoCaDO-7B 73.0 84.8 71.8 73.2 55.5
ASID-Captioner-7B 79.1 84.4 80.2 81.2 61.2
AVSCap-7B (Ours) 82.9 81.1 83.2 82.4 66.6

Ablation Study

The paper validates the relative impact of GRPO reinforcement learning against supervised data scaling, as well as the necessity of non-speech audio data augmentation.

Configuration Samples Training Strategy AVSCapBench Total UGC-VideoCap Score Note
50% SFT Subset 65K SFT only (w/o RL) 56.38 80.34 Baseline half-data scale
100% SFT Dataset 130K SFT only (w/o RL) 57.49 (+1.11) 81.17 (+0.83) Doubling SFT data yields minor gains
100% SFT + GRPO 130K + 2K SFT + GRPO 60.44 (+2.95) 82.35 (+1.18) Adding 2K RL samples gives substantial boost
SFT w/o Audio Aug. 40K video pairs SFT only 58.27 (Music: 35.98, SFX: 26.35) - Weak non-speech acoustic coverage
SFT w/ Audio Aug. 40K video + 10K audio SFT only 59.42 (Music: 39.50, SFX: 29.80) - +3.52% Music and +3.45% SFX recall gain

Key Findings

  • Reinforcement learning is vastly more sample-efficient than scaling SFT: Scaling SFT data from 65K to 130K yields a modest 1.11-point gain on AVSCapBench, whereas applying GRPO with just 2,000 additional video samples produces a 2.95-point jump (driven primarily by a surge in Synergy Recall). Exploring generation trajectories under verifiable reward signals proves far more potent than passive teacher-forcing.
  • Open-source architectures universally suffer from temporal context degradation: Across all evaluated open-source models, caption recall drops precipitously as video duration expands from 30-50s to over 90s (e.g., Qwen2.5-Omni-7B degrades from 23.4% to 11.0%). Effectively managing long-context attention without losing cross-modal synchronization remains a structural barrier.
  • Disparity in negative instruction adherence under modality shielding: In modality suppression tests ("describe only audio and do not mention visual cues"), general foundation models like Qwen2.5-Omni reduce leakage from 72% down to 18% when explicit negative prompts are added. Conversely, task-specific captioners like AVoCaDO-7B remain stuck at ~100% leakage, demonstrating that their cross-modal separation is an artifact of rigid training distributions rather than controllable instruction compliance.

Highlights & Insights

  • Decoupled-then-fused data curation paradigm: Generating unimodal anchors with silent placeholders before injecting temporally bound audio events bypasses modality cross-contamination, establishing a reproducible blueprint for multimodal dataset construction.
  • Rule-based and semantic hybrid reward design: Blending regex-driven LCS string matching for dialogue recall with LLM-based atomic synergy verification balances reinforcement learning training throughput with semantic rigor.
  • Transition from n-gram overlap to atomic event recall: Demonstrating that classic BLEU and CIDEr metrics correlate poorly with human judgments on complex audio-visual interactions, establishing atomic event breakdown as the modern standard for video evaluation.

Limitations & Future Work

  • Performance attenuation on extended videos: AVSCap-7B experiences measurable degradation on clips longer than 90 seconds due to attention dispersion and context window limits, indicating the need for streaming memory or hierarchical summarization.
  • Monolingual and cultural scope: The corpus and benchmark are currently restricted to English dialogue, leaving multilingual accents, non-English speech, and culturally specific audio styles for future expansion.
  • Sub-second temporal synchronization limits: While the model successfully generates descriptive temporal conjunctions, achieving exact sub-second physical sound-action alignment remains challenging without higher frame rate feature processing.
  • vs AVoCaDO: While AVoCaDO also targets temporal orchestration, it relies on conventional SFT without dedicated non-speech acoustic augmentation; AVSCap-7B outperforms AVoCaDO by 28.57 points in Synergy Recall (57.70% vs 29.13%) and demonstrates markedly superior controllable modality shielding.
  • vs video-SALMONN-2: video-SALMONN-2 integrates audio-visual encoders but remains heavily skewed toward speech transcription; AVSCap demonstrates that augmenting 10K non-speech audio samples combined with GRPO policy optimization elevates SFX and music comprehension by multi-fold margins.

Rating

  • Novelty: โญโญโญโญโ˜† The decoupled-then-fused data engine and hybrid-reward GRPO effectively break the modality isolation bottleneck.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 13 models, 3 external benchmarks, extensive ablations, and human-verified hallucination audits.
  • Writing Quality: โญโญโญโญโญ Highly coherent technical narrative, well-formulated methodology, and clear mathematical definitions.
  • Value: โญโญโญโญโญ Sets a new open-source standard for omni-modal captioning, with AVSCapBench providing a foundational evaluation suite.