---

title: >- [Paper Note] Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning description: >- [ECCV 2026][video_understanding][Audiovisual Video Captioning] TCA-Captioner and TCA-Bench tackle modality detachment and temporal incoherence in audiovisual video captioning via an Observer-Checker-Corrector data pipeline and decoupled evaluation protocol. tags: - ECCV 2026 - video_understanding - Audiovisual Video Captioning - Cross-Modal Binding - Temporal Coherence - MLLM date: 2026-09-19 content_hash: 98eb4336ac617cde

Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

Conference: ECCV 2026
Paper: ECCV Official
Code: https://huggingface.co/TCA-Captioner-ECCV-2026
Area: Video Understanding
Keywords: audiovisual video captioning, cross-modal binding, temporal coherence, multimodal LLM, iterative data synthesis

TL;DR

To tackle modality detachment and temporal incoherence in audiovisual video captioning, this paper introduces TCA-Captioner, an omnimodal captioner trained via an iterative Observer-Checker-Corrector (OCC) data refinement pipeline on high-density human interaction videos, and proposes TCA-Bench with a Decoupled Evaluation Protocol to systematically isolate cross-modal binding and temporal causal reasoning.

Background & Motivation

With the rapid maturation of Multimodal Large Language Models (MLLMs), video captioning is transitioning from vision-centric perception toward omnimodal, physically grounded understanding. In everyday scenarios, auditory cues—ranging from speech and vocal intonation to transient physical impacts and environmental soundscapes—are essential for resolving ambiguities, inferring human intent, and establishing narrative causality. Although unified omnimodal architectures like Qwen3-Omni and Gemini-3-Pro along with specialized captioners like AVoCaDO and video-SALMONN 2 process joint audiovisual inputs, their generated narratives frequently suffer from acute "modality detachment."

The core tension stems from two pervasive synchronization failures. First, models struggle with Audio-Visual Cross-Modal Binding: attributing an acoustic event to its corresponding visual entity remains error-prone, especially in multi-person or noisy environments where speech is routinely misassigned across speakers or off-screen sounds are erroneously mapped to visible actors. Second, models lack Audio-Visual Temporal Coherence: instead of preserving chronological interactions (such as anticipating an action via an auditory cue or tracing a physical reaction to a sound), current models tend to describe visual dynamics and audio events in disconnected, sequential blocks, obscuring the unfolding logic of multimodal streams.

Existing benchmarks exacerbate this issue by relying on flat atomic event matching or holistic scoring, conflating low-level unimodal recognition errors with high-order cross-modal relational failures. This paper attacks the problem at both the training synthesis and diagnostic evaluation stages. Core idea: develop an Observer-Checker-Corrector (OCC) iterative data refinement framework combined with high-density human interaction curation to train TCA-Captioner, and establish TCA-Bench with structured binding and temporal relation ground truths for decoupled diagnostic evaluation.

Method

Overall Architecture

TCA-Captioner addresses joint audiovisual alignment through a two-pillar design: the Observer-Checker-Corrector (OCC) iterative data synthesis pipeline that produces meticulously grounded training annotations, and a post-training regime applying LoRA SFT and Direct Preference Optimization (DPO) to an omnimodal backbone. OCC replaces error-prone single-pass generation with an iterative audit loop that decouples unimodal verification from granular temporal alignment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Audiovisual Video Stream<br/>(Audio Track + Video Frames)"] --> B["Global-Perceptive Observer<br/>(Gemini-3-Pro generates raw high-density description)"]
    B --> C["Tripartite Forensic Checker<br/>(Independent Audio / Visual / 2s Clip Audits)"]
    C --> D["Iterative Refinement Corrector<br/>(Fuses violation logs to eliminate hallucinations)"]
    D --> E["High-Density Interaction Dataset<br/>(3,500 HDI + 12k augmented clips)"]
    E --> F["TCA-Captioner Training<br/>(Qwen3-Omni backbone + LoRA SFT & DPO)"]

Key Designs

1. Observer-Checker-Corrector (OCC) Framework: Decoupled Multi-Agent Forensic Refinement Generating dense, synchronized multimodal narratives in a single prompt inevitably conflates acoustic hallucinations with visual misattributions. The OCC framework splits this challenge into specialized roles: The Global-Perceptive Observer utilizes Gemini-3-Pro as a comprehensive sensor operating under seven strict mandates, enforcing exhaustive inventories of acoustic physical parameters, cinematographic movements, explicit entity grounding without ambiguous pronouns, and sub-second temporal timestamps ([01.2s - 01.5s]). Next, the Tripartite Forensic Checker deploys three independent sub-systems: an Audio Forensic Checker (Gemini-3-Pro) auditing the isolated audio track for verbal omissions and phantom acoustic artifacts; a Visual Forensic Checker leveraging Doubao-1.8 on silent frames to circumvent Gemini's high hallucination rate in silent video and audit micro-expressions and physical trajectories; and a specialized Audio-Visual Clip Checker (Gemini-3-Flash) running a 2-second sliding window to verify lip-sync alignment, diegetic versus non-diegetic sound placement, and cross-modal temporal precedence. Finally, the Iterative Refinement Corrector aggregates structured violation reports from all three checkers to rewrite the description, guaranteeing both semantic depth and physical synchronization.

2. High-Density Human Interaction Dataset: Grounding Complex Multi-Speaker Dynamics Standard video captioning datasets are dominated by sparse acoustic events that fail to train models on entangled cross-modal reasoning. To bridge this gap, the authors curate the High-Density Human Interaction (HDI) dataset, containing 3,500 challenging clips from television and cinema characterized by overlapping multi-party dialogue, rapid bodily movements accompanied by contact noises, and subtle cross-modal reactions. Every HDI clip undergoes full OCC processing including sliding-window clip verification (consuming ~21,949 tokens per 10-second video). To expand distributional variety, an additional 5,000 FineVideo and 7,000 TikTok-10M clips are processed with a streamlined OCC pipeline (bypassing the clip checker), creating a robust post-training corpus.

3. TCA-Bench & Decoupled Evaluation Protocol: Diagnostic Metric Separation TCA-Bench comprises 459 densely annotated clips (5–30s) selected via automated person-count (YOLOv8) and speech-activity (Silero VAD) co-occurrence analysis followed by rigorous human curation. To prevent unimodal acoustic or visual misses from obscuring relational capabilities, TCA-Bench formalizes evaluation into two tiers: Base Perception measures unimodal fidelity across five sub-dimensions on a 0–10 scale. Audiovisual Integration utilizes structured ground truths: - Binding Accuracy: evaluated against [sound, source] pairs (categorized into foreground character, off-screen character, and environment). The LLM judge only assesses source attribution after confirming the acoustic event is detected, preventing transcription errors from penalizing binding proficiency. - Temporal F1: evaluated on [first_event, second_event, relation] triples (spanning A→V, V→A, and A⇌V). Judgments use a three-way verdict (correct, incorrect/reversed, or skipped when an event is missing), isolating chronological ordering from event recall:

\[ \text{Prec} = \frac{|\text{correct}|}{|\text{correct}| + |\text{incorrect}|}, \quad F_1 = \frac{2 \cdot \text{Prec} \cdot \text{Cov}}{\text{Prec} + \text{Cov}} \]

Loss & Training

TCA-Captioner is trained on a Qwen3-Omni-30B backbone using a two-stage regime: First, LoRA Supervised Fine-Tuning (SFT) is executed over the combined corpus of 3,500 HDI and 12,000 supplementary clips to internalize sub-second timestamping and source attribution. Second, Direct Preference Optimization (DPO) is conducted on 2,000 paired examples derived from OCC correction logs, where the refined annotation serves as the preferred sample and the initial uncorrected draft serves as the dispreferred sample, penalizing modality detachment in representation space.

Key Experimental Results

Main Results

On standard public benchmarks evaluated using official protocols with GPT-4.1 / GPT-4o judges:

Model	Size	Modality	Video-SALMONN-2 Miss ↓	Video-SALMONN-2 Hall. ↓	Video-SALMONN-2 Total ↓	UGC-VideoCap Avg. ↑
Gemini-3.0-Pro	Proprietary	A + V	18.9%	14.8%	33.7%	75.3
Qwen3-Omni	30B-A3B	A + V	32.0%	13.6%	45.6%	71.5
video-SALMONN-2	7B	A + V	21.2%	17.6%	38.8%	67.2
AVoCaDO	7B	A + V	21.1%	16.2%	37.3%	73.2
TCA-Captioner (Ours)	30B-A3B	A + V	18.6%	12.9%	31.5%	74.2

On the proposed diagnostic benchmark TCA-Bench (normalized 0–100 scale, GPT-4.1 judge):

Model	Audio Avg. ↑	Visual Avg. ↑	AV Binding Char. ↑	AV Binding Env. ↑	AV Binding Total ↑	AV Temporal Seq. F1 ↑	AV Temporal Sim. F1 ↑	AV Temporal Total F1 ↑
Gemini-3.0-Pro	70.2	60.9	87.3	42.8	76.8	80.9	78.3	80.0
Qwen3-Omni (Base)	39.0	51.4	48.8	22.2	42.5	46.7	49.9	47.9
AVoCaDO	60.8	62.8	76.7	36.8	67.2	68.8	67.3	68.3
TCA-Captioner (Ours)	61.2	63.6	82.1	44.6	73.2	76.7	77.1	76.9

Ablation Study

Ablation of data generation components in the OCC pipeline (Table 6, evaluated on TCA-Bench core metrics):

Training Data Generation Config	Audio Score	Visual Score	AV Binding Total	AV Temporal Total F1
Observer Only	57.6	56.1	66.5	69.8
Observer + Audio Checker	61.5	57.0	68.2	68.1
Observer + Visual Checker	58.8	64.8	67.8	69.5
Full OCC Framework (+ A/V-Clip-Checker)	61.2	63.6	73.2	76.9

Furthermore, applying DPO (Table 5) on top of SFT Qwen3-Omni boosts AV Binding from 73.2 to 75.3 and AV Temporal F1 from 76.9 to 79.2, demonstrating the efficacy of learning from uncorrected/corrected OCC draft pairs.

Key Findings

• 2-Second Clip Auditing Drives Relational Gains: While unimodal checkers primarily enhance their respective domain scores (Visual Checker raises visual score from 56.1 to 64.8), AV Temporal F1 remains stagnant below 70 until the sliding-window A/V-Clip-Checker is integrated, driving Temporal F1 up to 76.9 (+7.4 points) and AV Binding to 73.2.
• Specialized Visual Checkers Outperform Generic Omni Models: Utilizing Doubao-1.8 for silent video verification eliminates hallucinations that even Gemini-3-Pro manifests, enabling TCA-Captioner to achieve a higher visual score (63.6) than Gemini-3.0-Pro (60.9).
• Environmental Sound Attribution Remains Challenging: Open-source base models demonstrate extreme weakness in environmental binding (Qwen3-Omni scores 22.2). TCA-Captioner improves this to 44.6 (surpassing Gemini-3.0-Pro's 42.8), though environmental attribution lags far behind character speech binding (82.1), revealing a key area for future improvement.

Highlights & Insights

• Forensic Decomposition Strategy: Physically separating the audio-only audit from silent-video scrutiny cuts off the cross-modal assumption bias where models "hear what they expect to see" or "see what they expect to hear."
• Noise-Resilient Decoupled Evaluation: By introducing the skipped status in temporal relation validation, TCA-Bench avoids penalizing event omission as a temporal sequencing fault, establishing an interpretable diagnostic tool.
• Zero-Annotation DPO Generation: The initial erroneous draft and the final corrected caption from the OCC pipeline provide natural negative/positive preference pairs, circumventing the need for expensive human preference labeling.

Limitations & Future Work

• High Token Overhead during OCC Synthesis: The 2-second sliding window in A/V-Clip-Checker accounts for over 39% of the ~22k tokens per 10-second clip, constraining massive-scale pre-training annotation.
• Off-Screen and Reverberant Source Ambiguity: Acoustic reflections and multi-speaker overlapping dialogue still cause ambiguities in environment sound attribution (44.6% accuracy).
• Future Directions: Exploring end-to-end self-supervised reward modeling to internalize clip checking directly into model representations, and extending fine-grained temporal alignment to sound-conditioned video generation.

Related Work & Insights

• vs AVoCaDO (Chen et al., 2025): AVoCaDO relies on temporal orchestration and heuristic prompting, which remains prone to speaker confusion in dense conversations; TCA-Captioner's OCC pipeline and HDI dataset yield superior binding (73.2 vs 67.2) and temporal synchronization (76.9 vs 68.3).
• vs video-SALMONN 2 (Tang et al., 2025): video-SALMONN 2 flattens videos into independent atomic events, missing the causal chain connecting auditory cues with visual reactions; TCA-Bench explicitly evaluates temporal dependency triples, closing this diagnostic gap.

Rating

• Novelty: ⭐⭐⭐⭐ [Innovative decoupled forensic data synthesis pipeline and structured temporal relational evaluation benchmark]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on multiple public benchmarks and self-curated diagnostic tests across open- and closed-source models with extensive ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous problem formulation, precise diagnostic taxonomy, and clear presentation]
• Value: ⭐⭐⭐⭐ [Sets a solid data synthesis framework and diagnostic benchmark for omnimodal video understanding]

Related Papers

• [AAAI 2026] Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction
• [CVPR 2025] Cross-modal Causal Relation Alignment for Video Question Grounding
• [ICLR 2026] AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration
• [CVPR 2026] Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminability
• [CVPR 2026] Progressive Cross-Modal Causal Intervention for Long-Term Action Recognition