DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Models¶

Conference: ECCV2026
Paper: ECCV Paper
Project: https://diadem-captioner.github.io/
Authors: Xinlong Chen, Weihong Lin, Jingyun Hua, Linli Yao, Yue Ding, Bozhou Li, Bohan Zeng, Yang Shi, Qiang Liu, Yuanxing Zhang, Pengfei Wan, Liang Wang
Affiliations: Institute of Automation, Chinese Academy of Sciences; School of Artificial Intelligence, University of Chinese Academy of Sciences; Kling Team, Kuaishou Technology; Peking University
Area: Multimodal VLM
Keywords: audiovisual captioning, speaker attribution, utterance transcription, adaptive matching, staged reinforcement learning

Identity check: the official record and cached paper have exactly the same title. The first page identifies Xinlong Chen's internship at the Kling Team and Qiang Liu as the corresponding author. This note follows the conference PDF and does not mistake an arXiv identifier in its references for this paper's identifier.

TL;DR¶

DiaDem combines dialogue-aware synthetic data and difficulty-partitioned GRPO on AVoCaDO-7B to make complete audiovisual captions more accurate about "who said what," achieving REF 65.9 and ASR 79.3 on DiaDemBench while retaining competitive general caption quality.

Background & Motivation¶

Audiovisual captioning requires more than listing visible objects and background sounds. Attributing an utterance to the wrong character can change the apparent intentions and causality of a scene. Existing models such as AVoCaDO and UGC-VideoCaptioner produce detailed captions, but completeness does not guarantee dialogue accuracy: fluent text can transcribe an utterance correctly while assigning it to the wrong speaker, or hide attribution errors behind a vague description such as "a man."

Audio-only speaker diarization usually produces speaker IDs and time intervals; adding automatic speech recognition supplies attributed utterances, but those IDs do not directly identify characters by visual appearance. Audiovisual captions must also account for cuts, off-screen voices, and multiple visible people, then integrate dialogue with actions, scenery, and sounds. The task therefore requires reliable speaker-utterance binding in the final natural-language caption, not merely an additional transcription module.

Evaluation introduces another obstacle: a model can render consecutive utterances from one speaker as either a single sentence or several sentences. Rigid one-to-one matching can incorrectly penalize this segmentation choice. Core Idea: establish reliable feedback through adaptively merged dialogue matching, then use complementary teachers and difficulty-partitioned reinforcement learning to teach complete audiovisual captions to bind speakers to their utterances accurately.

Method¶

Overall Architecture¶

The input is a video with its audio track, and the output remains a complete audiovisual caption rather than a standalone transcription table. DiaDem retains the 7B AVoCaDO base model; its main contributions concern data, dialogue evaluation, and post-training rather than a new audiovisual encoder.

The dialogue matching protocol first defines evaluation for DiaDemBench and subsequently supplies training rewards. A synthetic annotation pipeline provides supervised fine-tuning data. The resulting model then undergoes two-stage GRPO on manually annotated dialogues, without requiring these teachers to generate an answer at every inference call.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Videos and dialogue annotations"] --> B["Adaptive Dialogue Matching"]
    A --> C["Complementary Teacher Synthesis"]
    C --> D["Supervised fine-tuning of AVoCaDO"]
    B -->|Dialogue reward| E["Difficulty-Partitioned GRPO"]
    D --> E
    E --> F["Complete audiovisual captions"]

Key Designs¶

1. Adaptive Dialogue Matching: prevent segmentation choices from corrupting feedback

Gemini-2.5-Pro first extracts an ordered sequence of speaker-description and utterance-text pairs from a generated caption. The protocol matches utterances before judging speaker consistency, since an utterance that was not transcribed cannot receive meaningful attribution credit. Equation (1) uses normalized Levenshtein edit distance to measure utterance similarity; the following restores the damaged cached equation according to its textual definition:

\[ \operatorname{Sim}(u,v)=1-\frac{\operatorname{ed}(u,v)}{\max(|u|,|v|)}. \]

Dynamic programming traverses prefixes of the prediction and reference sequences. It can skip an utterance or match groups of consecutive utterances at the ends of both prefixes. Each group contains at most \(W\) utterances and must belong to the same speaker; its merged similarity must exceed threshold \(\gamma\) to form a valid match. The objective maximizes cumulative utterance similarity. It neither forces all adjacent same-speaker utterances to merge nor freely reorders dialogue.

After matching, accumulated similarity gives the utterance score \(S_{\mathrm{utterance}}\). For matched speaker descriptions, Gemini-2.5-Flash makes a binary consistency judgment conditioned on the video, producing the accumulated speaker score \(S_{\mathrm{speaker}}\). Text determines alignment before video determines identity consistency, separating transcription errors from attribution errors.

Equation (5) normalizes precision and recall by the adaptively merged prediction and reference sequence lengths \(M',N'\) and then computes F1. The following is an equivalent shorthand for nonempty sequences, not a new metric:

\[ \operatorname{F1}(S,M',N')=\frac{2S}{M'+N'},\qquad \mathrm{REF}=\operatorname{F1}(S_{\mathrm{speaker}},M',N'),\quad \mathrm{ASR}=\operatorname{F1}(S_{\mathrm{utterance}},M',N'). \]

Tables report these dimensionless scores on a 0-100 scale, with higher values better. ASR here is not conventional word error rate. REF is also not an independent recognition accuracy over all visible people, detached from transcription quality. The cached text refers implementation details for unmatched items and merged lengths to Supplement A.5; this note does not invent code-level boundary rules.

2. Complementary Teacher Synthesis: correct dialogue before integrating scene context

The authors' case analysis finds that Gemini-2.5-Pro better preserves utterance content, whereas Gemini-3-Pro better corrects speaker attribution when transcription quality is comparable. The former therefore produces initial dialogues and the latter repairs their speaker assignments, yielding 70K dialogue descriptions. This observation does not imply that Gemini-3-Pro scores higher on every dialogue metric in the uncontrolled main comparison.

In parallel, AVoCaDO generates complete audiovisual captions for the same videos, preserving actions, environments, and background sounds. Gemini-3-Pro integrates the corrected dialogues into these captions to create 70K dialogue-aware examples. Another 15K captions of non-dialogue videos bring the SFT dataset to 85K. The two inputs respectively support dialogue accuracy and scene completeness, reducing the risk of turning a captioner into a model that outputs only spoken lines.

3. Difficulty-Partitioned GRPO: sustain learning through informative reward differences

Reinforcement learning uses another 3K manually annotated dialogue samples. Its reward includes the sum of REF and ASR, together with the base model's checklist-based and length-regularized rewards. The dialogue term directly targets attribution and transcription, while the other terms help preserve audiovisual content and regulate output length. Numerical weights are not specified in the main text, so equal weighting of all reward components must not be assumed.

GRPO samples several responses for the same video and normalizes their rewards by the within-group mean and standard deviation to obtain relative advantages. A clipped policy objective and KL constraint then update the model. When every response succeeds easily or all responses fail, little reward variation remains to supply a useful within-group ranking signal. More training examples alone need not resolve this issue.

Training first removes overly simple examples whose repeated rollouts receive nearly identical rewards. Stage 1 uses the complete remaining dataset. Before Stage 2, examples are partitioned by average dialogue reward and the high-difficulty subset is doubled to emphasize challenging cases. The main text explicitly specifies doubling that subset, but leaves the overall mixture and thresholds to the uncached Supplement B.2; it does not justify describing Stage 2 as training exclusively on hard examples.

A Worked Example¶

In one example from Figure 2, the reference records a woman's consecutive speech as 1 utterance, while the prediction divides it into 2 utterances; both sequences then contain 1 reply from a man. One-to-one alignment can omit the second predicted fragment. Adaptive matching can merge the woman's 2 fragments into 1 unit, align it with the reference, and then match the man's reply.

Merging is optional. If the model assigns consecutive utterances to different people, it cannot merge them merely to raise text similarity. The other example in Figure 2 shows that forcibly merging adjacent reference utterances from the same speaker can destroy otherwise valid individual matches. Dynamic programming must therefore retain the option not to merge.

This rule also changes training feedback. Splitting a correct utterance into short fragments should not incur an artificial penalty, but attributing it to the wrong person should still reduce REF. The resulting reward better matches the task than a fixed segmentation convention.

Loss & Training¶

The complete order is AVoCaDO, SFT on 85K captions, GRPO Stage 1 on filtered dialogue data, and GRPO Stage 2 with the difficult subset doubled. The 70K and 15K counts refer to supervised captions, whereas 3K refers to manually annotated dialogues; they do not represent identical annotation costs.

The available cache contains the main paper and references, not the cited supplementary material. It establishes the training paradigm, data sizes, and reward components, but not numerical learning rates, epoch counts, group sizes, \(W\), \(\gamma\), or difficulty thresholds. These are not reconstructed reproduction settings in this note.

Key Experimental Results¶

Main Results¶

DiaDemBench contains 1,039 manually annotated video clips, each no longer than 20 seconds, covering single-shot and multi-shot scenes, different speaker counts, and overlapping speech. Gemini-2.5-Pro provides initial annotations for human correction; acceptance requires consensus among 3 annotators and verification by a senior supervisor. In Table 1, the Overlap subset is mutually exclusive with the speaker-count groups and must not be interpreted as an overlapping statistical grouping.

The following excerpt is from Table 1. REF/ASR are F1 scores on a 0-100 scale, both higher-is-better. The Human row averages 3 authors' performance and is not a model training result.

Model	Size	REF ↑	ASR ↑
Human	Not applicable	97.9	97.1
Gemini-2.5-Pro	Undisclosed	63.6	74.8
Gemini-3-Pro	Undisclosed	63.1	71.0
Qwen3-Omni-Captioner	30B-A3B	43.9	58.8
AVoCaDO	7B	38.7	51.7
DiaDem	7B	65.9	79.3

Relative to Gemini-2.5-Pro, DiaDem improves by 2.3 and 4.5 score points, not relative increases of 2.3% and 4.5%. However, its multi-shot overlapping-speech subset scores are only 42.9/56.1 REF/ASR, below Gemini-3-Pro at 57.2/65.1. Overall leadership does not imply dominance on every difficult subset.

Ablation Study¶

The following excerpt is from Table 3. The DiaDemBench columns retain the definitions above. SALMONN-2 Total is the reported aggregate error score, lower-is-better; UGC-VideoCap Avg. is an aggregate judge score, higher-is-better. These general benchmarks do not measure accuracy in the same sense as REF/ASR, so their magnitudes must not be compared across metrics. As specified in Table 2, the SALMONN-2 judge is replaced with GPT-4.1.

Config	REF ↑	ASR ↑	SALMONN-2 Total ↓	UGC-VideoCap Avg. ↑
AVoCaDO base model	38.7	51.7	37.3	73.2
Add SFT	59.3	74.0	42.0	74.8
Then add GRPO Stage 1	64.7	77.6	38.4	74.7
Full DiaDem	65.9	79.3	36.5	75.6
GRPO without staged training	64.5	78.2	37.5	74.8
GRPO without easy filtering	63.2	78.2	37.7	74.5
GRPO without dialogue reward	60.1	74.3	37.0	75.2
GRPO data converted to additional SFT	59.2	73.6	42.4	75.0

The final row is an additional supervised-training control without GRPO, not further fine-tuning after the complete DiaDem model. It fails to reproduce the RL gains, supporting the importance of rewards and training strategy, but not proving that every possible SFT recipe would fail to match them.

Table 4 further compares the original AVoCaDO dialogue reward with the enhanced reward under two-stage GRPO. Only Stage 2 is shown below to isolate the reward formulation.

Stage 2 reward configuration	REF ↑	ASR ↑	SALMONN-2 Total ↓	UGC-VideoCap Avg. ↑
Original AVoCaDO dialogue reward	63.7	77.2	37.1	74.9
Enhanced DiaDem dialogue reward	65.9	79.3	36.5	75.6

Key Findings¶

SFT raises REF from 38.7 to 59.3 but increases SALMONN-2 error from 37.3 to 42.0. Only the complete post-training recipe combines REF 65.9 with error 36.5; preservation of general capability cannot be attributed to SFT alone.
Removing the dialogue reward lowers REF to 60.1, a loss of 5.8 score points against the full model and the largest REF loss among the listed GRPO variants. Difficulty filtering and staged training also help, but their effects should not be conflated with the larger SFT improvement.
Replacing both parsing and judging with Qwen3-VL-32B in Table 5 yields cross-model Pearson correlations of 0.995/0.983 for REF/ASR. The 200-example human evaluation in Table 6 yields corresponding correlations of 0.994/0.982. High correlation supports ranking stability, not an absence of judge bias.

Highlights & Insights¶

Dialogue metrics serve evaluation and reward learning simultaneously. Correcting segmentation penalties therefore improves not only leaderboard interpretation but also the direction of policy learning; the transferable lesson is to remove output-format artifacts before rewarding semantic correctness.
Teacher roles reflect different strengths in transcription and attribution rather than delegating all annotations to one nominally strongest model. Retaining the base captioner's scene descriptions also prevents dialogue from overwhelming the supervision signal.
The difficulty curriculum is motivated by within-group reward variation in GRPO. More difficult data does not automatically produce more useful gradients, a consideration relevant to other multimodal post-training tasks that depend on sampled response differences.

Limitations & Future Work¶

The authors identify multiple speakers, multiple visible people, overlapping speech, small faces, and mismatches between off-screen voices and visible characters as persistent difficulties. Table 1 supports these limitations, so overall averages alone do not establish reliability.
Synthesis, parsing, and judging depend on commercial models. Alternative-judge and human-correlation studies reduce this concern but cannot eliminate model preferences, costs, or version sensitivity; this is a cautious interpretation of the evaluation dependencies.
Clips are limited to 20 seconds, leaving long-video identity memory and cross-scene consistency unverified. Long-term identity tracking and stronger speech separation are possible extensions, not components validated in this paper.
The main text does not contain the full supplementary training configuration, and text extraction damages some equations. Exact reproduction requires checking the PDF and supplement rather than treating this note's equivalent formula cleanup as an official implementation.

vs AVoCaDO: DiaDem retains its audiovisual captioning capability and 7B base while strengthening dialogue annotations, matching rewards, and post-training. The experiments show that general descriptive quality does not automatically imply correct speaker binding.
vs UGC-VideoCaptioner: That work emphasizes general UGC detail and multidimensional scoring, whereas DiaDem separates dialogue into transcription fidelity and speaker attribution for diagnosis. Their goals overlap, but they are not the same data recipe evaluated on an identical benchmark.
vs speaker diarization and speaker-attributed ASR: Traditional outputs center on speaker IDs, time intervals, and transcripts. DiaDem binds utterances to visually identifiable speaker descriptions within complete scene text; the former tasks may help address overlapping speech, but their metrics are not interchangeable.

Rating¶

Novelty: 4/5. Adaptive matching and dialogue-oriented post-training form a targeted combination, while the architecture remains inherited.
Experimental Thoroughness: 4/5. The study covers main comparisons, general captions, stage and reward ablations, alternative judges, and human agreement.
Writing Quality: 4/5. The problem and evidence are clear, although important implementation details depend on the supplement.
Value: 4/5. The work provides an actionable evaluation and training approach to "who said what," with substantial room left in difficult dialogue scenes.