Self-Supervised On-Policy Distillation for Omni-Modal Video Captioning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Video Understanding
Keywords: Omni-Modal Video Captioning, Self-Supervised Distillation, On-Policy Distillation, Dual-Stream QA Auditing, Hallucination Mitigation
TL;DR¶
SODA introduces a self-supervised on-policy distillation framework for omni-modal video captioning that exploits intrinsic task and input-budget asymmetries, auditing low-budget student drafts via dual-stream QA and distilling high-budget teacher revisions back into the student through token-level KL divergence without external teachers or human annotations.
Background & Motivation¶
Recent advances in Video Large Language Models (VideoLLMs) have driven video understanding toward comprehensive omni-modal perception, where open-ended video captioning serves as an immediate reflection of fine-grained audio-visual grounding. However, synthesizing high-fidelity, detailed captions across visual, acoustic, and cross-modal dimensions has traditionally relied on two prohibitively expensive resources: extensive human annotations and stronger external proprietary teacher models such as Gemini 2.5 Pro. Fine-grained video annotation is notoriously labor-intensive and frequently suffers from human perceptual bias, prioritizing prominent foreground visual events while routinely ignoring subtle background soundscapes, ambient acoustic scenes, and peripheral on-screen text. Meanwhile, conventional distillation or reinforcement learning pipelines (e.g., RLHF, GRPO) typically supervise models using sequence-level scalar rewards or terminal response tokens, offering sparse guidance for autoregressive token generation.
On-policy distillation (OPD) mitigates the train-inference exposure discrepancy by matching distributions on trajectories sampled from the student itself, yet existing multimodal OPD methods remain strictly tied to larger external teachers. This limitation raises an intriguing question: can an omni-modal VideoLLM self-improve its own captioning fidelity using structures already intrinsic to the model, without any external supervision? Investigating this possibility reveals two fundamental asymmetries within current VideoLLMs. The first is a task asymmetry: open-ended captioning forces the model to globally explore all perceptual dimensions and allocate visual-audio attention without explicit guidance, whereas targeted question answering (QA) narrows attention to a specific query. Empirically, holding the model and inputs fixed, directly querying the model with an audio-visual QA probe achieves 77.3% accuracy, whereas reading the answer off the model's own draft caption yields only 43.5%โwith the largest gap exceeding 30 to 50 points in audio and audio-visual joint facets. The second is an information asymmetry (or input-budget asymmetry): without altering any model parameters, providing richer multimodal contextโhigher resolution, higher frame rates, and extended audio temporal windowsโsubstantially elevates reasoning reliability.
These two asymmetries provide the natural architecture for a closed-loop self-distillation framework. Because targeted QA is vastly more reliable than unprompted captioning, question answering can serve as an internal auditor for draft captions; because higher input budgets unlock superior perception, the same model operating under expanded inputs can act as an effective teacher for its own low-budget student mode. The key insight is to operationalize this internal information advantage into dense, per-token supervision directly on the model's own support. Core idea: exploit the intrinsic task asymmetry (QA outperforming open captioning) and input-budget asymmetry (high-budget views outperforming low-budget views) within the same VideoLLM to build SODAโa self-supervised on-policy distillation framework where low-budget student drafts are audited by dual-stream QA, rewritten under high-budget inputs, and distilled back into the student via token-level KL divergence.
Method¶
Overall Architecture¶
SODA alternates between two core operational phases: Phase A (Self-Supervised Caption Revision) and Phase B (On-Policy Self-Distillation). In Phase A, the omni-modal backbone \(\mathcal{M}_\theta\) parameterized by \(\theta\) generates an initial draft caption \(c_s\) from a lightweight, low-budget video-audio view \(V_{lo}\). This draft is then audited by a dual-stream QA mechanism across 29 defined multimodal dimensions: a Caption QA Stream checks for factual errors and hallucinated commissions in \(c_s\), while a Video QA Stream probes the high-budget video-audio view \((V_{hi}, A_{hi})\) to recover omitted acoustic and visual dimensions. Multi-sample self-consistency voting filters out low-confidence queries, yielding a verified Revision QA Bank \(\mathcal{E}_{rqb}\). Conditioned on \((V_{hi}, A_{hi})\), \(c_s\), and \(\mathcal{E}_{rqb}\), the same model \(\mathcal{M}_\theta\) greedily decodes a high-fidelity revised target caption \(c_r\). In Phase B, both student and teacher passes are teacher-forced to \(c_r\), but the student only observes \(V_{lo}\) with a standard prompt while the teacher accesses high-budget inputs and revision context. Minimizing the token-level Kullback-Leibler (KL) divergence between their output logits transfers the informational lift into the student's weights, leaving deployment-time inference cost completely untouched.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Omni-Modal Video<br/>Low-Budget View V_lo"] --> B["Draft Generation & Dual-Stream Auditing<br/>Caption Stream verifies commissions + Video Stream recovers omissions"]
B --> C["Self-Consistency Voting & High-Budget Revision<br/>Multi-sample voting builds E_rqb and greedy decoding generates c_r"]
C --> D["Cross-Budget On-Policy Self-Distillation<br/>Per-token KL divergence regularizes student logits on c_r"]
D --> E["Deployment & Inference<br/>Fine-grained captioning under standard low-budget input"]
Key Designs¶
1. Dual-Stream Auditing: Complementary Commission Checking and Omission Recovery
Free-form multimodal captions exhibit two distinct failure modes: asserting unverified, hallucinated facts (commission) and remaining completely silent on salient modalities such as audio events, background music, or fine spatial textures (omission). A single QA stream cannot resolve both issues: questions derived solely from the caption cannot explore omitted dimensions, whereas questions derived solely from the raw video lack the targeted focus needed to challenge specific hallucinated claims in the draft. SODA establishes a structured taxonomy of \(D=29\) audio-visual dimensions spanning visual entities, camera kinematics, on-screen text, audio acoustics, and audio-visual synchronization, and deploys two complementary QA auditing streams:
- Caption QA Stream (Precision / Commission Check): For each dimension \(d \in \{1, \dots, D\}\), \(\mathcal{M}_\theta\) inspects the draft caption \(c_s\). If \(c_s\) makes no claim regarding dimension \(d\), the generator outputs SKIP_NOT_IN_CAPTION and appends \(d\) to a skip set \(\mathcal{S}_{skip}\). If \(c_s\) does make a claim, the model formulates an explicit verification question to probe whether the claimed content holds true against the audio-visual evidence.
- Video QA Stream (Recall / Omission Recovery): All skipped dimensions \(\mathcal{S}_{skip}\), together with dimensions whose answers in the Caption QA stream were rejected or weakly supported under a pivot threshold \(\tau_{piv}\), form an unaddressed pool \(\mathcal{U}\) (capped at \(D_{max}=8\) dimensions per clip). \(\mathcal{M}_\theta\) generates fresh questions directly grounded in the high-budget view \((V_{hi}, A_{hi})\), proactively unearthing acoustic details, text overlays, and fine background attributes that the initial caption silently overlooked. Surviving questions from both streams are merged to form candidate revision evidence.
2. Self-Consistency Voting & High-Budget Revision: Reliable Evidence Curation and Target Formulation
Because single open-ended autoregressive generations are susceptible to sampling noise and stochastic hallucinations, unverified model responses cannot serve as dependable distillation targets. SODA implements an internal self-consistency voting mechanism coupled with strict template-guided greedy revision. For each generated question \(q\), the teacher samples \(K=5\) independent answers conditioned on the high-budget view \((V_{hi}, A_{hi})\):
$\(a^{(k)} \sim \mathcal{M}_\theta(\cdot \mid V_{hi}, A_{hi}, q), \quad k = 1, \dots, K\)$
Before tallying votes, candidate answers are normalized by stripping trailing parenthetical rationales, lowercasing, collapsing whitespaces, and mapping hedging phrases (e.g., "not sure", "cannot tell", "unknown") into a unified uncertain token. The majority answer \(\hat{a}\) and its agreement ratio \(\rho \in [0, 1]\) are computed; the pair \((q, \hat{a})\) is admitted into the Revision QA Bank \(\mathcal{E}_{rqb} = \mathcal{E}_{cap}^{rqb} \cup \mathcal{E}_{vid}^{rqb}\) if and only if \(\rho \ge \tau\) (with default \(\tau=0.6\)) and \(\hat{a} \ne \text{uncertain}\). In the subsequent revision stage, \(c_s\) and \(\mathcal{E}_{rqb}\) are organized into a structured revision prompt \(\text{prompt}_r(c_s, \mathcal{E}_{rqb})\). The prompt instructs the model to treat verified answers as final ground truth, preserve uncontradicted draft spans, remove audio claims evaluated as none, and strictly avoid fabricating facts outside the evidence. Conditioned on \((V_{hi}, A_{hi})\) and this prompt, the model generates the target caption \(c_r\) via a single deterministic greedy decoding pass:
$\(c_r = \arg\max \mathcal{M}_\theta(\cdot \mid V_{hi}, A_{hi}, \text{prompt}_r(c_s, \mathcal{E}_{rqb}))\)$
This process triggers surgical revisions in 61.0% of training clips while maintaining a low character edit distance (3.2%), preventing drifting into an unstable foreign distribution.
3. Cross-Budget On-Policy Self-Distillation: Dense Token-Level Guidance on Model Support Once high-fidelity targets \(c_r\) are generated, they must be internalized into the student parameters without introducing test-time overhead or capacity mismatch. Unlike conventional sequence-level SFT or scalar reinforcement learning, SODA implements weight-shared on-policy self-distillation. For every cached tuple \((c_s, \mathcal{E}_{rqb}, c_r)\), two forward passes are executed, both teacher-forced on \(c_r\): $\(\ell_s = \mathcal{M}_\theta(V_{lo}, \text{prompt}_c; \text{forced}=c_r)\)$ $\(\ell_t = \mathcal{M}_\theta(V_{hi}, A_{hi}, \text{prompt}_r(c_s, \mathcal{E}_{rqb}); \text{forced}=c_r)\)$ where \(\ell_s, \ell_t \in \mathbb{R}^{|c_r| \times |\mathcal{V}|}\) represent the predicted logit distributions across the vocabulary \(\mathcal{V}\) at every token position of \(c_r\). The student branch receives only the minimal captioning prompt \(\text{prompt}_c\) and low-budget view \(V_{lo}\), while the teacher branch accesses \((V_{hi}, A_{hi})\) and the audited context. A stop-gradient operator \(\text{sg}(\cdot)\) is applied to the teacher logits, and the student parameters \(\theta\) are updated by minimizing the sequence-averaged per-token KL divergence: $\(\mathcal{L}_{\text{SODA}}(\theta) = \frac{1}{|c_r|} \sum_{j=1}^{|c_r|} D_{KL}\left(\text{softmax}(\ell_s^{(j)}) \parallel \text{sg}\left(\text{softmax}(\ell_t^{(j)})\right)\right)\)$ Because teacher and student share identical weights, \(c_r\) naturally lies within the model's own distribution support. The per-token KL objective acts not to force the model into an alien distribution, but to reassign probability mass within its existing capacity, compelling the student to produce detailed, audited descriptions directly from low-budget inputs.
A Worked Example¶
Consider an in-the-wild video showing a person's hand pointing at two leaves resting on soil:
1. Initial Draft: The student generates \(c_s\) from \(V_{lo}\), noting two leaves on soil, small wood fragments, a pointing finger, and the camera zooming in to show leaf "texture" and veins, concluding with natural lighting and no audio.
2. Dual-Stream Audit & Voting: In the Caption QA Stream, visual questions verify leaf texture as "smooth" (\(\rho=0.67\)), environment as "outdoor" (\(\rho=1.0\)), soil color as "brown" (\(\rho=0.67\)), and depth of field as "shallow" (\(\rho=0.67\)). In the Video QA Stream, probed audio dimensions confirm the absence of background music, dialogue, sound effects, and narration (\(\rho=0.67 \sim 1.0\), all answering none). The verified pairs populate \(\mathcal{E}_{rqb}\).
3. Teacher Revision: The teacher decodes \(c_r\) greedily using \((V_{hi}, A_{hi})\) and \(\mathcal{E}_{rqb}\). The generic phrase "showcasing their texture" is surgically updated to "showcasing their smooth texture", while all verified visual contexts and negative audio assertions are preserved.
4. On-Policy Distillation: During Phase B, teacher-forced forward passes align student logits to teacher logits over \(c_r\). Gradients heavily concentrate on newly injected informative tokens (e.g., "smooth"), enabling the student in subsequent inference to recognize fine tactile and acoustic features under low-budget inputs without requiring explicit QA queries.
Loss & Training¶
- Objective Function: The primary objective is the sequence-averaged token-level KL divergence in Eq. (6). Jensen-Shannon divergence (\(\beta=0.7\)) can be optionally utilized in early iterations for training stabilization.
- Model Architecture & LoRA: Built upon Qwen2.5-Omni-7B. The visual encoder and audio tower remain completely frozen; LoRA adapters with rank \(r=64\) and \(\alpha=128\) are inserted into the \(\{q, k, v, o\}_{proj}\) matrices of the language backbone.
- Input Budget Specification: Student view \(V_{lo}\) samples \(F_s \le 100\) frames at 1 fps with per-frame pixel budget \(R_s=151,200\) and mono audio window \(T_s=120\)s. Teacher view \((V_{hi}, A_{hi})\) samples \(F_t \le 100\) frames at 2 fps with pixel budget \(R_t=453,600\) (\(3\times\) pixels) and audio window \(T_t=240\)s (\(2\times\) duration).
- Training Schedule: Optimization uses AdamW with a constant learning rate of \(5 \times 10^{-5}\), linear warmup of 5%, per-device batch size of 2, and maximum sequence length of 32,768 tokens. Training spans 2 on-policy rounds over 2,000 unlabeled videos from Panda-70M. The teacher checkpoint is frozen at the start of each round to avoid optimization turbulence.
Key Experimental Results¶
Main Results¶
SODA is benchmarked against proprietary omni-modal foundation models and leading open-source models on Omni-Cloze (modality-specific cloze accuracy), VCapsBench-Omni (Accuracy Rate AR, Inconsistency Rate IR, and Coverage Rate CR), and UGC-VideoCap (caption quality sub-scores).
| Method | Omni-Cloze Visโ | Omni-Cloze Audโ | Omni-Cloze AVโ | Omni-Cloze Totโ | VCapsBench ARโ | VCapsBench IRโ | VCapsBench CRโ | UGC-Cap Audโ | UGC-Cap Visโ | UGC-Cap Detโ | UGC-Cap Avgโ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Proprietary Models | |||||||||||
| Gemini 2.0 Flash | 32.3 | 31.7 | 40.1 | 33.2 | - | - | - | - | - | - | - |
| Gemini 2.5 Flash | 24.4 | 36.2 | 41.4 | 33.0 | - | - | - | 74.2 | 78.8 | 77.2 | 76.7 |
| Gemini 2.5 Pro | 40.8 | 44.1 | 52.8 | 43.6 | 62.1 | 15.4 | 73.5 | 70.8 | 75.8 | 74.8 | 73.8 |
| Open-Source Models | |||||||||||
| video-SALMONN-13B | 3.5 | 2.3 | 4.6 | 3.3 | - | - | - | - | - | - | - |
| VideoLLaMA2-7B | 7.6 | 4.3 | 8.7 | 6.6 | - | - | - | - | - | - | - |
| Gemma-3n-E2B-it | - | - | - | - | 46.0 | 29.3 | 67.4 | 52.0 | 52.4 | 49.6 | 51.3 |
| Gemma-3n-E4B-it | - | - | - | - | 47.8 | 29.7 | 69.4 | 54.2 | 54.6 | 51.2 | 53.3 |
| Qwen2.5-Omni-3B | - | - | - | - | 37.6 | 28.5 | 52.6 | 48.2 | 55.6 | 52.6 | 52.2 |
| MiniCPM-o-2.6-8B | - | - | - | - | 30.6 | 29.7 | 41.7 | 46.0 | 59.7 | 62.4 | 56.3 |
| video-SALMONN 2+ | - | - | - | 28.3 | 37.0 | - | 47.2 | - | - | - | 68.1 |
| Qwen2.5-Omni-7B (base) | 18.3 | 14.1 | 21.9 | 16.6 | 44.5 | 29.7 | 60.7 | 57.6 | 59.4 | 57.0 | 58.0 |
| + SODA (Ours) | 27.1 | 21.9 | 24.8 | 25.0 | 47.7 | 30.3 | 68.4 | 73.1 | 57.2 | 73.3 | 67.8 |
Ablation Study¶
Ablation experiments isolate the contribution of each design component, contrasting inference token consumption against caption quality and general video QA accuracy on Video-MME v2.
| Setting | Omni-Cloze Visโ | Omni-Cloze Audโ | Omni-Cloze AVโ | Omni-Cloze Totโ | VCapsBench ARโ | VCapsBench IRโ | VCapsBench CRโ | Video-MME L1 | Video-MME L2 | Video-MME L3 | Inference Tokensโ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-Omni-7B (Base) | 18.3 | 14.1 | 21.9 | 16.6 | 44.5 | 29.7 | 60.7 | 32.6 | 17.8 | 19.1 | 830 |
| Revised Caption Only (\(c_r\)) | 27.3 | 13.0 | 22.3 | 21.8 | 45.8 | 26.6 | 62.4 | - | - | - | 6,936 |
| w/o Extra Vid/Aud Info | 26.5 | 20.0 | 24.4 | 23.9 | 46.8 | 29.1 | 67.8 | 31.3 | 17.2 | 19.1 | 830 |
| w/o Audit | 20.3 | 8.8 | 16.9 | 15.9 | 45.4 | 29.2 | 63.8 | 30.7 | 16.5 | 18.6 | 830 |
| EMA update | 26.7 | 22.2 | 25.0 | 24.9 | 47.6 | 30.1 | 68.2 | 31.3 | 18.6 | 19.0 | 830 |
| Joint update | 25.9 | 22.5 | 22.8 | 24.3 | 47.2 | 29.1 | 66.5 | 31.6 | 18.7 | 19.6 | 830 |
| Single-Stream (Caption-only) | 25.0 | 23.5 | 24.4 | 24.6 | 46.2 | 28.1 | 65.6 | 30.2 | 17.8 | 18.5 | 830 |
| SODA Round 1 | 26.7 | 19.2 | 22.3 | 23.6 | 47.6 | 29.4 | 67.8 | 32.0 | 17.4 | 19.0 | 830 |
| SODA Round 3 | 22.4 | 24.9 | 20.9 | 23.0 | 45.8 | 29.9 | 65.3 | 30.5 | 18.7 | 19.8 | 830 |
| SODA (Full Model) | 27.1 | 21.9 | 24.8 | 25.0 | 47.7 | 30.3 | 68.4 | 32.1 | 18.7 | 19.4 | 830 |
Key Findings¶
- Gains Concentrate Precisely Where the Audit Intervenes: The most substantial improvements over base Qwen2.5-Omni-7B appear along the audio and detail axes (Omni-Cloze Audio \(+7.8\), UGC-VideoCap Audio \(+15.5\), Detail \(+16.3\)). These match the exact dimensions where the Revision QA Bank most frequently overturns or supplements the draft, confirming targeted supervision.
- Recall, Rather Than Precision, Is the Core Bottleneck: On VCapsBench-Omni, coverage rate (CR) surges by \(+7.7\) points (from 60.7% to 68.4%), while accuracy rate (AR) increases modestly (\(+3.2\)) and inconsistency rate (IR) slightly drifts up by \(0.6\). This reflects the fact that omission is the dominant failure mode of baseline VideoLLMs; forcing coverage on unaddressed dimensions transforms silent omissions into explicit, falsifiable assertions.
- Phase B Internalizes Audited Evidence with \(\sim 8\times\) Token Compression: Generating revised captions at test time via Phase A prompting (\(c_r\) only) requires 6,936 input text tokens per video. Phase B distillation trains the student to natively output detailed captions under standard prompting (830 tokens), compressing inference computation by roughly 8-fold while outperforming test-time prompting.
- Benign Positive Transfer to General Video Understanding: Despite training solely on caption generation without seeing multiple-choice question tokens, SODA improves overall factual accuracy on VideoHallucer from 44.8% to 51.5% (hallucination slice improving from 58.1% to 62.4%) and slightly improves higher-tier reasoning on Video-MME v2 (Level 2: 17.8% \(\to\) 18.7%, Level 3: 19.1% \(\to\) 19.4%). Because teacher targets are generated by the model itself, optimization merely redistributes probability mass within the model's existing support without catastrophic forgetting.
Highlights & Insights¶
- Inverting Task Asymmetry as a Self-Supervised Oracle: By uncovering that a VideoLLM's direct QA accuracy (77.3%) vastly surpasses its ability to articulate answers within open captions (43.5%), SODA turns directional QA into an internal self-supervision signal without human labels or external reward models.
- Weight-Shared Cross-Budget Distillation: Demonstrates that an effective teacher does not require larger parameter capacity or deeper layers; providing an identical model with higher spatial-temporal-acoustic input budgets suffices to create a potent, well-calibrated information gap for knowledge transfer.
- Complementary Dual-Stream Auditing: Decoupling audit queries into caption-grounded checks (for precision/commission) and video-grounded pivots (for recall/omission) systematically resolves both hallucinated over-claims and silent perceptual omissions.
Limitations & Future Work¶
- Diminishing Returns in Self-Evolution: Performance gains plateau and show slight regression by Round 3 on Omni-Cloze (dropping from 25.0% to 23.0%), indicating that as the student improves, extractable confident revision signals saturate, risking reinforcement of residual internal biases.
- Acoustic-Temporal Granularity Limitations: Current omni-modal encoders remain unreliable at estimating exact acoustic event onset/offset boundaries and dense event counts, requiring the authors to exclude acoustic-temporal questions from the evaluation benchmark and audit scope.
- Future Directions: Exploring semi-supervised setups that inject minimal ground-truth metadata to break self-supervised saturation, and conducting scaling-law investigations across larger video corpuses and backbone model sizes.
Related Work & Insights¶
- vs. On-Policy Distillation (Agarwal et al., 2024; Ko et al., 2024): Traditional OPD aligns student generations against a substantially larger external teacher. SODA demonstrates that a weight-shared teacher equipped with higher input budgets and QA auditing eliminates the dependency on external teachers while maintaining on-policy benefits.
- vs. Self-Training & Bootstrapping (Zelikman et al., 2022; Wang et al., 2023): Existing self-training methods (STaR, ReST) typically perturb models along stochastic/noise axes and supervise on discrete final answers. SODA separates teacher and student along an information axis, providing dense token-level KL divergence over rewritten captions.
- vs. Multimodal Hallucination Mitigation (Leng et al., 2024; Yu et al., 2024): Prevailing hallucination mitigation relies on contrastive decoding or DPO/RLHF preference pairs over response tokens. SODA embeds calibrated assertiveness into the caption generation distribution itself during self-distillation, incurring zero decoding overhead.
Rating¶
- Novelty: โญโญโญโญโญ Elegant exploitation of intrinsic task and budget asymmetries to build a closed-loop self-distillation paradigm.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across three caption benchmarks, two general video understanding suites, per-modality breakdowns, and detailed ablations.
- Writing Quality: โญโญโญโญโญ Rigorous conceptual framing, clear mathematical formulations, and exhaustive empirical transparency.
- Value: โญโญโญโญโญ Provides a practical, highly scalable blueprint for enhancing omni-modal foundation models in low-resource and annotation-scarce environments.