Skip to content

Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-visual Language Models

Conference: ECCV 2026
Paper: ECCV 2026 · Project Page
Area: Alignment & RLHF
Keywords: audio-visual language model, cross-modal hallucination, DPO, visual dominance, audio attribution

TL;DR

For the cross-modal hallucination in which audio-visual language models (AVLMs) invent sounds from what they see, this paper proposes ACPO: preference pairs built along two orthogonal axes — an output-contrastive axis that penalizes visual captions offered as answers to audio-focused prompts under an audio-swapped input, and an input-contrastive axis that requires an answer to become untenable once its supporting audio track is replaced — which, while fine-tuning only the audio projection layer, lifts AVHBench audio-hallucination F1 from 79.0 to 80.4 and unimodal audio-captioning CIDEr from 33.0 to 43.6.

Background & Motivation

Audio-visual language models (AVLMs) extend vision-language models with an audio stream, letting video, sound, and text enter an LLM together for question answering and captioning, and holding promise for applications such as autonomous systems and assistive technology that must interpret their surroundings reliably. Their reliability, however, is bottlenecked by a failure mode of their own: cross-modal hallucination. Unlike unimodal errors, these claims are plausible under one modality yet unsupported by the full context. The canonical instance is video-driven audio hallucination: a police car drives past, the audio track contains only birdsong, and the model confidently reports a blaring siren. Such answers read fluently and match real-world co-occurrence statistics perfectly; what they expose is a failure of modality attribution — the model treats what it saw as what it heard.

Worse, the failure is asymmetric. Prior work has observed that audio tokens receive disproportionately low attention during decoding, and that models hallucinate far more on audio-focused tasks than on visual ones. The paper first pins this down with a clean analysis: on the AVHBench video-driven audio hallucination task (V→A), going from 0 frames (audio only) at 76.8 to 4 frames at 76.4 and 10 frames at 74.0 means more visual input makes audio question answering strictly worse; conversely, on the audio-driven video hallucination task (A→V), adding audio on top of no-audio lifts accuracy from 61.5 to 77.2. The problem is therefore not mutual interference between streams but a one-sided suppression of auditory evidence by visual priors. The authors trace it to two compounding causes: the strong sight-sound correlation in large-scale audio-visual data, which lets models take co-occurrence shortcuts and never learn true modality attribution; and an architectural imbalance, since vision encoders enjoy far more data and stronger supervision than audio encoders.

Given this, simply applying preference learning to static audio-visual inputs cannot work: because audio and video are highly correlated, the model can "guess" the preferred audio caption from the visual content alone, and the preference signal is defeated by the very shortcut it was meant to remove. Core idea: construct preference pairs along two orthogonal axes — an output-contrastive axis that uses audio-swapped inputs to explicitly penalize visual descriptions given in response to audio queries, and an input-contrastive axis that swaps audio tracks and penalizes predictions invariant to the true auditory evidence — optimize the union of both, and fine-tune only the audio projection layer so the backbone's existing vision-language ability is left intact.

Method

Overall Architecture

ACPO asks how an AVLM can stay faithful to the audio track when the visual prior is highly tempting. Its answer recasts alignment as largely a data-construction problem: from existing audio-visual clips it first produces modality-separable supervision targets and controllably mismatched inputs, then assembles these along two axes into preference pairs, and finally trains a standard DPO objective over the union of pairs while touching only the small audio projection layer.

The pipeline is serial with a dual branch: an incoming VALOR clip is decomposed into modality-specific captions and simultaneously paired with a foreign audio track to form an "audio-swapped input"; these two products feed two families of preference pairs (the output-contrastive audio-attribution pairs and the input-contrastive audio-sensitivity pairs); both families converge into one DPO optimization. At inference the resulting model needs no extra computation and no change to the decoding procedure.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["VALOR clip<br/>video + audio + joint caption"] --> B["Modality-decoupled data curation<br/>caption split into y_vis / y_aud<br/>audio swap + similarity tiers"]
    B --> C["Audio-attribution pairs (output-contrastive)<br/>prefer what is heard, penalize what is seen"]
    B --> D["Audio-sensitivity pairs (input-contrastive)<br/>swap the track, the answer must change"]
    C --> E["Joint DPO optimization<br/>audio projection layer only"]
    D --> E
    E --> F["Inference: no extra cost<br/>decoding unchanged"]

Key Designs

1. Modality-decoupled data curation: split "what is seen" and "what is heard" into two comparable supervision targets

Standard audio-visual captions entangle visual and auditory evidence in one sentence ("a police car drives by while birds are chirping"), and such a joint caption provides no gradient signal for modality-level grounding — a model that writes only the first half is never penalized. ACPO's first step uses an LLM (GPT-5 in the paper) to decompose each joint caption into a visual caption \(y_{\text{vis}}\) describing only visually verifiable content and an audio caption \(y_{\text{aud}}\) describing only audibly verifiable content; these modality captions are the supervision targets for every preference pair below.

The second step builds "audio-swapped inputs": keep the original video \(v_A\), replace the audio track with that of a partner clip B, yielding \((v_A, a_B)\). To make the degree of mismatch controllable, the authors measure similarity between the original video and candidate audio tracks with a multimodal embedding model (ImageBind) and split swaps into high- and low-similarity tiers by empirical quantiles: low-similarity swaps create stronger audio-visual conflict, while high-similarity swaps are subtler mismatches. This step is the foundation of the whole method — without it, preference pairs could only be built on original clips, where sight and sound already agree, and the model can hit the preferred answer through visual shortcuts. The ablation row with no audio swap scoring lowest confirms exactly this.

2. Audio-attribution pairs (output-contrastive): penalizing visual descriptions given in response to audio questions

This family targets the failure mode the paper opens with: asked what is heard, the model answers what is seen. A pair takes the swapped input \((v_A, a_B)\) together with an audio-focused instruction \(x_{\text{aud}}\) (e.g., "Describe what you hear."); the preferred response is the audio caption \(y^{B}_{\text{aud}}\) belonging to the current track \(a_B\), and the dispreferred response is the visual caption \(y^{A}_{\text{vis}}\) belonging to the video \(v_A\):

\[p_\theta\big(y^{B}_{\text{aud}} \mid v_A, a_B, x_{\text{aud}}\big) > p_\theta\big(y^{A}_{\text{vis}} \mid v_A, a_B, x_{\text{aud}}\big)\]

Notably, the penalized caption may itself be fluent and perfectly consistent with the video; its only fault is that its source is vision rather than audition. The pair turns "attribution error" into a directly optimizable contrast: to keep the preferred answer likely, the model must actually read \(a_B\) instead of looking at \(v_A\). Empirically this family mainly raises precision (it stops confirming sounds that are not there); removing it in the ablation drops precision from 78.2 to 76.2.

3. Audio-sensitivity pairs (input-contrastive): penalizing answers that survive an audio swap

Attribution pairs police the source of an answer, but a model insensitive to the track itself — one that says the same thing whatever it hears — can still slip through. The second family closes this hole from the input side: fix the video \(v_A\), take the aligned input \((v_A, a_A)\) and a swapped input \((v_A, a_B)\), and let the same original joint caption \(y^{A}_{\text{av}}\) be preferred under the former and penalized under the latter:

\[p_\theta\big(y^{A}_{\text{av}} \mid v_A, a_A, x\big) > p_\theta\big(y^{A}_{\text{av}} \mid v_A, a_B, x\big)\]

The rationale is direct: \(a_B\) does not support the auditory details in \(y^{A}_{\text{av}}\), so if the model still assigns that caption a comparably high likelihood after the track is replaced, its prediction is unrelated to the actual auditory evidence. This family does not specify what to say, only that the answer must change with what is heard, so it mainly improves recall (82.8; removing it drops recall to 81.0) and shows the opposite similarity preference from attribution pairs: high-similarity (subtler) swaps work better, because they form harder negatives that expose audio-invariant behavior.

A Worked Example

Take the scenario of Figure 1. Clip A shows a police car driving by, its audio track actually contains only birdsong, and the joint caption is "a police car drives by while birds are chirping." Decomposition gives \(y^{A}_{\text{vis}}\) = "a police car drives by" and \(y^{A}_{\text{aud}}\) = "birds are chirping." A foreign track B (low-similarity tier, containing a siren) supplies \(y^{B}_{\text{aud}}\) = "a siren is sounding," producing the swapped input \((v_A, a_B)\).

  • Attribution pair: input \((v_A, a_B)\) with "Describe what you hear.", preferring "a siren is sounding" and penalizing "a police car drives by." A model that lazily answers from the frame is explicitly punished here — this is precisely the shortcut that makes the base model answer "a police car drives by with its siren blaring."
  • Sensitivity pair: under the aligned input \((v_A, a_A)\) (police car + birdsong) the caption "a police car drives by while birds are chirping" is preferred, while under the swapped input \((v_A, a_B)\) (police car + siren) the very same caption is penalized. The model cannot lean on "there is a police car in the frame" to justify that sentence.

Both families, mixed with OmniDPO's noise-based and text pairs, enter a single DPO objective; the trained model answers the same police-car clip with "a police car drives by, while birds are heard chirping."

Loss & Training

The training objective is the standard DPO loss applied to the union of all pair types: with a frozen reference model as the anchor, it raises the relative likelihood of the preferred response and lowers that of the dispreferred one, with \(\beta\) controlling the deviation penalty.

\[\mathcal{L}_{\text{DPO}} = -\log\sigma\Big(\beta\log\tfrac{p_\theta(y^+\mid c)}{p_{\text{ref}}(y^+\mid c)} - \beta\log\tfrac{p_\theta(y^-\mid c)}{p_{\text{ref}}(y^-\mid c)}\Big)\]

(⚠️ This equation is garbled in the cached PDF extraction; it is written here in the standard DPO form, matching the paper's notation — refer to the original paper.) The context \(c\) is no longer a fixed joint audio-visual context but varies with the pair type among \((v_A, a_B, x_{\text{aud}})\), \((v_A, a_A, x)\), and \((v_A, a_B, x)\) — this is the key difference from applying DPO directly to static audio-visual inputs.

Training data comes from 5,000 VALOR clips with joint captions; captions are decomposed with GPT-5 and audio swaps are built as described in the Method. The authors combine their audio-contrastive pairs with OmniDPO's noise-based pairs and text pairs, sampling randomly within each batch at a 60/40 ratio of audio-contrastive to other multimodal pairs. On the Video-LLaMA2 backbone, the video encoder, audio encoder, and LLM are frozen and only the audio projection layer is fine-tuned; AdamW with a learning rate of 2e-5, cosine scheduling with warmup, DPO \(\beta = 0.1\), global batch size 8, one epoch, roughly 3 hours on a single NVIDIA GH200 120GB. Training beyond one epoch yields no additional gain. For video-SALMONN-2 and Qwen2.5-Omni, low-rank adaptation (LoRA) is additionally applied to the LLM alongside the audio projection module, while the ACPO objective and the same 5,000 preference pairs are kept unchanged.

Key Experimental Results

Main Results

Evaluation covers two public benchmarks. AVHBench probes cross-modal hallucination with yes/no questions whose ground truth depends on a single modality while the other may be misleading; the paper reports its video-driven audio (V→A) hallucination task. CMM decomposes hallucination into unimodal dominance and spurious inter-modality correlations; the paper reports the Audio-Language and Overreliance-on-Vision subcategories plus the overall score (vision-language tasks are out of scope, since only the audio projector is modified). All controlled baselines are reimplemented on the same Video-LLaMA2 backbone under the same training paradigm.

Method AVHBench Audio Halluc. Prec↑ Rec↑ F1↑ Acc↑ CMM Overrely Vision Acc↑ CMM Overall Acc↑
Gemini-Flash-1.5 † 57.9 94.7 71.9 63.0 57.8 76.3
Qwen2.5-Omni † 60.8 98.8 75.3 67.6 75.8 81.0
MiniCPM-o † 70.4 78.6 74.4 72.8 73.8 76.0
Base (Video-LLaMA2-7B-AV) 68.6 88.5 77.3 74.0 73.3 82.5
SFT 72.8 83.5 77.8 76.2 76.0 82.5
DPO 70.6 88.9 78.7 76.0 77.3 82.9
OmniDPO 76.6 81.6 79.0 78.4 77.3 82.4
ACPO (Ours) 78.2 82.8 80.4 79.9 81.3 83.4

† These three models are zero-shot reference points reported in the cited work, not reimplementations under the same training paradigm; they situate the numbers but are not directly comparable to the controlled baselines above.

The authors additionally build a unimodal captioning evaluation to measure modality grounding in free-form generation: 400 clips with diverse, distinct audio events are filtered from AVHBench's audio-visual captioning task, and 400 audio-swapped counterparts are constructed. Ground-truth audio captions are generated by prompting an AVLM with the original caption plus the raw audio, and vision-only ground truth with the original caption plus raw frames (a random 10% was manually verified, 95% correct in both factual content and modality attribution). At evaluation time all models still receive both video and audio, mirroring the cross-modal hallucination setting: the model must describe what it hears despite potentially misleading visual input.

Method Audio (Original) Audio (Swap) Video (Original) Video (Swap) Average
Base 18.8/27.9 15.0/13.1 22.7/28.5 23.2/29.7 19.9/24.8
SFT 22.4/31.3 17.3/16.9 19.9/33.6 20.9/40.4 20.1/30.5
DPO 22.4/27.6 18.0/16.5 22.9/23.3 23.2/27.1 21.6/23.6
OmniDPO 23.1/33.0 18.9/18.9 21.9/25.1 21.4/26.1 21.3/25.8
ACPO (Ours) 27.0/43.6 23.3/30.6 20.4/27.6 20.2/29.7 22.7/32.9

Cells are METEOR/CIDEr (×100), higher is better; each column uses modality-specific references, so the metrics capture fidelity to the intended modality.

Ablation Study

Config Acc↑ Prec↑ Rec↑ Note
Full ACPO (both pair types) 79.9 78.2 82.8 Best precision/recall balance, best F1 at 80.4
w/o attribution pairs 79.1 76.2 84.6 Precision drops markedly; the model more readily confirms sounds that are absent
w/o sensitivity pairs 79.3 78.3 81.0 Recall drops; attention to the true audio track weakens
Attribution only (low-sim swap) 77.7 76.0 81.1 Best single-type attribution configuration
Attribution only (high-sim swap) 77.0 74.9 81.1 Subtler mismatches give a weaker signal
Attribution only (no swap) 76.9 77.6 75.7 Lowest overall: sight and sound stay aligned, so visual shortcuts defeat the training signal
Sensitivity only (low-sim swap) † 79.4 78.9 80.1 Video hallucination falls below the base model — overcorrection toward audio
Sensitivity only (high-sim swap) † 79.8 78.9 81.3 Best single type on audio hallucination, yet also overcorrects toward audio

† Configurations where video hallucination task accuracy fell below the base model, indicating overcorrection toward audio reliance.

Generalization across backbones (same objective and same 5,000 preference pairs, only the parameter-efficient fine-tuning scope is adapted):

Backbone Audio Halluc. Acc↑ / F1↑ Overall Acc↑ / F1↑
video-SALMONN-2 63.2 / 54.0 → 68.8 / 71.6 71.6 / 67.0 → 75.0 / 76.5
Qwen2.5-Omni 66.7 / 74.8 → 69.3 / 76.2 68.4 / 74.8 → 71.0 / 76.1

Key Findings

  • The two pair families are complementary rather than redundant. Attribution pairs carry precision and sensitivity pairs carry recall: removing attribution drops precision to 76.2 (−2.0), removing sensitivity drops recall to 81.0 (−1.8), and only the union reaches the best F1 of 80.4. This matches the two failure modes they target — answering from the wrong source, and not listening at all.
  • The audio swap is the precondition for attribution pairs to work. With no swap, accuracy falls from 77.7 to 76.9 and F1 from 78.4 to 76.6, the worst of all single-type configurations: when audio and video stay aligned, the model reaches the preferred answer through visual co-occurrence and the preference signal degenerates into a useless visual shortcut.
  • The similarity direction is opposite for the two families. Attribution pairs want low-similarity swaps (strong conflict, 77.7/78.4 versus 77.0/77.9 for high similarity), sensitivity pairs want high-similarity swaps (harder negatives, 79.8/80.1 versus 79.4/79.5 for low similarity). Sharing one swap tier across both families would cost each of them part of its gain.
  • Sensitivity pairs alone overcorrect. Both sensitivity-only configurations fall below the base model on video hallucination (marked †): pushing hard on "punish audio insensitivity" alone tips the model to the opposite extreme. The full ACPO objective avoids this trade-off.
  • On captions the trade is visual bias for auditory grounding, and it pays off. Audio CIDEr rises from 27.9 to 43.6 on original clips and from 13.1 to 30.6 on swapped clips — the largest single gain in the paper — while video-side scores dip slightly (e.g., video original CIDEr 28.5 → 27.6). The authors' reading is that the base model is heavily visually dominant and part of its high video-captioning score comes from producing purely visual captions that trivially match video-only references; ACPO trades a little of that bias for real audio grounding, and its average across conditions on both METEOR and CIDEr (22.7/32.9) is the best of all methods.
  • The gains hold across three distinct backbones, indicating this is a general failure mode of audio-visual grounding rather than a single-backbone artifact.
  • Inference is free of extra cost: unlike contrastive-decoding approaches that adjust logits at inference time, ACPO changes parameters, so the inference procedure is exactly as before.

Highlights & Insights

  • It rewrites "attribution error" as contrastable preference pairs instead of teaching the model what to say. The sensitivity pair only requires that the same sentence stop holding once the audio is swapped; it needs no extra annotation of which auditory content would be correct, yet hits the root cause — predictions unrelated to what is heard. This "use invariance to construct negatives" recipe transfers to any multimodal attribution problem, e.g., in temporal video, "does the answer change if the future frames are swapped."
  • The penalized response is fluent and reasonable, and that is the sharpest move in the paper. In an attribution pair, \(y^{A}_{\text{vis}}\) is not wrong; it merely answers in the wrong modality. Explicitly treating such plausible-but-mis-sourced samples as negatives is closer to the real failure mode than noise- or corruption-based negatives (as in OmniDPO), whose perturbed inputs remain broadly aligned with the original video and therefore never create genuine cross-modal conflict.
  • Swap tiers are stratified by similarity, and the two pair families use different tiers. This is an easy-to-miss detail that the ablation validates: comparison difficulty should match the behavior a pair is meant to shape, and "strong conflict" and "hard negative" are two different data requirements that should not share one sampling policy.
  • Only the audio projection layer is tuned. With 5,000 preference pairs it takes one GPU and roughly 3 hours, and it barely perturbs established vision-language ability. For gap-closing alignment, restricting trainable parameters to the bottleneck module (here the projection layer that directly mediates audio-language alignment) is cheaper than full fine-tuning and less prone to catastrophic forgetting.
  • The self-built unimodal captioning evaluation is a reusable protocol. Having a strong AVLM generate modality ground truth from "original caption + one raw modality" and then scoring with modality-specific references sidesteps the vague audio annotations of existing datasets (e.g., "someone is speaking"). This "modality-specific reference + swapped input" design can be borrowed directly for evaluating multimodal grounding.

Limitations & Future Work

  • Visual detail is sacrificed. The authors' first failure case is a person opening a package with a crinkling sound: the base model, SFT, and DPO describe only the frame and ignore the audio entirely, OmniDPO captures part of the sound but drops visual details, and while ACPO identifies both, it generalizes the visual content (a package with a sandwich becomes "crumpling plastic"). That is, on joint captioning ACPO prioritizes auditory cues and approximates away visual content — a real cost where precise visual description matters.
  • Brief, subtle audio events are still missed. In the second failure case, a faint coin clinking accompanies speech, and every method including ACPO misses it. Better modality attribution is not the same as better perceptual sensitivity to audio events, which points to explicit event-level supervision or finer-grained audio token modeling.
  • Training depends on LLM-decomposed modality captions, whose quality is not evaluated on its own. The cleanliness of \(y_{\text{vis}}\) / \(y_{\text{aud}}\) bounds what the preference pairs can achieve; the paper reports a 10% manual check of the ground-truth captions for its evaluation set (95% correct) but not the decomposition quality of the 5,000 training captions. Multi-model voting or manual filtering of noisy decompositions is a natural improvement.
  • Swap plausibility is controlled by similarity quantiles, with limited coverage. Swapped tracks are drawn from the same data distribution, so outside the training distribution (rare instruments, non-speech vocalizations) low-similarity swaps may create physically impossible audio-visual combinations, and the model could learn the coarse rule "trust audio" rather than fine-grained attribution. Diversity of the swap pool deserves its own ablation.
  • Only the audio projection layer is modified. That is both the efficiency win and the ceiling: if part of the visual dominance originates in the LLM's internal attention allocation (prior work observes low attention weights on audio tokens), training the projector alone may not fully correct it. Combining ACPO's objective with attention-level intervention is a direct follow-up.
  • vs OmniDPO: the closest prior work adds noise to the audio or video stream and trains the model to prefer responses conditioned on the clean input. Noisy inputs remain broadly aligned with the original video, so it never directly creates cross-modal conflict and does little for video-driven audio hallucination specifically; ACPO instead substitutes another clip's real audio track, making the two modalities genuinely conflict and forcing an explicit judgment about which to trust. ACPO is better on audio hallucination F1 (80.4 vs 79.0) and on CMM Overreliance on Vision (81.3 vs 77.3), and it mixes OmniDPO's noise and text pairs into its training at 40%, treating them as a complement rather than a replacement.
  • vs DPO on audio-visual pairs directly: naive DPO builds preference pairs on a fixed joint audio-visual context, but real audio and video are highly correlated, so the model can earn the "preferred" audio caption from visual co-occurrence alone and the alignment objective is defeated. ACPO's data curation (modality decomposition plus audio swapping) exists precisely to make the pairs unsatisfiable by visual shortcuts; the lowest-scoring "no swap" attribution ablation is the most direct evidence.
  • vs AVCD (audio-visual contrastive decoding): AVCD masks modalities and adjusts logits at inference time, leaving underlying representations unchanged and adding inference cost; ACPO is a training-side method that changes the audio projection layer, adds no inference overhead, and avoids paying that cost repeatedly across generation rounds.
  • vs V-DPO: V-DPO constructs visual-hallucination preference pairs with out-of-distribution objects (e.g., cutting a rock instead of a cake) — the same constructive idea — but handles only a single non-text modality and ignores interactions between multiple input streams. ACPO generalizes "construct a conflicting input" to the space between modalities, which is exactly what distinguishes cross-modal from unimodal hallucination.

Rating

  • Novelty: ⭐⭐⭐⭐ The dual-axis (output-contrastive / input-contrastive) preference pair is a clean and well-targeted formulation that turns the abstract problem of modality attribution into constructible data; a real advance over contrastive decoding and noise-based DPO, though still within the mature DPO-plus-synthetic-pairs paradigm.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Two public benchmarks, a self-built unimodal captioning evaluation, generalization to three backbones, and paired design ablations (pair type, swap similarity) make the coverage quite complete; the decomposition quality of modality captions and the diversity of the swap pool are not evaluated separately, and the audio evaluation set is small (400 clips).
  • Writing Quality: ⭐⭐⭐⭐ The motivation pins the problem down cleanly with the asymmetry evidence (more frames hurts, more audio does not), and the dual-axis design maps one-to-one onto the ablations; the limitations section candidly gives two counterexamples. The equations are garbled in the cached PDF extraction, which slightly hampers reproduction.
  • Value: ⭐⭐⭐⭐ Training one projection layer for about 3 hours on a single GPU substantially reduces audio hallucination and still works across backbones, so the deployment cost is low; the self-built unimodal evaluation protocol is reusable for the whole AVLM hallucination line of research.