Direct Preference Optimization for Perceptual Alignment via Vision-Language Consistency¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Dialogue Systems
Keywords: Large Vision Language Models, Direct Preference Optimization, Vision-Language Consistency, Hallucination Mitigation, Cross-Modal Alignment

TL;DR¶

VLC-DPO is an unsupervised multimodal alignment framework that constructs preference pairs using complementary image- and description-based responses guided by a joint vision-language consistency score, incorporating consistency discrepancies as adaptive penalties in DPO to effectively suppress hallucinations and advance zero-shot VQA performance.

Background & Motivation¶

Large Vision Language Models (LVLMs) have achieved substantial progress in visual question answering, complex reasoning, and multimodal instruction following. Nonetheless, existing models frequently suffer from severe object hallucination, outputting plausible-sounding responses that describe non-existent objects or hallucinate factual details unsupported by visual evidence. Conventional alignment paradigms depend heavily on resource-intensive human annotations or reinforcement learning with human feedback (RLHF), where training complex reward models and managing policy stability impose tremendous engineering overhead. Direct Preference Optimization (DPO) has emerged as an elegant alternative by optimizing directly on preference pairs without explicit reward modeling; however, existing multimodal DPO methods primarily focus on data synthesis or model-based ranking without addressing the intrinsic multimodal representational gap.

The fundamental tension lies in the perceptual discrepancies between vision and language modalities. LVLMs interpret and represent identical visual semantics differently depending on the input modality format. For example, when responding to an image of a truck, an LVLM directly viewing the image may erroneously claim that the right door is open; yet when presented with a textual description derived from the same image, the model accurately identifies that the left door is open. Leaving such modality-specific perceptual discrepancies unaddressed injects distorted, conflicting supervision signals into preference data, leading to suboptimal alignment and degraded generation fidelity.

To resolve this limitation, this paper exploits the complementary nature of vision-language representations by transforming visual inputs into explicit textual descriptions, leveraging cross-modal consistency as an autonomous alignment signal without requiring proprietary models or human annotations. Core idea: construct complementary image- and description-based response pairs using open-source models, filter and curate high-margin preference data via a composite Vision-Language Consistency (VLC) score, and inject the cross-modal score discrepancy as an adaptive penalty into the DPO implicit reward objective.

Method¶

Overall Architecture¶

The proposed VLC-DPO framework operates in three modular stages: first, initial response generation produces dual-modality responses (an image-grounded response and an intermediate detailed description-grounded response); second, preference data construction evaluates both inter-response semantic consistency and image-text matching via the VLC score, filtering low-quality candidates and injecting controlled noise into low-score responses to enlarge preference margins; third, VLC-DPO preference training adaptively scales penalties on dispreferred responses according to the cross-modal consistency gap. The overall workflow is illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image & Instruction"] --> B["Dual-Modality Complementary Response Generation<br/>Produce image- and description-based responses"]
    B --> C["VLC Score-Based Preference Pair Construction<br/>Joint semantic consistency & image-text matching"]
    C --> D["Adaptive Penalty Preference Optimization<br/>VLC-DPO training scaled by discrepancy |wc|"]
    D --> E["Perceptually Aligned Multimodal Policy"]

Key Designs¶

1. Dual-Modality Complementary Response Generation: Eliciting Cross-Modal Perspectives Existing LVLMs often overlook subtle visual entities when directly processing raw pixel tensors end-to-end, whereas detailed caption generation can activate latent linguistic knowledge. To capture these complementary perspectives without proprietary APIs (such as GPT-4V), the method relies purely on open-source models: given an input image $I_i$ and an instruction, LLaVA-1.5 generates the direct image-based response $R_I^i$. Simultaneously, an LVLM generates a detailed image description $D_i$, which is subsequently passed alongside the original instruction into a language-only LLM (Llama 2) to generate the description-based response $R_T^i$. This yields an initial response set $\mathcal{D} = \{R_I^i, R_T^i\}_{i=1}^N$ bridging the cross-modal perceptual gap through intermediate textual representations.

2. VLC Score-Based Preference Pair Construction: Dual Verification & Margin Amplification To reliably select preference pairs from noisy candidate responses, the framework introduces a composite Vision-Language Consistency score ($S_{\text{VLC}}$). First, sentence-level semantic consistency $S_{\text{SC}}$ is evaluated using a sentence encoder pre-trained on natural language inference (nli-mpnet-base-v2), computing the cosine similarity between attention-weighted token embeddings $e_1$ and $e_2$ of the two responses. Second, fine-grained image-text matching $S_{\text{ITM}}$ is computed using a pre-trained encoder-decoder model (CLIP + Flan-T5), breaking each response into sentences $\{s_1, \dots, s_n\}$ and calculating the average likelihood of predicting "yes" to the prompt "Does this figure show $s_i$? Please answer yes or no". The composite VLC score is formulated with threshold $\tau$:

\[S_{\text{VLC}} = \begin{cases} \alpha \cdot S_{\text{SC}} + (1 - \alpha) \cdot S_{\text{ITM}}, & \text{if } S_{\text{SC}} \ge \tau \\ 0, & \text{otherwise} \end{cases}\]

Setting $\alpha=0.5$ and $\tau=0.7$, scores $S_{\text{VLC}}^I$ and $S_{\text{VLC}}^T$ are computed for $R_I$ and $R_T$. The higher-scoring response is designated as $R_{\text{chosen}}$ ($y_w$) and the lower as $R_{\text{rejected}}$ ($y_l$). Furthermore, when candidates have excessively similar scores, controlled perturbations (a mixture of 40% random word replacement, 40% masking, and 20% word order shuffling) are selectively injected into the lower-scoring response to create a distinct, unambiguous reward margin while maintaining syntactic plausibility.

3. Adaptive Penalty Preference Optimization: Discrepancy-Modulated Implicit Rewards Standard DPO applies a uniform regularization hyperparameter $\beta$ across all pairs, ignoring sample-level discrepancy severity. VLC-DPO leverages the absolute difference between modal scores $|w_c| = |S_{\text{VLC}}^I - S_{\text{VLC}}^T|$ as an adaptive penalty factor. The weight $|w_c|$ is scaled by 10 and capped at 3 to preserve training stability. When two candidate responses diverge significantly in consistency or image alignment, the dispreferred response receives a harsher log-likelihood penalty:

\[\mathcal{L}_{\text{VLC-DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - |w_c| \cdot \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]\]

Loss & Training¶

The backbone model is LLaVA-1.5 (Vicuna 1.5 + CLIP ViT-L/14@336px) optimized via LoRA with rank 128 and alpha 256. Training runs for 5 epochs with a learning rate of $2 \times 10^{-7}$, batch size 16, $\beta = 0.1$, and $\alpha = 0.5$ across 8 NVIDIA RTX A6000 GPUs utilizing Flash Attention. The preference dataset is curated from 16k pairs of RLAIF-V down to 11.5k high-confidence pairs (~72% retention rate) using VLC filtering.

Key Experimental Results¶

Main Results¶

VLC-DPO was evaluated on four hallucination benchmarks (MMHal-Bench, Object HalBench, AMBER discriminative task, POPE adversarial setting) and two knowledge-based VQA benchmarks (A-OKVQA val, OK-VQA test).

Benchmark	Metric	LLaVA-1.5 Baseline (7B)	VLC-DPO (Ours, 7B)	Previous SOTA (Open-Source)
MMHal-Bench	Score (↑)	1.86	2.95	2.83 (OPA-DPO 7B)
MMHal-Bench	Hallucination Rate (↓)	0.64	0.33	0.41 (DAMA 7B) / 0.32 (RLAIF-V 4-iter)
Object HalBench	Response Hallucination (↓)	54.5%	9.6%	9.1% (DAMA 7B) / 10.5% (RLAIF-V)
Object HalBench	Mention Hallucination (↓)	27.8%	5.0%	4.25% (OPA-DPO 7B) / 4.7% (DAMA)
AMBER (Disc.)	Accuracy (%) (↑)	73.5%	83.1%	83.3% (DAMA 7B) / 80.2% (oDPO 7B)
AMBER (Disc.)	F1-Score (%) (↑)	77.7%	87.2%	87.0% (DAMA 7B) / 84.5% (RLAIF-V)
POPE (Adv.)	Accuracy (%) (↑)	80.8%	85.2%	84.7% (POVID 7B) / 82.6% (OPA-DPO 7B)
A-OKVQA (Zero-shot)	VQA Score (↑)	46.2 (7B) / 52.3 (13B)	60.7 (7B) / 66.5 (13B)	57.8 (Brote-IM-XXL)
OK-VQA (Zero-shot)	VQA Score (↑)	46.4 (7B) / 51.8 (13B)	57.3 (7B) / 59.2 (13B)	56.2 (VCTP)

Ablation Study¶

Component ablations and threshold sensitivity on AMBER and POPE benchmarks (LLaVA-1.5 7B backbone):

Configuration	AMBER Accuracy (%)	AMBER F1-Score (%)	POPE Adversarial Acc. (%)	Note
VLC-DPO (Full Model)	83.1	87.2	85.2	Combines dual-modality, VLC filtering & $
w/o description	80.4	82.0	82.6	Image-only response pair construction without text bridge
w/o consistency	78.5	81.7	81.8	Eliminates semantic consistency threshold filtering
w/o penalty ($	w_c	$)	75.0	77.9
Threshold $\tau = 0.5$	81.0	82.9	83.2	Overly loose filtering retains inconsistent pairs
Threshold $\tau = 0.6$	82.4	86.0	84.1	Moderate filtering
Threshold $\tau = 0.7$ (Default)	83.1	87.2	85.2	Optimal trade-off between quality and sample volume
Threshold $\tau = 0.8$	82.6	86.1	84.7	Overly strict filtering excessively prunes training data
Noise: Noise-free	80.1	81.4	81.6	Narrow preference margin degrades DPO gradient updates
Noise: Word Replacement	81.9	86.1	83.9	Single perturbation strategy
Noise: Word Masking	80.8	82.5	83.0	Single perturbation strategy
Noise: Order Shuffling	76.2	79.0	81.3	Severe syntax destruction hurts preference learning
Noise: Mixture (Default)	83.1	87.2	85.2	40% replacement + 40% masking + 20% shuffling

Key Findings¶

Consistency penalty $|w_c|$ is the most vital component: Removing $|w_c|$ results in a steep decline on AMBER (F1 plunges from 87.2% to 77.9%), proving that dynamically penalizing responses with high modality inconsistency prevents policy collapse into superficial language patterns.
$\tau=0.7$ provides the sweet spot for consistency filtering: Lowering $\tau$ introduces hallucinated pairs, while increasing it to 0.8 excessively restricts training diversity.
Controlled synthetic noise widens reward margins: When response candidates are too close in score, standard DPO struggles. Applying a balanced mixture of word replacement, masking, and mild shuffling creates a robust margin without corrupting linguistic coherence.

Highlights & Insights¶

Zero Proprietary Model Dependency: Unlike OPA-DPO or HSA-DPO which rely heavily on GPT-4V feedback, VLC-DPO achieves SOTA hallucination mitigation using only open-source LLaVA-1.5, Llama 2, and lightweight inference models, offering an accessible pipeline for the open research community.
Turning Cross-Modal Discrepancy into Self-Supervision: Recognizing that models fail differently when seeing pixels versus reading captions, the framework elegantly turns an architectural deficiency into an unsupervised calibration signal.
Discrepancy-Modulated Implicit Rewards: Incorporating the continuous score gap $|w_c|$ as an adaptive multiplier on the rejected loss term provides a generic formulation readily transferable to other preference optimization tasks.

Limitations & Future Work¶

Author-Acknowledged Limitations: The optimization operates strictly at the response level rather than offering fine-grained token-level or object-level supervision; moreover, if both image and description branches suffer from the exact same shared hallucination, the consistency filter cannot detect the error.
Identified Limitations: Generating full intermediate descriptions introduces substantial pre-processing computational cost; furthermore, for abstract imagery or highly dense scenes where natural language struggles to describe every relation, caption bridging may become a representational bottleneck.
Future Directions: Extending VLC-DPO to token-level credit assignment, and integrating grounding models (e.g., Grounding DINO / SAM) to spatially anchor entities mentioned in intermediate descriptions.

vs RLAIF-V: RLAIF-V requires multi-iteration (4-iter) divide-and-conquer feedback from stronger models; VLC-DPO achieves superior POPE and AMBER scores in a single training run via vision-language consistency verification.
vs OPA-DPO & HSA-DPO: Both baseline methods depend on GPT-4V teacher feedback to correct hallucinations or rate severity; VLC-DPO establishes fully self-supervised alignment.
vs mDPO & V-DPO: While mDPO and V-DPO use static reward anchors or visual negative prompts, VLC-DPO leverages bidirectional cross-modal consistency scores to dynamically modulate optimization penalties.

Rating¶

Novelty: ⭐⭐⭐⭐☆ Captures the core perceptual gap between vision and language modalities and formulates it into an elegant unsupervised alignment mechanism.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations across four hallucination benchmarks, two VQA benchmarks, extensive ablations, and perturbation analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, clean mathematical formulation, and well-structured empirical validation.
Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, open-source paradigm for aligning LVLMs without external commercial APIs or manual labeling.