Skip to content

CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://aimagelab.github.io/CounterVid
Area: Multimodal VLM
Keywords: Video-Language Models, Video Hallucination, Counterfactual Video Generation, Preference Alignment, Direct Preference Optimization (DPO)

TL;DR

To eliminate action and temporal hallucinations caused by Video-Language Models over-relying on linguistic priors, this paper introduces CounterVid—a scalable generative pipeline synthesizing scene-consistent counterfactual videos differing solely in action dynamics—and proposes MixDPO, a unified alignment objective jointly optimizing textual and visual preferences.

Background & Motivation

Large-scale Video-Language Models (VLMs, such as Qwen2.5-VL, InternVL3, and LLaVA-NeXT) have demonstrated remarkable multimodal comprehension across open-ended video QA and long-horizon video reasoning benchmarks. However, when deployed in complex real-world environments, these models exhibit persistent and severe dynamic hallucinations. While they can reliably identify static objects, recognize background settings, and describe fine-grained visual attributes, they struggle profoundly with semantic action recognition and temporal event ordering. Empirical investigations reveal that such dynamic errors do not stem primarily from spatial resolution bottlenecks; instead, they originate from models excessively leaning on linguistic statistical priors. For example, upon recognizing a kitchen setting, a model frequently assumes a conventional sequence of washing, cutting, and cooking, asserting that an action occurred even when the video depicts an inverted sequence, an omitted step, or an entirely unrelated gesture.

Existing mitigation paradigms primarily focus on architectural interventions, inference-time contrastive decoding, or preference-based alignment. Architectural and decoding-time adjustments, such as reweighting visual attention or contrasting against degraded visual inputs, incur substantial computational overhead or require bespoke decoding routines during inference, limiting throughput. On the alignment front, preference learning methods like Direct Preference Optimization (DPO) have shown promise, but prior video-specific approaches (such as PaMi-VDPO) synthesize negative samples using naive perturbations, including random spatial cropping, frame shuffling, or temporal reversal. While these mechanical augmentations disrupt temporal coherence, they introduce unnatural visual artifacts and motion discontinuities without altering the underlying action semantics in a controlled manner. Consequently, VLMs exploit low-level artifacts and static visual cues as trivial shortcuts to separate chosen from rejected videos, failing to learn fine-grained action physics and temporal dynamics.

The fundamental barrier to solving dynamic video hallucinations is the lack of content-aware, semantic counterfactual video data that decouples static scene context from dynamic action trajectories. When two videos share an identical scene, background, and camera viewpoint but exhibit distinct action evolutions, the model can no longer rely on contextual shortcuts. Furthermore, traditional preference learning only penalizes hallucinated textual answers under fixed visual inputs, leaving models prone to ignoring visual evidence altogether when the text prompt looks linguistically plausible. Core idea: build CounterVid, a scalable counterfactual video generation pipeline combining multimodal LLM planning, image editing, and image-to-video diffusion to synthesize semantic hard negatives differing strictly in action dynamics while preserving identical scene context, and align VLMs using MixDPO, a unified framework jointly coupling textual and visual preference optimization to enforce both output grounding and input sensitivity.

Method

Overall Architecture

The proposed framework comprises two interconnected stages: a counterfactual video synthesis pipeline yielding the CounterVid dataset, followed by preference alignment via MixDPO. In the generation stage, real video clips are used solely to retrieve a stable, representative anchor frame. Guided by multimodal LLM planning, the pipeline proposes physically feasible alternative actions, uses an image-editing model to generate the corresponding action-completed end frame with iterative verification, and synthesizes temporally coherent clips via an image-to-video diffusion model. In the alignment stage, these synthetic counterfactuals form paired preference samples spanning action recognition and temporal ordering, which are fed into MixDPO to simultaneously optimize textual answer discrimination and visual input sensitivity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Real Video-Caption Anchor<br/>(Extract keyframe I_start from PVD clip)"] --> B["Context-Preserving Counterfactual Video Generation<br/>Action proposal filtering + end-frame editing + I2V synthesis"]
    B --> C["Fine-Grained Action & Temporal Preference Construction<br/>Action recognition & temporal ordering multi-format pairs"]
    C --> D["Unified Dual-Preference Optimization (MixDPO)<br/>Joint textual and visual preference loss alignment"]
    D --> E["Hallucination-Mitigated Video LLM<br/>Grounded action recognition & temporal order sensitivity"]

Key Designs

1. Context-Preserving Counterfactual Video Generation: Decoupling static context from dynamic action semantics Direct text-to-video synthesis of negative clips inevitably shifts the entire scene background and camera perspective, violating the strict counterfactual control necessary for hard-negative preference learning. To isolate action dynamics, the pipeline implements a four-step generative process: First, SigLIP-SO400M extracts a representative starting frame \(I_{start}\) from short (<10s) real video clips (from the PVD dataset) using a coarse-to-fine temporal search (2 fps coarse retrieval followed by 12 fps refined sampling across multiple spatial crops). Second, conditioned on \(I_{start}\) and the original caption, Claude-4-Sonnet proposes \(N\) candidate actions that are physically plausible, involve visible subjects, and remain semantically distinct, followed by automated prompt-based filtering to eliminate redundant or unrealistic suggestions. Third, Qwen-Image-Edit (with Lightning LoRA in FP8) translates each action caption into a structured instruction to generate the action-completed end frame \(I_{end}^i\), incorporating an LLM-guided iterative refinement loop (up to 5 attempts) to rectify semantic drift or identity loss. Finally, Wan2.2-I2V-14B synthesizes a smooth ~3.06s clip (\(680\times 384\), 49 frames at 16 fps) conditioned on \((I_{start}, I_{end}^i, C_i)\). By anchoring generation to \(I_{start}\), all candidate clips share the exact same background, camera angle, and scene geometry while differing strictly in action trajectories.

2. Fine-Grained Action & Temporal Preference Construction: Dual-preference pairs for action recognition and sequence ordering Once the synthetic action set \(\mathcal{G} = \{(V_i, C_i)\}_{i=1}^N\) is constructed, original real videos are discarded to prevent visual distribution leakage. For action recognition, the system forms textual preference pairs (t-pref) by holding the video fixed (\(x=(V_i, Q_{act})\)) and contrasting the grounded action \(A^+=C_i\) against an ungrounded alternative \(A^-=C_j\) sampled from the same anchor pool; conversely, visual preference pairs (v-pref) fix the text \((Q_{act}, A=C_i)\) and contrast the true action video \(V^+=V_i\) with a counterfactual video \(V^-=V_j\). For temporal ordering, \(K\) distinct clips from the same anchor are concatenated into sequence \(S=[V_{i_1}; \dots; V_{i_K}]\). The forward concatenated order defines \(S^+\), while non-identity permutations define counterfactual video \(S^-\). In t-pref, the model discriminates correct action ordering descriptions from permuted or swapped text captions under fixed video \(S^+\); in v-pref, the model discriminates between \(S^+\) and \(S^-\) given the correct sequence description. Across free-form, binary choice, multiple choice, and order list formats, CounterVid curates 26,167 high-quality pairs (70% visual preferences, 30% textual preferences) specifically targeting language-prior guessing.

3. Unified Dual-Preference Optimization (MixDPO): Jointly optimizing output grounding and input sensitivity Standard DPO only contrasts output textual candidates under fixed visual inputs, which fails to penalize "blind" generation where a VLM outputs a plausible answer without properly attending to the visual tokens (i.e., \(\pi_\theta(A|V^+, Q) \approx \pi_\theta(A|V^-, Q)\)). MixDPO overcomes this by defining an implicit reward \(r_\theta(x, y) = \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}\) and formulating a joint objective: $\(\mathcal{L}_{MixDPO}(\theta) = \mathcal{L}_{t-pref}(\theta) + \lambda \mathcal{L}_{v-pref}(\theta)\)$ The textual preference term sharpens output likelihoods over visually supported text: $\(\mathcal{L}_{t-pref}(\theta) = -\mathbb{E} \left[ \log \sigma \left( \beta \left( r_\theta((V, Q), A^+) - r_\theta((V, Q), A^-) \right) \right) \right]\)$ while the visual preference term explicitly forces the implicit reward of the correct video to dominate that of the counterfactual video under identical textual queries: $\(\mathcal{L}_{v-pref}(\theta) = -\mathbb{E} \left[ \log \sigma \left( \beta \left( r_\theta((V^+, Q), A) - r_\theta((V^-, Q), A) \right) \right) \right]\)$ Setting \(\lambda=1.0\) maintains an optimal equilibrium between output token sharpening and input-evidence sensitivity, penalizing models that attempt to guess answers from linguistic habits when temporal cues in the video contradict the text.

Loss & Training

The framework adopts parameter-efficient fine-tuning on Qwen2.5-VL (3B and 7B) and InternVL3-9B. The vision encoder remains frozen except for the vision-language merger module, while LoRA is applied to the language model's attention projections (rank 64, \(\alpha=16\)). Fine-tuning is conducted for 3 epochs at 2 fps using the AdamW optimizer with a learning rate of \(1\times 10^{-6}\), an effective batch size of 8 distributed across 8 GPUs, DPO temperature \(\beta=0.7\), and preference loss balancing weight \(\lambda=1.0\). Evaluation enforces strict pixel limits (maximum 151,200 pixels, minimum 100,352 pixels) and a temporal sampling cap of 32 frames across all video benchmarks.

Key Experimental Results

Main Results

Evaluation spans the held-out CounterVid benchmark (2,910 samples partitioned at the anchor level) and four recognized external video hallucination benchmarks: EventHallusion, VidHalluc, VideoHallu, and VideoHallucer.

1. CounterVid Benchmark Performance

MixDPO delivers extensive gains across all action recognition and temporal ordering task formats compared to the base checkpoints, supervised fine-tuning (SFT), and the perturbation-based PaMi-VDPO baseline.

Backbone Model Training Protocol Temp: FF Temp: OL Temp: BC Act: FF Act: MC Act: BC Overall Avg
Qwen2.5-VL-3B Base 29.8 1.6 48.9 28.0 48.4 64.6 36.9
SFT 29.4 2.0 47.9 27.2 51.9 65.2 37.2
T-pref-DPO 32.2 15.7 56.0 29.6 58.7 69.5 43.6
PaMi-VDPO 32.9 1.4 47.5 29.1 52.3 68.0 38.5
MixDPO (Ours) 31.0 16.7 64.0 29.5 60.0 70.3 45.2 (+8.3)
Qwen2.5-VL-7B Base 70.3 16.5 57.8 55.1 68.6 78.2 57.8
SFT 69.3 14.9 55.9 55.3 67.7 79.2 57.1
T-pref-DPO 72.2 37.1 67.4 56.0 70.8 79.8 63.9
PaMi-VDPO 42.2 13.4 57.6 32.9 64.4 74.5 47.5
MixDPO (Ours) 72.2 43.8 72.7 56.3 71.2 80.7 66.2 (+8.4)
InternVL3-9B Base 66.3 13.2 53.0 49.9 65.3 75.4 53.8
SFT 66.2 16.9 51.7 53.2 69.7 74.8 55.4
T-pref-DPO 72.5 56.4 80.9 62.2 78.5 81.5 72.0
PaMi-VDPO 66.5 14.9 52.3 53.7 65.3 74.3 54.5
MixDPO (Ours) 74.1 61.3 84.7 63.2 79.6 81.1 74.0 (+20.2)

(Note: FF = Free-Form, OL = Order List, BC = Binary Choice, MC = Multiple Choice)

2. External Hallucination Benchmark Generalization

MixDPO demonstrates strong out-of-distribution transfer to external benchmarks without task-specific tuning, yielding notable improvements on temporal action sequences (VidHalluc TSH).

Model Variant EventHallusion (Avg) VidHalluc (TSH) VideoHallu (Overall) VideoHallucer (Avg)
Qwen2.5-VL-3B Base 59.4 78.5 35.1 52.6
T-pref-DPO 64.3 80.0 36.3 53.6
PaMi-VDPO 66.4 76.2 36.3 50.7
MixDPO (Ours) 64.4 80.0 37.9 54.6
Qwen2.5-VL-7B Base 70.1 82.7 38.2 54.1
T-pref-DPO 72.7 87.8 39.8 55.1
PaMi-VDPO 68.3 83.0 38.7 55.1
MixDPO (Ours) 72.5 90.5 39.3 55.7
InternVL3-9B Base 64.6 39.3 26.5 54.6
T-pref-DPO 73.5 83.5 25.5 59.5
PaMi-VDPO 58.6 65.7 30.7 50.7
MixDPO (Ours) 73.7 88.8 25.7 58.9

Ablation Study

Loss Weight, Preference Ratio, and Regularization Sensitivity (CounterVid)

A controlled sensitivity analysis evaluates variations in the visual preference weight \(\lambda\), the visual/textual data ratio (\(v/t\)), and the KL temperature coefficient \(\beta\):

Model Backbone Configuration Setting Temp. Ordering (Avg) Action Recog. (Avg) Overall Benchmark Avg
Qwen2.5-VL-7B Default (\(\lambda=1, \beta=0.7, 70/30\)) 62.9 69.3 66.1
\(\lambda = 0.5\) 62.9 69.0 66.0
\(\lambda = 2.0\) 62.3 69.1 65.7
Ratio \(v/t = 50/50\) 62.7 69.3 66.0
Ratio \(v/t = 90/10\) 64.2 68.8 66.5
\(\beta = 0.5\) 67.0 69.0 68.0
\(\beta = 1.0\) 58.4 69.1 63.8
InternVL3-9B Default (\(\lambda=1, \beta=0.7, 70/30\)) 73.2 74.7 74.0
\(\lambda = 0.5\) 73.1 74.9 74.0
\(\lambda = 2.0\) 73.0 74.6 73.8
Ratio \(v/t = 50/50\) 72.0 74.8 73.4
Ratio \(v/t = 90/10\) 74.1 74.7 74.4
\(\beta = 0.5\) 74.1 75.1 74.6
\(\beta = 1.0\) 71.0 74.5 72.7

Key Findings

  • Visual preferences provide indispensable input sensitivity: While textual preference optimization (T-pref-DPO) outperforms SFT, adding visual preferences in MixDPO yields substantial extra gains in complex temporal ordering tasks. Specifically, order-list accuracy increases from 37.1% to 43.8% on Qwen2.5-VL-7B (+6.7pp) and from 56.4% to 61.3% on InternVL3-9B (+4.9pp), demonstrating that penalizing invariant predictions under counterfactual videos forces the model to attend to motion details.
  • Random perturbation negatives fail at semantic action reasoning: PaMi-VDPO, which relies on random frame cropping and shuffling, achieves poor order-list accuracy (1.4% to 14.9%) on CounterVid, performing on par with or worse than the base model. This confirms that VLMs quickly learn to detect low-level motion discontinuities without acquiring genuine action or temporal understanding.
  • Preservation of general video understanding capabilities: Across general benchmarks evaluated without duration filtering (VideoMME, NExT-QA, and TempCompass), MixDPO preserves broad multimodal performance. On Qwen2.5-VL-7B, VideoMME accuracy remains at 61.5% (vs. 61.4% base), NExT-QA MC stays solid at 74.9%, and TempCompass Caption Matching improves from 76.2% to 80.2%.

Highlights & Insights

  • Agentic feedback loop for image editing: Employing multimodal LLMs as reflective critics to iteratively refine editing instructions (up to 5 iterations) resolves semantic drift, ensuring end frames reliably portray the intended action outcome while preserving background fidelity.
  • Formal visual preference loss in DPO: By holding the text query and answer constant while swapping positive and negative videos, MixDPO introduces an elegant mathematical formulation for visual input sensitivity, penalizing models that ignore visual tokens.
  • Reusable synthetic counterfactual paradigm: The "keyframe extraction \(\to\) action proposal \(\to\) endpoint editing \(\to\) video diffusion" workflow provides a cost-effective, scalable template for synthesizing high-purity hard negatives beyond video hallucination, applicable to robotic trajectory evaluation and multimodal safety alignment.

Limitations & Future Work

  • Diffusion synthesis fidelity constraints: Human evaluation reveals that 12.0% of synthetic samples exhibit visual flaws or incomplete action execution (e.g., objects failing to move). While artifact shortcut effects are mitigated by quality filters, future work should incorporate advanced physical world models.
  • Short-duration clip focus: Current counterfactuals center on short clips (~3 seconds) and their direct concatenations, without addressing long-range causal dependencies, complex multi-agent interactions, or partially overlapping events.
  • Frozen visual encoder adaptation: To manage computational costs, LoRA adaptation is restricted to language attention layers while freezing the ViT backbone. End-to-end joint fine-tuning of the visual backbone could further sharpen temporal feature sensitivity.
  • vs. PaMi-VDPO (Ding et al., 2025): PaMi-VDPO utilizes frame shuffling and spatial cropping as negative instances, teaching models to identify low-level disruption artifacts; CounterVid synthesizes semantic counterfactuals that preserve scene context, resulting in vastly superior temporal sequence discrimination.
  • vs. VISTA-LLAMA (Ma et al., 2024) & Contrastive Decoding: These methods modify attention distribution or introduce extra decoding passes during inference; MixDPO directly aligns model weights, incurring zero additional runtime latency or memory overhead during deployment.
  • vs. V-DPO / mDPO (Xie et al., 2024; Wang et al., 2024a): While V-DPO and mDPO formulate input-level visual preferences for static images, MixDPO extends the concept to video temporal sequences, addressing the unique challenges of action progression and event ordering.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering counterfactual generative pipeline for semantic hard-negative video synthesis and video-level visual DPO formulation.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 26k curated pairs, 3 backbone models, 4 external hallucination benchmarks, and human verification studies.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, seamlessly connecting theoretical motivation, pipeline implementation, mathematical formulations, and empirical analyses.
  • Value: ⭐⭐⭐⭐⭐ Provides an impactful open-source dataset, codebase, and alignment recipe for tackling dynamic hallucinations in video VLMs.