Test-time Counterfactual Calibration for Hallucination-Resistant Temporal Grounding

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/CVL-hub/HRVTG
Area: Multimodal VLM / Video Understanding
Keywords: Video Temporal Grounding, Temporal Hallucination, Test-time Adaptation, Reinforcement Learning, Counterfactual Calibration

TL;DR

To alleviate temporal hallucinations caused by the closed-world always-answer bias in multimodal video LLMs, HRVTG introduces a test-time adaptation framework that constructs self-supervised textual, visual, and hybrid counterfactual probes and dynamically reshapes decision boundaries during inference via GRPO with consistency anchor rewards and asymmetric KL elastic constraints.

Background & Motivation

Video temporal grounding (VTG) aims to identify the precise start and end timestamps of a queried event described by natural language within an untrimmed video. With the rapid evolution of multimodal large language models (Video MLLMs), the generative VTG paradigm has become dominant. By autoregressively generating reasoning rationales and structured time intervals, MLLMs demonstrate remarkable spatial-temporal reasoning capabilities. However, conventional training frameworks and standard evaluation benchmarks uniformly operate under a closed-world assumption: they implicitly presuppose that every user query must correspond to a verifiable event somewhere in the video. This ingrained always-answer bias compresses the model's decision boundary during training, compelling models deployed in the open world to forcibly align counterfactual queries with irrelevant segments based merely on superficial linguistic priors or coarse visual similarities.

To prevent such forced-alignment temporal hallucinations, recent query-refusal studies have attempted to inject predefined negative samples or explicit abstention annotations during offline training. Nevertheless, these offline approaches inherently learn a static and isolated set of fabrication patterns. When deployed in open-world environments featuring unseen query phrasing, severe video domain shifts, or deceptively matched background segments, such static rejection classifiers fail catastrophically. The core tension is that temporal hallucination does not stem from the mere absence of an offline rejection label; rather, it reflects a miscalibrated decision boundary during inference that leaves the model incapable of distinguishing genuine multimodal causal evidence from null-attribution states under insufficient evidence.

Addressing this limitation without relying on costly human negative annotations or static retraining, this paper proposes an online, test-time boundary calibration paradigm. Core idea: establish a cognitive taxonomy of temporal hallucinations spanning textual, visual, and multimodal triggers, synthesize high-quality counterfactual probes on the fly in a self-supervised manner, and dynamically recalibrate the decision boundary via GRPO equipped with consistency anchor rewards and asymmetric KL elastic constraints during zero-shot streaming inference.

Method

Overall Architecture

HRVTG reframes test-time adaptation (TTA) as an online policy optimization problem in reinforcement learning over continuous streaming data. Operating on top of a frozen base grounding backbone augmented with lightweight LoRA adapters, the system processes a stream of paired genuine samples and self-supervised counterfactual probes. The pipeline proceeds across three primary phases: first, counterfactual probes with verifiable event-absence pseudo-labels are synthesized on the fly; second, candidate outputs are generated via group sampling to compute consistency anchor, hallucination repulsion, and formatting rewards; third, policy parameters are updated online using GRPO regularized by an asymmetric KL constraint, suppressing hallucinated temporal intervals while safeguarding genuine grounding accuracy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Streaming Test Data<br/>(Original Video Vori, Genuine Query Qori)"] --> B["Self-supervised Counterfactual Probe Construction<br/>Textual Alteration / Visual Pruning / Hybrid Fabrication"]
    B --> C["Mixed Stream & Candidate Action Sampling<br/>Frozen Backbone + LoRA Policy πθ samples G outputs"]
    C --> D["Verifiable Reward Computation<br/>Consistency Anchor Rca + Repulsion Rhr + Format Rfmt"]
    D --> E["Asymmetric KL Elastic Constraint & GRPO Update<br/>High penalty on positive samples / Flexible on counterfactuals"]
    E --> F["Calibrated Decision Output<br/>Accurate interval for real event / 'not found' for fabricated event"]

Key Designs

1. Self-supervised Counterfactual Probe Construction: Exposing Language Priors and Pseudo-Evidence Alignment Traps Targeting the distinct misattribution pathways in multimodal temporal hallucinations, this paper develops three self-supervised counterfactual probe generation strategies that operate without manual annotation: - Textual Alteration: Keeping the original video \(V_{\text{ori}}\) unchanged, the query \(Q_{\text{ori}}\) undergoes action reversal (e.g., open \(\to\) close), temporal misordering (inverting logical precedence), or attribute/entity substitution (e.g., watermelon \(\to\) potato). Candidate counterfactual queries \(Q_{\text{ta}}\) are filtered via retrieval consistency against the frozen base model to guarantee event absence, explicitly probing whether language priors trick the model into forced alignment. - Visual Pruning: Keeping the original query \(Q_{\text{ori}}\) intact, the high-confidence prediction of the frozen grounder is extracted as a Pseudo-GT evidence interval. This segment is excised from the video, and the remaining clips are concatenated to form \(V_{\text{ngt}}\). This probe preserves the surrounding scene background and actor identities while completely removing the decisive action evidence, directly exposing inertial alignment driven solely by coarse visual context. - Hybrid Fabrication: On the pruned video \(V_{\text{ngt}}\), the original query is used to retrieve the most semantically relevant interval as a pseudo hard-negative clip. A captioning model then produces a natural query \(Q_{\text{phn}}\) mimicking the original style, and the video is recomposed by removing this hard-negative clip from \(V_{\text{ori}}\) while retaining the genuine segment. This creates a deceptive scenario where the video contains rich background cues matching the query vocabulary, but genuine supporting evidence remains absent.

2. Rule-based Verifiable Reward System: Harmonizing Grounding Retention and Hallucination Repulsion Because ground-truth timestamps are absent at test time, the framework employs a piecewise rule-based reward function \(r\). For positive streaming samples \(x_{\text{pos}}\), a consistency anchor reward \(R_{\text{ca}}\) is introduced using the greedy output \(y_{\text{ref}}\) of the frozen initial grounder \(\pi_{\text{ref}}\) as an anchor: $\(R_{\text{ca}}(y, y_{\text{ref}}) = \begin{cases} r_{\text{ca}}, & \text{IoU}(y, y_{\text{ref}}) \ge \delta \land \hat{\tau} \ne \emptyset \land \hat{\tau}_{\text{ref}} \ne \emptyset \\ 0, & \text{otherwise} \end{cases}\)$ No negative penalties are assigned, preventing aggressive updates from eroding the base model's localization capacity. A lightweight format reward \(R_{\text{fmt}}\) is additionally assigned to positive samples to ensure structured tag syntax. For counterfactual samples \(x_{\text{cf}}\) equipped with verifiable absence pseudo-labels, a hallucination repulsion reward \(R_{\text{hr}}\) encourages explicit rejection tokens (such as not found): $\(R_{\text{hr}}(y) = \begin{cases} r_{\text{hr}}, & g(y) = \emptyset \\ 0, & \text{otherwise} \end{cases}\)$ Driven by within-group advantage normalization in GRPO, candidates that reject nonexistent events receive higher relative advantage, yielding an implicit contrastive effect without destructive gradient shocks.

3. Asymmetric KL Elastic Constraint: Decoupling Anchor Stability from Boundary Plasticity Standard GRPO enforces a single, symmetric KL divergence penalty coefficient \(\beta\) across all samples. However, in a mixed streaming environment, positive and counterfactual samples impose contradictory demands on policy drift: positive samples require rigid adherence to the anchor model to prevent catastrophic forgetting, whereas counterfactual samples require substantial deviation to escape the base model's hallucinated outputs. To resolve this dilemma, the framework introduces a sample-aware asymmetric weighting factor \(\lambda(x)\): $\(\lambda(x) = \begin{cases} \rho, & x \in \mathcal{X}_{\text{pos}} \\ 1, & x \in \mathcal{X}_{\text{cf}} \end{cases}\)$ The effective KL penalty becomes \(\beta \cdot \lambda(x)\). By setting \(\rho > 1\) (fixed to \(\rho = 4\) in implementation), positive samples receive strong regularization to maintain grounding capability, while counterfactual probes enjoy greater plasticity to form an adaptive decision boundary.

Loss & Training

During inference, each streaming sample \(x\) is processed by the policy model \(\pi_\theta\) (the frozen backbone with LoRA), which samples a group of \(G\) responses \(\{o_1, \dots, o_G\}\). Based on composite rewards \(r_i\), relative advantages \(\hat{A}_i = \frac{r_i - \text{mean}(r)}{\text{std}(r) + \epsilon}\) are computed. The policy is updated online via GRPO: $\(J(\theta) = \mathbb{E}_{x \sim \mathcal{S}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{\pi_\theta(o_i | x)}{\pi_{\theta_{\text{old}}}(o_i | x)} \hat{A}_i - \beta \cdot \lambda(x) \cdot D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right]\)$ By caching and reusing video visual tokens, backward passes are restricted solely to the LoRA adapters, enabling practical online calibration during streaming test-time execution.

Key Experimental Results

Decoupled Evaluation Metrics

Existing refusal-aware benchmarks couple localization metrics with refusal decisions by assigning an arbitrary IoU of 1 whenever a model refrains from answering a negative sample. This artificially inflates performance for overly conservative models. Furthermore, single-class F1 metrics ignore severe class imbalance. This paper addresses these issues via two decoupled metrics: 1. Conditional IoU (\(mIoU_{\text{Cond}}\)): Measures temporal overlap exclusively across True Positive (TP) instances where both ground truth and model output agree that the event exists, isolating pure grounding precision from refusal mechanics. 2. Balanced Decision F1-score (\(F1_{\text{Balanced}}\)): Computes the harmonic mean of answering and refusal F1 scores (\(F1_{\text{Answer}}\) and \(F1_{\text{Refuse}}\)), heavily penalizing degenerate models that blindly refuse or blindly accept all queries.

Main Results

Evaluations were conducted on Charades-RF, ActivityNet-RF (both with a 1:1 positive-to-negative ratio), and TVGBench-RF (spanning textual, visual, and multimodal composite fabrications), using Qwen2.5-VL at 3B and 7B parameter scales.

Dataset	Model	Size	[email protected]	[email protected]	[email protected]	\(mIoU_{\text{Cond}}\)	\(F1_{\text{Ans}}\)	\(F1_{\text{Rfs}}\)	\(F1_{\text{Bal}}\)
Charades-RF	Base Model	3B	50.36	25.90	13.67	30.27	13.64	67.42	22.69
	Only-positive Model	3B	67.60	38.53	14.90	41.48	69.50	33.38	45.12
	HRVTG (Ours)	3B	71.24	40.49	15.58	43.29	74.86	62.80	68.30
	Base Model	7B	29.02	13.20	5.88	19.50	55.31	73.56	63.14
	Only-positive Model	7B	72.59	53.43	25.55	48.33	81.36	75.40	78.27
	HRVTG (Ours)	7B	73.97	54.07	27.01	48.94	83.00	80.91	81.94
ActivityNet-RF	Base Model	3B	17.70	12.60	5.41	17.70	52.31	73.17	61.00
	Only-positive Model	3B	38.46	21.95	10.19	26.61	73.90	51.22	60.51
	HRVTG (Ours)	3B	40.79	23.00	10.89	28.03	83.81	84.01	83.91
	Only-positive Model	7B	44.15	27.22	14.04	30.85	85.15	85.43	85.29
	HRVTG (Ours)	7B	47.50	28.03	13.03	31.87	89.37	87.71	88.53
TVGBench-RF	Only-positive Model	3B	32.59	18.35	6.33	23.34	66.18	5.28	9.78
	HRVTG (Ours)	3B	40.26	18.53	7.99	26.09	66.67	52.65	58.83
	Only-positive Model	7B	36.70	24.47	10.37	26.31	70.74	42.09	52.78
	HRVTG (Ours)	7B	37.65	24.10	12.35	26.93	72.17	62.35	66.90

Ablation Study

Ablations on TVGBench-RF with Qwen2.5-VL-3B investigated the asymmetric KL weight \(\rho\), anchor IoU threshold \(\delta\), and adaptation learning rates:

Experiment Group	Parameter Variation	\(mIoU_{\text{Cond}}\)	\(F1_{\text{Balanced}}\)	Analysis & Observations
Symmetric vs. Asymmetric KL	\(\rho = 1\) (Standard symmetric GRPO)	21.15	53.16	Symmetrical penalty degrades positive grounding while restricting refusal exploration
	\(\rho = 2\)	24.30	56.40	Partial relaxation begins restoring genuine event precision
	\(\rho = 4\) (Ours, Default)	26.09	58.83	Optimal balance between stability on real events and plasticity on counterfactuals
	\(\rho = 8\)	25.10	57.20	Excessive rigidity on positive samples hampers adaptation flexibility
Anchor IoU Threshold \(\delta\)	\(\delta = 0.1\)	23.80	55.30	Loose threshold rewards low-quality predictions, causing localization drift
	\(\delta = 0.2\) (Default)	26.09	58.83	Successfully balances high evidence fidelity and regular guidance frequency
	\(\delta = 0.5\)	22.40	51.10	Overly stringent criteria cause sparse anchor signals; model collapses toward blanket refusal

Key Findings

1. Hallucination resistance without grounding sacrifice: Compared to conventional positive-only models, HRVTG achieves drastic improvements of 14–49 points in \(F1_{\text{Bal}}\) (e.g., leaping from 9.78 to 58.83 on TVGBench-RF 3B) while simultaneously improving pure localization accuracy \(mIoU_{\text{Cond}}\) by +1.8 to +2.7. This confirms that consistency anchoring effectively counteracts catastrophic forgetting during TTA.
2. Failure of symmetric constraints: Standard GRPO (\(\rho = 1\)) experiences severe competition between retaining grounding fidelity and learning rejection, yielding inferior metrics across both fronts (21.15 and 53.16). Imposing an asymmetric factor (\(\rho = 4\)) resolves this friction.
3. Deceptiveness of hybrid fabrications: While 7B models display partial rejection capability on simple negatives, the 3B positive-only model collapses to an \(F1_{\text{Refuse}}\) of just 5.28 on TVGBench-RF. Cross-modal pseudo-evidence presents an immense trap for generative VLMs, underscoring the necessity of test-time counterfactual probes.

Highlights & Insights

• From static offline fitting to dynamic inference calibration: Rather than attempting to enumerate all negative combinations offline, HRVTG constructs tailored counterfactual probes directly from incoming test videos, calibrating decision boundaries to the live distribution on the fly.
• Feasible online RL closed-loop: Leveraging critic-free GRPO, video feature caching, and parameter-efficient LoRA updates, HRVTG demonstrates that online policy optimization during streaming inference is practically viable.
• Decoupled evaluation exposes metric inflation: By eliminating the artificial convention of setting IoU to 1 upon refusal, the proposed \(mIoU_{\text{Cond}}\) and \(F1_{\text{Balanced}}\) metrics establish a rigorous, cheat-proof benchmark for open-world video understanding.

Limitations & Future Work

• Computational overhead during inference: Although video features are cached and only LoRA parameters are tuned, generating \(G\) outputs and performing backward passes at test time introduces higher latency compared to simple forward inference, posing challenges for hard real-time edge devices.
• Dependence on initial Pseudo-GT quality: Visual pruning and hybrid probes rely on the base model's initial grounding predictions. If the initial model fails severely on long or ambiguous videos, the pruned evidence may contain residual target signals.
• Future directions: Integrating lightweight memory mechanisms for historical scene statistics and extending test-time counterfactual probes to spatiotemporal action detection and long-horizon embodied agent planning.

Related Work & Insights

• Generative VTG foundation: Extends models like VTimeLLM, TimeChat, and Time-R1 that discretize temporal intervals into text tokens, but pioneers the application of DeepSeek-R1 style rule-based GRPO in test-time adaptation.
• Selective prediction & learning to reject: Builds upon the insights of RaTSG, CausalVTG, and RA-RFT regarding always-answer bias, replacing offline contrastive learning and manual negative collection with an entirely self-supervised, online calibration pipeline.

Rating

• Novelty: 4.5 / 5.0 (Pioneering integration of GRPO, self-supervised counterfactual probes, and TTA for temporal grounding)
• Experimental Thoroughness: 4.5 / 5.0 (Thorough evaluations across 3 benchmarks, dual parameter scales, detailed ablations, and decoupled metrics)
• Writing Quality: 4.5 / 5.0 (Cohesive formulation, clean mathematical notation, and systematic hallucination taxonomy)
• Value: 4.5 / 5.0 (Provides an actionable and effective path toward reliable, hallucination-free multimodal video models)