Learning to Deny: Action Denial in Multimodal Large Language Models¶

Conference: ECCV2026
arXiv: 2606.31187
Paper: Project Page raiyaan-abdullah.github.io/Learn-to-Deny-webpage
Code: https://github.com/raiyaan-abdullah/Learn-to-Deny
Area: Multimodal VLM / Video Understanding / Causal Reasoning
Keywords: Action Denial, Video MLLM, Causal Graph, Shortcut Learning, Sycophancy Bias

TL;DR¶

This paper proposes the UCF101-AD benchmark (11k paired videos of "Action-Presence/Action-Denial") specifically to evaluate whether MLLMs can correctly deny an action when "the scene, objects, and actors are all present but the defining movement does not occur." The study reveals that while 20 state-of-the-art (SOTA) models achieve >85% accuracy in identifying positive samples, their accuracy on denial samples generally falls below 50%. Consequently, the CausalAct causal graph framework is proposed (representing the scene as a Directed Acyclic Graph (DAG) with seven nodes: P/O/L/S/I/M/A, and fine-tuning with natural language prompts and graph concepts), which significantly reduces false positives and demonstrates that "denial" is a learnable reasoning skill.

Background & Motivation¶

Background: Video MLLMs represented by Qwen-VL, InternVideo, VideoLLaMA, Ovis, etc., achieve high zero-shot accuracy on standard action recognition benchmarks (e.g., UCF101, Kinetics). However, an increasing number of analyses indicate that this performance is largely driven by spurious correlations—the models infer actions directly from contextual cues like backgrounds and objects, rather than verifying the "defining movement" of the action. The lack of systematic study on this issue stems from the fact that existing benchmarks (UCF101, Kinetics, HMDB51, ActivityNet) almost exclusively consist of positive samples, where the annotated actions actually occur and the defining movements are visible. They rarely include hard counterfactual negative samples where the relevant objects, actors, and scenes are present, but the action does not take place. Since models have never been trained or evaluated on segments with similar contexts but absent movements, whether they can disentangle true action evidence from misleading contexts remains an invisible blind spot.

Key Challenge: Robust video understanding requires models to not only confirm actions through defining movements but also reliably deny actions when the movement is absent. However, MLLMs naturally suffer from dual biases that counteract this: perceptually, they rely on static shortcuts (context-to-action shortcuts); linguistically, they exhibit "sycophancy" (a tendency to confirm premises rather than deny them). The overlay of these biases leads models to claim an action is occurring simply because the context looks correct. The goal of this paper is to build a benchmark specifically for evaluating "action denial" to expose this vulnerability, and to provide a method that teaches models how to deny.

Core Idea: The authors' core idea is to reconceptualize actions from "correlates of context" to "products of structured causal chains". An action is the result of a structured combination of actors, environment, interactions, and motion dynamics, with motion being the most critical factor. Only when MLLMs are guided to verify evidence step-by-step along the causal chain of "context → spatial relations → interaction → movement → action", rather than short-circuiting from context directly to action, will they correctly deny the action when the critical evidence (defining movement) is missing. This conceptualization leads to a dual-track design: UCF101-AD (diagnosis) and CausalAct (remediation).

Method¶

This paper makes two separable contributions: a diagnostic benchmark (UCF101-AD) and an improvement method (CausalAct). The details of the method revolve around these two components and how they are integrated.

Overall Architecture¶

Diagnostic Side (UCF101-AD): Starting from UCF101 action classes, paired "Action-Presence" and "Action-Denial" segments are constructed for each action. Negative samples deliberately retain the scene, object, and actor context of the positive samples but remove the defining movement. These are categorized into two types: Type 1 "Context Only" (e.g., "Not Playing Piano" = a person is near a piano but not playing), and Type 2 "Same Context, Different Movement" (e.g., "Not Basketball Dunk" = someone is dribbling and passing on a court but not dunking). Evaluation is conducted using a fixed 11-way Multiple Choice Question (MCQ) VQA format, which includes a primary distractor (corresponding to the original UCF101 target action), other randomly selected actions, and a "None" option. All annotations undergo 100% human verification (with no MLLM involvement to avoid model bias), totaling 11,283 samples.

Remediation Side (CausalAct): "Whether an action occurs" is modeled as a causal verification on a Directed Acyclic Graph (DAG) consisting of contextual nodes (Actor P, Object O, Location L) → relational nodes (Spatial relation S, Interaction I) → dynamic node (Movement M) → action node (A). This graph is first translated into natural language prompts (CausalAct-0, for zero-shot use) to guide strong language backbones to verify evidence along the causal chain. Then, smaller models undergo graph concept fine-tuning using auxiliary VQA tasks which only involve graph structures without exposing action labels, teaching the models to internalize these causal dependencies.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    UCF["UCF101 Action Classes"] --> BENCH["UCF101-AD Paired Benchmark<br/>Presence vs Denial"]
    BENCH -->|"Type 1 Context Only<br/>Type 2 Same Context, Diff Action"| EVAL["11-way MCQ Evaluation<br/>20 SOTA MLLMs"]
    EVAL --> DIAG["Diagnosis: Presence >85%<br/>Denial <50% Dual Biases"]
    DIAG --> GRAPH["CausalAct Causal Graph<br/>P·O·L→S·I→M→A"]
    GRAPH -->|"Translate to Natural Language"| PROMPT["CausalAct-0 Causal Prompt<br/>Zero-shot Guided Verification"]
    GRAPH --> FT["Graph Concept Fine-Tuning<br/>14 Graph-Structure VQA Tasks<br/>No Action Labels Exposed"]
    PROMPT --> GAIN["Denial Accuracy Improves<br/>False Positives Decrease"]
    FT --> GAIN

Key Designs¶

1. Paired Counterfactual Negatives: Forcing "Defining Movement" as the Sole Discriminative Signal

The limitation of prior benchmarks is that negative samples are either localized distractor frames, failed attempts, or visually blurry scenes, none of which isolate the "movement" variable. UCF101-AD addresses this by creating a negative sample for each positive class with virtually identical context, differing only in the defining movement. Two key constraints are applied: first, it only selects actions that "require modeling temporal motion dynamics" (excluding static atomic states like sitting/standing), and requires the action to be fully executed to count as Action-Presence (merely swinging a golf club slightly does not count as a swing); second, the Type 1 / Type 2 dichotomy explicitly decouples "contextual cues" from "movement evidence"—Type 1 retains the object/environment but lacks interaction, while Type 2 retains the same scene but replaces the action with another reasonable one. Consequently, to answer correctly, the model must verify the movement instead of guessing based on context. To further reduce ambiguity, each short UCF101 label is rewritten as a descriptive sentence for the distractors (e.g., "HulaHoop" → "A person is spinning a hula hoop").

2. Dual-Failure Diagnosis: Proving Shortcut Learning and Sycophancy Stem from the Same Underlying Bias

This is a highlight of the benchmark analysis. Not content with merely observing "low model scores", the authors ask: Are shortcut learning (relying on context short-circuiting) and sycophancy (the tendency to answer yes) two independent failures, or two sides of the same bias? To test this, an additional Binary Yes/No setup is constructed to directly measure sycophancy (e.g., "Is a person applying makeup to their eyes?"). The Pearson correlation is then calculated between "MCQ error rate" (the frequency of failing to select "None") and "sycophancy rate" (the frequency of incorrectly answering "yes") for each video. Results show: all models exhibit a positive correlation, with an average of \(r=0.388\) and \(p<0.01\) for each model; Type 2 (\(r=0.409\)) exhibits a stronger correlation than Type 1 (\(r=0.367\)). Crucially, for every model, \(P(\text{yes}\mid\text{MCQ Error})>P(\text{yes}\mid\text{MCQ Correct})\) and \(P(\text{yes}\mid\text{Primary Distractor})>P(\text{yes}\mid\text{Random Distractor})\) (with a difference of 0.2–0.6). This demonstrates that sycophancy is highly concentrated on samples where the model has already locked onto a misleading action hypothesis. The tight coupling of these two errors points to the same root cause: inferring actions from context rather than verifying defining movements.

3. CausalAct Causal Graph: Blocking the context→action Short-Circuit via DAG

The key limitation is that models operate under an "association paradigm", implicitly equating "presence of context" with "presence of action". Inspired by ActionGenome, CausalAct decomposes the action scene into a DAG consisting of seven variables, with the dependencies defined as:

\[S\leftarrow f_S(P,O,L),\quad I\leftarrow f_I(P,O),\quad M\leftarrow f_M(I),\quad A\leftarrow f_A(I,M)\]

The intuition is: context (actor/object/environment) determines spatial configuration → spatial configuration enables interaction → interaction generates movement patterns → an action should only be inferred if the appropriate movement is observed. This structure explicitly disables shortcuts like \(A\leftarrow L\) or \(A\leftarrow O\) that directly connect context to action, forcing the model to first verify the presence of interactions and movements. The graph can also adapt to different action types: actor-centric actions can link movement directly to action, while object-centric actions require both interaction and movement to be satisfied simultaneously. In implementation, the DAG is translated into a structured natural language prompt (defining each node, clarifying directed edges like P→I, I→M, M→A, and providing positive and negative examples like "playing guitar" vs. "not playing guitar"). This is prepended to the user query, instructing the model to "infer the components internally without outputting the reasoning process, and only return the option number".

4. Graph Concept Fine-Tuning: Teaching Models to "Understand the Graph" Rather Than "Memorize the Answers"

The limitation is that models with weak vision-language alignment (e.g., VideoLLaMA3) struggle to follow structured prompts, and CausalAct may even degrade performance under zero-shot settings. The solution is an auxiliary fine-tuning phase: for each training video, a graph \((P,O,L,S,I,M,A)\) is constructed, and VQA questions that only probe the graph itself are automatically generated. These cover 14 question types across 4 major categories: graph structure queries (node/edge counting, membership/edge validity checks), node relations (parent-child enumeration, cardinality, adjacency verification), path finding (enumerating all paths from P to A, verifying path validity), and attribute consistency (replacing a node description with attributes of a conflicting class to have the model identify inconsistencies, causal antecedent identification, reverse causal verification). Crucially, these questions never expose the actual action labels, avoiding label leakage. The model learns a transferable skill to "verify the causal chain before asserting an action", rather than memorizing the negative labels of UCF101-AD. This explains why training solely on abstract graph questions yields significant improvements on the Action-Denial test set.

Key Experimental Results¶

Main Results: Zero-Shot Performance of 20 MLLMs on UCF101-AD¶

Model	Type 1↑	Type 2↑	Overall-AD↑	Presence↑	HM↑
VideoLLaMA3-7B	49.4	53.4	51.5	96.0	67.0
Qwen2.5-VL-72B	42.6	47.8	45.7	97.6	62.3
Ovis2.5-9B(thinking)	36.7	43.8	40.4	96.7	57.0
Ovis2.5-9B	34.5	33.8	34.1	97.5	50.5
GPT-4o-mini	20.7	22.3	21.5	90.1	34.7
Valley-Eagle-7B	8.8	13.1	11.1	96.4	19.9

Human Baseline: 36 participants achieved 86.6% Overall-AD on UCF101-AD, proving that denial samples are reliably distinguishable by humans, and model failures are not due to data noise.
The strongest model, VideoLLaMA3-7B, achieved only 51.5% Overall-AD, with most models falling between 20% and 35%. Meanwhile, the Presence accuracy (positive samples) of all models was often >90%, confirming they "can recognize but cannot deny".

Performance Improvement of CausalAct (Smaller Models After Graph Concept Fine-Tuning)¶

Model	Base Overall-AD	CausalAct	Δ
Ovis2.5-2B	27.0	52.3	+25.3
Qwen2.5-VL-3B	19.8	40.7	+20.9
VideoLLaMA3-2B	25.4	43.3	+17.9

On external datasets (Ovis2.5-9B zero-shot CausalAct-0 vs. Base), denial accuracy improved across the board: HMDB51 +23.1, SSv2 +21.4, Diving48 +22.7, K400 +11.5, showing that causal verification is transferable across distributions.

Ablation Study (Unrelated to ScanNet, All Evaluated on UCF101-AD)¶

Ablation Dimension	Key Findings
Is Graph Structure Important?	Insensitive to Pruned/Random graphs in the zero-shot setting (does not know how to read graphs without training); after fine-tuning, disrupting causal dependencies mostly leads to performance drops—proving that the model indeed learns to utilize the complete structure.
Is It Just Language Fine-Tuning?	Updating only the LLM + projector and freezing the vision encoder leads to an average drop of ~15%—the vision stack must be updated to ground the causal graph to visual evidence; it is not pure prompt-following.
Progressive Disambiguation	Standard → Explicit Denial (replacing 'None' with an explicit denial sentence) → removing primary distractors; accuracy progressively approaches positive sample levels—indicating models can deny, but require sufficiently low ambiguity.
Number of Options	11-way MCQ → 4-way MCQ (retaining primary distractor + None); most models still score <50%—the difficulty lies not in the number of options, but in the act of denial itself.

Key Findings¶

"Thinking" models are actually worse: Except for Ovis2.5, reasoning models generally underperform their standard versions. Their chain-of-thought often relies too heavily on visual context, confidently choosing the most plausible distractor action, and even hallucinating non-existent action descriptions. Currently, thinking models are poorly calibrated for negative constraints: they tend to explain "what might be happening" rather than demonstrating "why nothing is happening".
Scale helps, but depends on architecture: Qwen2.5-VL increases by +25.9 from 3B to 72B, and VideoLLaMA3 increases by +26.1 from 2B to 7B, while Ovis2.5 only increases by +7.1 from 2B to 9B—scale can be converted into denial capability, provided the architecture can leverage this capacity.
Type 1 is generally harder than Type 2: This indicates that models excessively index on the "presence of objects/environments". The gains from CausalAct are larger on Type 2, as graph knowledge is particularly effective at distinguishing target actions from similar non-target movements.
CausalAct-0 is a low-cost reasoning alternative: Ovis2.5-9B-Thinking requires 639 minutes to reach 40.4% Overall-AD, whereas the standard Ovis2.5-9B combined with CausalAct-0 achieves comparable performance in only 109 minutes—making structured prompting more cost-effective than expensive CoT.

Highlights & Insights¶

The "paired counterfactual + movement-only difference" benchmark construction paradigm is highly clean: By strictly controlling for identical contexts and only removing the defining movement, the ambiguous question of "whether the model truly understands the action" is converted into a quantifiable "action denial accuracy". This methodology can be transferred to any discriminative task where models might cheat via shortcuts—simply by constructing paired negative samples where surface cues are present but crucial evidence is missing.
Unifying shortcut learning and sycophancy into a single bias (\(r=0.388\), \(p<0.01\)) is a highly convincing piece of diagnosis: it connects two failure modes (perceptual short-circuiting / language conformity) that were previously studied in isolation under a single correlation coefficient, suggesting that debiasing should address both.
Graph concept fine-tuning avoids label leakage: Teaching only graph structures without providing action labels allows the model to learn a transferable "verify causal chain before making assertions" skill. This is a beautiful exemplar of "teaching the method rather than teaching the answer", and can be generalized to any scenario where one wants the model to learn a "reasoning protocol" rather than memorizing outcomes.
Discovery of poor calibration of thinking models on negation: An counter-intuitive and highly valuable observation—more reasoning steps do not equate to greater robustness. When the task is to "argue a denial", existing CoT actually amplifies the affirmation bias.

Limitations & Future Work¶

Closed-set, short videos, and MCQ evaluation: Currently constrained to a closed set of actions derived from UCF101, an average duration of ~7 seconds per clip, and an 11-way MCQ. This is still far from real-world open-world settings, long videos, and free-form QA. The authors also acknowledge that the "Explicit Denial" setting (explicitly telling the model that the action does not occur) is unavailable in real-world scenarios.
Skepticism on the scalability of causal graphs: Complex real-world videos contain numerous objects, making it impractical to fit all visible entities into the graph/prompt. The authors propose actor-centric filtering (retaining only objects spatially close to or interacting with the actor) as a future direction.
CausalAct gains depend on the capability of the language backbone: In zero-shot settings, weakly aligned models (VideoLLaMA3) may even suffer performance drops when using the prompt, necessitating fine-tuning. Furthermore, fine-tuning requires updating the vision encoder (otherwise dropping performance by 15%), which incurs non-trivial computational costs.
Unverified in embodied/interactive environments: Whether denial capability generalizes to embodied or interactive environments remains an open question. Future directions include making causal verification a differentiable module rather than pure prompting, extending to open vocabularies, and integrating actor-centric graph pruning to handle crowded long videos.

vs. Existing Action Recognition Benchmarks (UCF101/Kinetics/HMDB51): These benchmarks mostly have implicit negative samples (background/unlabeled frames) or are designed to detect adjacent failure modes (similar actions in the same scene, failed attempts). UCF101-AD is the first to be explicitly constructed around "action denial"—removing the defining movement while keeping misleading context in paired positive-negative setups, thereby filling the gap in evaluating whether models can deny when movement is absent.
vs. Causal Inference / Scene Graph Works (ActionGenome, VCDN, CLADDER): These mostly utilize graphs for prediction or explanation. This paper inverts this by asking "whether explicit structures can help models decide when to deny under strong context", transforming causal graphs from a predictive tool into a verification tool.
vs. Debiasing Methods (decoupling background/object, e.g., ALBAR): Existing debiasing methods primarily operate in the positive sample regime, rewarding correct labels rather than rewarding explicit denial when movement is absent. CausalAct directly targets denial scenarios, combating contextual shortcuts through structured verification of preconditions.
vs. Sycophancy / Conformity Bias Studies: Prior works studied these biases or reasoning in isolation. This paper uses "video action denial" as a testing ground for the intersection of both and presents a de-sycophantic solution.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The overlooked capability of "action denial" + paired counterfactual benchmark + unifying shortcut learning and sycophancy + using causal graphs for verification, exhibiting strong originality in multiple aspects.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Highly solid evaluation covering 20 SOTA models, human baselines, Type split analysis, dual-failure correlation, generalization on 6 external datasets, multi-dimensional ablations (graph structure, language fine-tuning, number of options, model scale), and computational cost analysis.
Writing Quality: ⭐⭐⭐⭐ Logic flows progressively, the dual diagnostic/remediation threads are clear, and the appendix (complete prompts + 14 question types) is detailed. The main text contains many figures/tables that require cross-referencing.
Value: ⭐⭐⭐⭐⭐ Reveals a critical blind spot in modern video MLLMs (the failure to causally determine if movement actually occurs), which is highly relevant to false-positive sensitive areas like surveillance, autonomous driving, and sports analytics, while providing reusable diagnostic tools and low-cost improvement methods.