HeadHunt-VAD: Hunting Robust Anomaly-Sensitive Heads in MLLM for Tuning-Free Video Anomaly Detection

Conference: AAAI 2026
arXiv: 2512.17601
Code: N/A
Area: Multimodal VLM / Video Understanding
Keywords: Video Anomaly Detection, Multimodal Large Language Model, Attention Head Selection, Tuning-Free, Internal Representation Probing

TL;DR

This paper proposes HeadHunt-VAD, which systematically identifies a sparse set of anomaly-sensitive and stable attention heads within a frozen MLLM, bypassing the information loss inherent in text-based outputs. Using a lightweight classifier, it achieves efficient tuning-free video anomaly detection, establishing state-of-the-art performance among tuning-free methods on UCF-Crime and XD-Violence.

Background & Motivation

Background: Video Anomaly Detection (VAD) aims to localize events in video that deviate from normal patterns. Traditional approaches (supervised, weakly supervised, unsupervised) have achieved competitive results but generally require large-scale annotated data and substantial computational overhead. Recently, tuning-free methods based on frozen MLLMs (e.g., LAVAD, VERA) have emerged as a promising new direction, leveraging the rich world knowledge encoded in such models.

Limitations of Prior Work: Current tuning-free methods primarily rely on the text output of MLLMs to determine anomaly presence, which introduces three critical issues: (1) Information loss — converting high-dimensional visual information into natural language inevitably discards subtle anomaly cues; (2) Normality bias — MLLMs tend to describe common objects while overlooking unusual details that define anomalies; (3) Prompt sensitivity — semantically equivalent but differently worded prompts applied to the same video may yield inconsistent predictions.

Key Challenge: These methods depend on the final text output layer, whereas research has shown that intermediate layers contain richer representations than the output layer. However, directly using entire layer features is a coarse-grained strategy — within the multi-head attention of a Transformer layer, different heads serve distinct functions, and the signals from a few discriminative heads are overwhelmed by the many heads attending to background features, causing a representation dilution problem.

Goal: (1) How to identify attention heads that are genuinely sensitive to anomalies at the head level? (2) How to ensure that the selected heads remain stable and effective across diverse prompts? (3) How to achieve efficient detection with minimal data?

Key Insight: The authors observe that individual attention heads in intermediate layers exhibit far greater discriminative power for normal/anomaly distinction than the aggregated output. This suggests that directly exploiting head outputs prior to aggregation can circumvent representation dilution.

Core Idea: Within a frozen MLLM, systematically hunt a small set of anomaly-sensitive heads via multi-metric saliency analysis combined with cross-prompt stability evaluation, and employ a lightweight classifier to achieve tuning-free anomaly detection.

Method

Overall Architecture

HeadHunt-VAD consists of two stages: offline preparation and online inference. The offline stage includes: (1) Robust Head Identification (RHI) — a multi-criterion analysis that filters all attention heads to produce a sparse set of "consensus expert heads"; and (2) training a lightweight Anomaly Scorer and Temporal Locator on the expert head features. During online inference, a single forward pass is performed on the input video, features are extracted exclusively from the expert heads, and the scorer and locator jointly perform real-time anomaly detection and localization.

Key Designs

1. Robust Head Identification Module (RHI):

   • Function: Select the top-K expert heads that are cross-prompt stable and highly discriminative from all \(N_{total} = N_{layers} \times N_h\) attention heads in the MLLM.
   • Mechanism: Proceeds in two steps. Step 1 — Head Saliency Evaluation: For each head, the feature vector of the first generated token is extracted to construct normal/anomaly feature sets. Saliency scores are computed along four complementary dimensions: LDA score (linear separability), symmetric KL divergence (distributional discrepancy), MMD (kernel-space distributional distance), and NMI (mutual information between clustering and labels). Step 2 — Robust Head Selection: Comprehensive saliency is computed for each head across \(M\) diverse prompts, and a robust saliency score \(RSS(k) = \mu_k - \lambda \sigma_k\) is defined (mean minus a standard deviation penalty), analogous to a risk-aversion principle — requiring both high mean (strong discriminability) and low variance (cross-prompt stability). Heads are ranked by RSS and the top-K are selected.
   • Design Motivation: A single metric is prone to bias; multi-dimensional evaluation ensures that selected heads possess both linear separability (suitable for lightweight classifiers) and information-theoretic discriminability. The stability penalty addresses prompt sensitivity — heads that perform exceptionally well under specific prompts but poorly under others are filtered out.

2. Anomaly Scorer:

   • Function: Maps expert head features to anomaly probabilities.
   • Mechanism: For each video, the feature vectors of \(K\) expert heads are concatenated to obtain \(\mathbf{z}_i \in \mathbb{R}^{K \cdot d_h}\), and logistic regression is trained by minimizing binary cross-entropy loss. At inference, \(p_i = \sigma(\mathbf{w}^T \mathbf{z}_i + b)\).
   • Design Motivation: Logistic regression is chosen over more complex models for efficiency and interpretability, consistent with the overall lightweight design philosophy. Ablation experiments show that an MLP yields only a marginal improvement of 0.22%.

3. Temporal Locator:

   • Function: Converts raw frame-level anomaly probability sequences into precise temporal event localization.
   • Mechanism: The frame-level anomaly probability sequence is first temporally smoothed using a 1D Gaussian kernel \(p'_t = (\mathbf{p} * G_{\sigma_g})_t\), then binarized with a data-driven threshold \(\tau\). Both \(\sigma_g\) and \(\tau\) are determined by grid search to maximize frame-level F1 score on the validation set.
   • Design Motivation: Gaussian smoothing eliminates noise from isolated frames, while the data-driven threshold avoids performance degradation associated with manually fixed thresholds.

Loss & Training

• The Anomaly Scorer uses standard binary cross-entropy: \(\mathcal{L} = -\frac{1}{N}\sum[y_i \log p_i + (1-y_i)\log(1-p_i)]\)
• Only 1% of training set data is required for few-shot calibration training.
• The MLLM remains fully frozen throughout; no fine-tuning is performed.

Key Experimental Results

Main Results

Dataset	Metric	HeadHunt-VAD	HiProbeVAD	VERA	Type
UCF-Crime	AUC(%)	87.03	86.72	86.55	Tuning-Free
XD-Violence	AP(%)	82.63	82.15	-	Tuning-Free

Compared to weakly supervised methods on UCF-Crime: HeadHunt-VAD (87.03%) approaches CLIP-TSA (87.58%) and VadCLIP (88.02%), but requires neither annotated data nor training.

Ablation Study

Configuration	AUC(%)	AP(%)	Note
Full Model	87.03	82.63	Complete model
w/ Full Layer Features	80.15	72.10	Entire layer features; −6.88% AUC
w/ Random-K Heads	66.65	45.33	Random head selection; catastrophic drop
w/ Single Coarse Prompt	81.86	74.52	Single coarse prompt; −5.17% AUC
w/o Gaussian Smoothing	82.44	75.88	No temporal smoothing; −4.59% AUC
w/ Fixed τ=0.50	80.32	71.49	Fixed threshold underperforms data-driven

Key Findings

• RHI contributes most significantly: Random head selection causes AUC to plummet to 66.65%, while full-layer features achieve only 80.15% due to representation dilution, validating the centrality of precise head selection.
• Cross-prompt robustness is critical: Single-prompt RHI underperforms multi-prompt RHI by approximately 5%, confirming the importance of cross-prompt consistency.
• Efficiency advantage is substantial: Feature dimensionality is compressed from 100K+ (full layer) to only 640; only 1% of training data is needed; a single forward pass avoids the high overhead of autoregressive decoding.

Highlights & Insights

• Probing MLLM internal representations at the head level rather than the layer level is an elegant contribution. Prior work (e.g., HiProbeVAD) operates at the layer granularity, whereas head-level analysis is finer-grained and avoids signal interference among functionally diverse heads. This paradigm is transferable to any task requiring discriminative feature extraction from MLLM internals.
• The Robust Saliency Score (RSS) draws inspiration from the risk-aversion principle in finance, applying a mean-variance trade-off to attention head selection — a clever cross-domain transfer.
• The overall framework is highly lightweight: frozen MLLM + logistic regression + 1% data, making it highly amenable to engineering deployment.

Limitations & Future Work

• Validation is currently limited to InternVL3; it remains unclear whether the identified "expert heads" transfer across different MLLM architectures.
• The number of expert heads \(K=5\) is a fixed hyperparameter, lacking an adaptive selection mechanism.
• The "optional" event description generation step mentioned in the paper requires full autoregressive decoding, which contradicts the efficient inference objective.
• The offline calibration phase of RHI still requires a small number of labeled normal/anomaly samples, precluding fully zero-shot operation.

Related Work & Insights

• vs HiProbeVAD: HiProbeVAD employs layer-level features; this work further refines the analysis to the head level, avoiding representation dilution and improving AUC from 86.72 to 87.03.
• vs VERA: VERA improves MLLM text-based reasoning through prompt optimization, remaining dependent on text outputs; HeadHunt-VAD entirely bypasses text generation by directly exploiting internal representations.
• vs LAVAD: LAVAD requires an additional LLM for auxiliary reasoning, incurring greater overhead; HeadHunt-VAD completes detection with a single model and a single forward pass.

Rating

• Novelty: ⭐⭐⭐⭐ Probing MLLM internal representations at the head level for VAD is pioneering, though the overall framework is relatively straightforward.
• Experimental Thoroughness: ⭐⭐⭐⭐ Two mainstream benchmarks with detailed ablations and visualization analysis, though generalization experiments across more MLLM backbones are lacking.
• Writing Quality: ⭐⭐⭐⭐ Clear structure, well-articulated motivation, and complete technical details.
• Value: ⭐⭐⭐⭐ Offers meaningful insights for the MLLM internal probing paradigm with strong practical utility.

Related Papers

• [CVPR 2026] No Need For Real Anomaly: MLLM Empowered Zero-Shot Video Anomaly Detection
• [AAAI 2026] Learning to Tell Apart: Weakly Supervised Video Anomaly Detection via Disentangled Semantic Alignment
• [ICLR 2026] Steering and Rectifying Latent Representation Manifolds in Frozen Multi-Modal LLMs for Video Anomaly Detection
• [AAAI 2026] Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding
• [AAAI 2026] Harnessing Vision-Language Models for Time Series Anomaly Detection