CLUE-VAD: Structured Semantic Clues for Understanding Explainable Events in Video Anomaly Detection

Conference: ECCV2026
Paper: Official page · PDF
Project: CLUE-VAD
Area: Video Understanding
Keywords: Weakly supervised video anomaly detection, structured semantic clues, category-aware fusion, segment attribution, explanation generation

TL;DR

CLUE-VAD organizes video evidence into Action, Environment, and Object captions, performs weakly supervised detection through category-aware fusion, and generates explanations using segment attention and a clue-conditioned language model, achieving 89.47% AUC / 88.24% AP on UCF-Crime / XD-Violence without establishing superiority over every multimodal detector.

Background & Motivation

Weakly supervised video anomaly detection treats a long video as a bag of segments and learns when anomalies occur from video-level supervision, avoiding expensive frame-level annotation. The difficulty is not just that anomalous segments are rare: abnormality depends on context. The same activity can carry different implications in different environments or when different objects are involved. A holistic visual representation may support binary classification while offering little insight into which kind of evidence drove the decision.

Methods such as VadCLIP introduce vision-language alignment, but scene-level category text still bundles different semantic factors together. Visual mixture-of-experts methods can separate action and background representations without automatically exposing language-grounded contributions. Models such as Holmes-VAU generate natural-language explanations, yet a fluent description does not necessarily identify the evidence used by the detector. CLUE-VAD therefore changes the semantic representation supplied to the detector, rather than simply enlarging an explanation model or changing anomaly-score smoothing.

The paper describes each segment through Action, Environment, and Object clues, then distinguishes two kinds of importance: which clues usually matter for a category, and which clues stand out in the current segment. Core idea: establish an evidence interface that can be inspected by semantic role, then learn category-dependent detection fusion and segment-dependent explanation weights so that anomaly scores and language explanations share structured clues.

Method

Overall Architecture

The input is an untrimmed video. Outputs include temporal anomaly scores, clue-level explanations, and optional natural-language event descriptions. Witness turns the video into three semantic evidence streams; Detective combines those streams with visual features using category-aware weights; Reporter derives local attribution from segment clues and supplies clue representations to a multimodal language model.

Detective's category weights are parameters stored by category, not attention predicted afresh for every segment. Reporter handles segment-level variation. Both use the same semantic evidence source, but the paper explicitly positions normalized explanation scores as post-hoc analysis rather than a replacement for scalar anomaly scores during training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Untrimmed video"] --> Witness["Witness Semantic<br/>Evidence Extraction"]
    Witness --> Detective["Detective Category<br/>Inference"]
    Witness --> Reporter["Reporter Segment<br/>Explanation"]
    Detective --> Score["Category and anomaly scores"]
    Detective --> Reporter
    Reporter --> Output["Clue attribution and optional<br/>natural-language explanation"]

Key Designs

1. Witness Semantic Evidence Extraction: replace a holistic description with three independently encoded clues

Pretrained InternVideo 2.5 receives three prompts for each video segment, describing the main activity, contextual setting, and involved entities. This is not an arbitrary split of one long caption: each semantic role is queried separately, reducing the risk that the dominant event description obscures environmental or object information. A frozen CLIP text encoder encodes the three descriptions individually, while a frozen CLIP visual encoder extracts a visual representation from a segment keyframe. Downstream modules can therefore operate on named semantic channels while retaining visual information omitted from the captions.

This step does not supply human frame-level anomaly labels. The text comes from a pretrained video-language model, making visual content inspectable but also turning caption errors into downstream inputs. The paper additionally uses LLaMA3 to extract top-k keywords from each clue and aggregate their frequencies by category and normal/abnormal label. This checks whether the roles capture discriminative vocabulary; it does not establish that every highlighted word has a verified causal effect on detection.

A temporal-alignment ambiguity must remain explicit. The captioning pipeline in Section 3 uses non-overlapping 64-frame segments, whereas Witness in Section 4.1 specifies 16 frames per segment. The cache does not explain the mapping between these granularities, so no sampling or repetition rule should be invented. Segment descriptions, the three text embeddings, and the visual embedding need to correspond, but their precise temporal alignment requires clarification from the implementation.

2. Detective Category Inference: estimate the event category before retrieving its clue weights

Different anomaly categories depend on different semantic roles, so one shared set of fusion weights would suppress useful differences. Detective maintains three learnable weights per category, one each for Action, Environment, and Object. After softmax normalization, these weights scale the respective text embeddings, which are concatenated and projected before concatenation with the visual embedding. A two-layer fully connected scorer with ReLU and Sigmoid produces segment anomaly scores. The category-indexed weight table is called the Case Log Embedding; it is a parameter lookup, not an external memory that retrieves past video cases.

Ground-truth video categories can select weights during training, but are unavailable at inference. The paper therefore adds a video-level category classifier. It concatenates visual and three textual embeddings into snippet tokens, uses a frozen visual saliency selector to retain top-M candidates, and uniformly samples K snippets from that set. A fusion MLP aligns the tokens, a Transformer models relationships among selected snippets, and learnable attention pooling produces a video representation. A cosine classification head with a learnable scale predicts the category. The highest-probability category selects the clue weights, avoiding an indiscriminate average over the many irrelevant snippets in a long video. The cached main text does not specify numerical values for M and K.

Compared with fixed concatenation, this provides an explicit semantic selection mechanism, but it also creates an error pathway: an incorrect category prediction retrieves inappropriate weights. The interaction ablation supports category-aware fusion, yet the available main text does not separately report category accuracy or compare ground-truth-category lookup with predicted-category lookup. A diagram connecting training and inference is therefore not evidence that error propagation has been resolved.

3. Reporter Segment Explanation: use local attention for attribution and clue tokens to condition generation

A category prior cannot explain how evidence changes within a video. Reporter applies a learnable linear transform, ReLU, and a clue-specific query vector to each segment clue, then normalizes across the three clues with softmax. The resulting attention weights form a weighted sum of clue features, which passes through a scoring network. Scores are subsequently z-score normalized using the mean and standard deviation across the video, emphasizing relative temporal outliers. This is within-video deviation, not a calibrated anomaly probability across videos; attention magnitude is also not a causal effect.

For language generation, a projector maps the fused CLUE representation into the language model's embedding space and injects it as prefix tokens. LoRA adapts the pretrained model. The explanation experiment uses InternVL2-2B and evaluates generated text against UCFA test annotations. Unlike merely inserting a predicted category into a prompt, this pathway exposes the generator to structured clue representations. Natural-language generation can be invoked only for selected high-anomaly segments rather than for every segment.

The paper describes its output as clue-level scores and keyword-level evidence, with examples showing particular words and weights. However, the available main text does not sufficiently specify the word-level attribution algorithm, construction of explanation fine-tuning data, LoRA configuration, or the complete training schedule for both branches and the projector. The readable equations and prose establish the overall interface, but do not support reproducing a precise word-contribution algorithm. Those missing details are not supplied here.

A Worked Example

Consider Explosion008 in the paper's Figure 7. Witness extracts activity, environmental, and object information from the input segment. Detective predicts Explosion, retrieves that category's fusion weights, and produces anomaly scores. Reporter displays Action: Smoke 0.34, Environment: Explosion 0.43, and Object: Car 0.23, indicating that environmental evidence has the largest relative weight in this example. These are values from a qualitative figure, not dataset-average weights.

The language model then generates an explanation centered on spreading gray smoke. The original output also repeats itself. This example thus illustrates both evidence visibility and the limits of generation: structured conditioning does not guarantee accurate, complete, or non-repetitive text. Assigning Smoke to Action also shows that the three semantic roles are not inherently mutually exclusive.

Loss & Training

The category classifier uses focal loss to address class imbalance. Detection uses a top-k MIL ranking objective, encouraging the mean of the k highest segment scores in an abnormal video to exceed the corresponding mean in a normal video. This k serves a different purpose from the category classifier's sampled K snippets. Supervision is video-level, but category lookup and category classification additionally require video-level category information; the entire framework should not be described as using only one binary label.

Equation (10) is damaged in the text extraction. The expression below restates the ranking mechanism from the adjacent prose and visible unit-margin term, rather than claiming to recover the original equation in full. P and N denote segments in abnormal and normal videos, respectively; the original paper or code remains authoritative for implementation details:

\[ \mathcal{L}_{\mathrm{MIL}}=\max\left(0,1-\operatorname{TopMean}_{k}(\{\hat{s}_t:t\in P\})+\operatorname{TopMean}_{k}(\{\hat{s}_t:t\in N\})\right). \]

Attention also receives a negative-entropy regularizer to discourage excessive concentration on one clue, and a squared-difference penalty between neighboring segments to discourage abrupt changes. The main text explicitly gives the combined objective:

\[ \mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{MIL}}+\lambda_{\mathrm{AE}}\mathcal{L}_{\mathrm{AE}}+\lambda_{\mathrm{smooth}}\mathcal{L}_{\mathrm{smooth}}. \]

This expression does not include the category focal loss or specify a language-generation loss. Section 4.4 mentions MIL training of the segment explanation mechanism while also stating that the attention z-score branch is solely for post-hoc explanation. It is clear that normalized scores are not the detection ranking target, but the joint or staged optimization of all learnable components remains underspecified. Learning rates, training epochs, regularization coefficients, and sampling parameters are not filled in from generic practice.

Key Experimental Results

Main Results

The following detection results come from the paper's Table 1. Values are percentages and higher is better. UCF-Crime uses AUC and XD-Violence uses AP; these columns must not be compared as if they were the same metric.

Method	UCF-Crime AUC (%)	XD-Violence AP (%)
VadCLIP	88.02	84.15
Ex-VAD	88.29	86.52
Holmes-VAU	88.96	87.68
LEC-VAD	89.97	88.47
π-VAD	90.33	85.37
CLUE-VAD	89.47	88.24

The gains over Holmes-VAU are 0.51 / 0.56 percentage points. However, LEC-VAD is higher in both columns, and π-VAD has higher UCF-Crime AUC. The supported claim is leadership within the listed explainable-VAD group, not absolute superiority across all VAD methods.

Ablation Study

The table below selects interaction ablations from Table 4 and modality analysis from Table 5. Interaction variants are compared in the text-only UCF-Crime setting; the difference between 88.34 and the multimodal 89.47 must not be attributed to a change in interaction structure.

Experiment Scope	Configuration	UCF-Crime AUC (%)	XD-Violence AP (%)
Text interaction	Single-layer self-attention fusion	86.89	Not reported
Text interaction	Graph message-passing fusion	86.13	Not reported
Text interaction	Pairwise clue interaction	87.46	Not reported
Text interaction / text modality	Category-aware clue weighting	88.34	87.77
Modality	Visual only	87.46	86.35
Modality	Text and visual fusion	89.47	88.24

Category-aware weighting improves on pairwise interaction by 0.88 percentage points. Adding visual input improves on text alone by another 1.13 / 0.47 percentage points. Text and vision therefore provide complementary evidence, but these results do not establish that every caption is reliable.

Explanation generation is evaluated against UCFA test annotations. The following selection from Table 3 preserves the original decimal scale rather than converting it to percentages.

Method	Parameters	BLEU-4	CIDEr	METEOR	ROUGE
Video-LLaVA	7B	0.0310	0.0393	0.1398	0.2085
InternVL2	8B	0.0339	0.0713	0.1424	0.2158
CLUE-VAD	2B	0.0739	0.0423	0.1729	0.2762

Key Findings

• Generation gains are not uniform across metrics: CLUE-VAD's CIDEr is below InternVL2's. Better text-matching metrics do not directly prove faithful explanations or correct causal attribution.
• Table 6 reports 0.66 ms for online detection modules, 542 ms for explanation generation, and 542.66 ms for the online total. The 59.57 ms caption precomputation is excluded from that total. The 0.66 ms figure must not be presented as end-to-end real-time latency starting from new raw video.
• The available main text does not report random-seed variance, statistical significance, or complete hyperparameter sensitivity results. The stability of small detection gains still needs replication.

Highlights & Insights

• Semantic roles become an explicit interface. The same Action, Environment, and Object channels support detection and inspection of explanation evidence, making input-side error analysis easier than tracing an unconstrained generated paragraph.
• Category priors and segment variation are modeled separately. One describes the evidence typically associated with an anomaly category; the other captures what stands out now. This distinction can transfer to long-video event analysis with category priors.
• Generation is an optional downstream operation. Separating lightweight scoring from expensive explanation can control generation cost, but deployment still needs to account for captioning newly arriving video.

Limitations & Future Work

• Input dependence acknowledged by the authors: clue quality depends on the captioning model, and recognition errors can propagate through category prediction, anomaly scores, and explanations. Segment-level representations also struggle to separate simultaneous activities or region-specific anomalies.
• Attribution lacks direct validation: the evidence mainly consists of weight visualizations, qualitative examples, and text-generation metrics. Clue deletion, counterfactual replacement, and visual grounding checks could test whether highly weighted clues actually change detection decisions.
• Reproduction details remain incomplete: the 64 / 16-frame inconsistency, damaged equations, explanation training details, and branch optimization relationships need clarification. The local cache does not include the appendix repeatedly referenced by the main text, so its supplementary settings cannot be treated as verified.
• Hard category selection and offline captions introduce assumptions: incorrect categories can amplify fusion errors, while online cost relies on precomputation. Soft mixing by category probabilities and full latency measurement on fresh video streams are possible directions proposed by this note, not findings established by the paper.

Related Work & Insights

• Compared with VadCLIP: both use CLIP, but CLUE-VAD emphasizes explicit semantic roles for each segment and category-dependent fusion rather than only improving scene-level vision-text alignment.
• Compared with TDSD, Sherlock, and Skynet-V1: TDSD introduces contextual and object text, while the latter methods emphasize visual expert decomposition. CLUE-VAD fixes Action, Environment, and Object as language-grounded channels and exposes inspectable category and segment weights.
• Compared with Holmes-VAU: Holmes-VAU emphasizes long-video, multi-granularity understanding and anomaly-focused sampling; CLUE-VAD emphasizes the types of evidence supporting an explanation. The tabulated gains support the practicality of this organization, not direct comparability across all understanding tasks.

Rating

• Novelty: 4/5. Connects category-level semantic priors, segment attribution, and a generation interface; the novelty is evidence organization rather than a new backbone.
• Experimental Thoroughness: 3/5. Covers two detection benchmarks, interaction and modality ablations, and explanation generation, but lacks sufficient faithfulness validation and reproduction settings.
• Writing Quality: 3/5. The three-module narrative is clear, but temporal granularity and training-branch descriptions are ambiguous, and cached equations are damaged.
• Value: 4/5. A useful structured baseline for explainable video analysis, pending robustness, calibration, and end-to-end latency evaluation before deployment.