Pistachio: Towards Synthetic, Balanced, and Long-Form Video Anomaly Benchmarks¶
Conference: ECCV2026
Paper: ECCV Paper
Area: Video Understanding
Keywords: video anomaly detection, video anomaly understanding, synthetic data, class balance, long-form narratives
TL;DR¶
Pistachio uses scene-constrained storylines to drive continuous video generation for a synthetic detection-and-understanding benchmark, exposing difficulties with complex normal behavior and long-term understanding while showing that CLIP-TSA trained with synthetic data and 10% of UCF-Crime training data outperforms training on the complete UCF-Crime training set alone.
Background & Motivation¶
Video anomaly detection (VAD) identifies when an anomaly occurs, whereas video anomaly understanding (VAU) also describes what happens and how events unfold. The tasks share video inputs but not all annotation requirements: frame-level anomaly labels cannot replace event descriptions and complete narratives. Traditional surveillance datasets often concentrate on a small set of fixed scenes, allowing models to associate backgrounds with anomalies instead of understanding behavior. Scaling Internet collection does not necessarily solve this problem because events likely to be uploaded and shared dominate, while rare environmental hazards and equipment failures remain scarce. When normal samples mostly show simple actions such as walking, models may also classify complex but ordinary interactions as anomalous. Higher scores on established benchmarks therefore do not necessarily indicate robustness to new scenes, moving viewpoints, or complex normal activities.
Building a richer real-world benchmark involves more than collecting additional videos. Rare anomalies are difficult to obtain on demand, deliberately recreating hazardous events has practical constraints, and long videos require detailed temporal boundaries and semantic annotations. Videos containing multiple events also require distinctions between local actions and the main anomaly in the overall narrative. Video generation models offer a new control point: specify scenes and events before synthesizing footage instead of accepting the distribution of available online material. However, a single long prompt can be forgotten during generation, while independently concatenated clips can disrupt identities, viewpoints, and event continuity. The value of synthetic data consequently depends on turning control into verifiable visual content, not merely writing richer prompts.
The paper uses storylines as a shared intermediate representation for generation and semantic annotation. Scenes constrain plausible anomalies, segmented storylines specify event progression, and the preceding segment's final frame supplies the next visual starting point. Quality filtering and visual verification remain necessary because an intended event in a prompt is not proof that it appears in the video. Core Idea: actively design anomaly distributions through scene constraints, segmented narratives, and continuous synthesis, then establish separate frame-level detection labels and multi-granularity understanding annotations so that synthetic data complements real-world benchmarks.
Method¶
Overall Architecture¶
Pistachio contributes a dataset construction pipeline and evaluation benchmark rather than a new anomaly detection network. Inputs are real scene images and manually specified scene-anomaly configurations; outputs are synthetic videos, VAD frame-level labels, and VAU textual descriptions. Section 3.2 starts from COCO 2017 images, while Section 4.1 additionally specifies MIT Indoor Scenes and web images of infrastructure to cover indoor environments, bridges, and tunnels. The pipeline performs scene-constrained assignment, segmented storylines and continuous synthesis, quality filtering, and task-separated annotation with visual refinement. These stages respectively control event plausibility, temporal coherence, video usability, and whether annotations reflect the generated content. Downstream detectors consume the videos or features required by their evaluation protocols; inference does not rerun the data generation pipeline.
The VAD component contains 4,962 videos and 1,676,822 frames, reported as 29.11 hours at 16 FPS, spanning 6 broad scene categories and 31 anomaly categories. Table 1 reports 3,896 individual scenes, which should not be confused with the 6 higher-level scene categories. The VAU component contains 1,385 videos and 517,514 frames, with both event-level and video-level text. Target durations are 41 seconds for long videos and 16 seconds for short videos; long-form here is relative to short generated clips, not hour-long surveillance.
Key Designs¶
1. Scene-constrained assignment: make anomalies compatible with the initial image
Randomly assigning any anomaly to any image can create stories with implausible locations or missing required participants. A vision-language model (VLM) first assigns images to 6 scene categories, with an additional Other category for unmatched images. The categories cover commercial and entertainment locations, indoor premises, outdoor natural environments, critical infrastructure, industrial and construction zones, and public roads and transportation areas. Each scene category has a manually defined set of plausible anomaly candidates rather than an unrestricted global label pool. The VLM selects an anomaly using the specific image and scene-conditioned candidates, after which the corresponding scene-anomaly storyline template is used. For example, infrastructure and natural environments support different risks, so these constraints remove some semantic conflicts before generation begins.
This design turns category coverage from a post-collection sampling problem into a pre-generation configuration problem. Normal stories are also explicit generation targets, covering complex but ordinary activities such as handshaking and vehicle boarding. This matters because enriching anomalies without enriching normal samples encourages models to treat action complexity as an anomaly cue. The authors emphasize a more balanced category distribution, but the main text provides no per-category quotas, resampling rules, or precise balancing objective. The defensible interpretation is improved long-tail coverage through controlled assignment, not a guarantee of identical video counts for every category. Likewise, category balance does not imply the absence of demographic, photographic-style, or generator biases.
2. Segmented storylines and continuous synthesis: control event progression and visual transitions together
Qwen2.5-VL-32B-Instruct interprets the input image and generates the storyline. A long storyline contains 7–8 descriptive segments, while a short storyline contains 2–3 segments. Each segment specifies actions, viewpoints, and temporal progression rather than merely describing a static scene, assigning a new event step to the next clip. This explicit sequence differs from slightly modifying the previous prompt and asking the generator to continue. The latter can simply prolong the previous state instead of executing the planned new event. The storyline should also include normal activity, anomaly onset, and subsequent reactions rather than only the anomalous instant.
Wan generates the video segment by segment, conditioning the first segment on the original scene image. Each subsequent segment takes the preceding segment's final frame as its image condition together with the next event description. The generated clips are concatenated in sequence, avoiding independent visual restarts from unrelated images. Final-frame propagation supplies the next visual starting point, while segmented text specifies what should happen next; both controls are necessary. The authors qualitatively compare this strategy with direct long-form generation, identical-first-and-last-frame looping, and naive storyline chaining. Direct generation can drift semantically and produce ghosting, while forcing a return to the initial frame can induce unnatural acceleration or reversal. These failure analyses motivate the design but are not controlled numerical ablations and cannot quantify each component's independent contribution. Local continuity also does not guarantee long-term identity consistency; the limitations section acknowledges subtle appearance changes.
3. Quality filtering: inspect visual content and event logic after generation
A correctly planned event does not guarantee a correctly generated video. VideoScore first evaluates multiple quality dimensions and removes the bottom 5%, corresponding in the text to average scores below 3.98. This automatic stage rejects conspicuously poor candidates rather than proving anomaly-label accuracy. Human reviewers then check real-world event plausibility, object consistency, and abrupt visual distortions. Camera angles, viewpoints, and lighting are also checked to reject footage that looks clear but does not resemble plausible photography. The automatic-then-human sequence concentrates manual effort on candidates that pass preliminary filtering.
Section 4.1 reports approximately 2 person-days of manual filtering, with acceptance rates of 90% for normal videos and 80% for anomaly videos. These figures describe filtering effort and acceptance, not annotation accuracy or a unified generation-success estimate. Generation itself took approximately 20 days on 32 NVIDIA A100 80GB GPUs, demonstrating that scalability does not imply low compute cost. Manual filtering also means the complete benchmark is not produced without human involvement. Passing filtering does not establish that all generation artifacts are absent, since long-term identity shifts can be subtle. A more defensible conclusion is that the pipeline reduces conspicuous quality problems while remaining domain differences require real-world evaluation.
4. Task-separated annotation and visual refinement: distinguish frame-level ground truth from semantic descriptions
VAD requires anomaly positions on the timeline, and the paper explicitly uses human annotation to define frame-level ground truth. VAU instead reuses local event descriptions from the storyline to reduce the cost of writing long textual annotations from scratch. These descriptions serve both as generation prompts and as the foundation of event-level annotations. Qwen3-8B aggregates local events into a video summary, moving from individual actions to the complete narrative. Aggregation must preserve event order and context rather than merely concatenate prompts. The model called Qwen3-VL-32B Refiner in the paper subsequently compares summaries with the actual video and corrects factual or temporal mismatches.
Visual refinement is essential because prompts express intentions, while the generator may omit actions or alter details. Without inspecting the footage, an understanding model could be penalized for failing to describe an event that never occurred visually. Event text therefore provides a scalable draft, while visual checking reduces discrepancies between descriptions and generated content. The claim of annotation without manual effort should be restricted to VAU text generation, not extended to VAD labels or human quality filtering. The introduction reports 35 multi-anomaly videos, whereas the extracted text of Figure 3 shows 36, leaving a source-level or extraction-level inconsistency. The available cache does not contain the supplementary split definitions and prompt templates, so exact train/test counts or annotation-agreement statistics cannot be supplied.
Loss & Training¶
The paper introduces no new training loss and does not describe task-specific training of Wan or the storyline models. Trainable VAD baselines learn on the Pistachio training set and are evaluated on its test set, using pretrained I3D or ViT video features. VADTree is an exception: it processes videos directly using event-boundary detection and video-language models according to its original implementation, without parameter updates. This qualifies Table 2's blanket statement that every method is trained; the specific protocol in Section 4.4 takes precedence. Detection metrics are frame-level ROC-AUC and AP, reflecting discrimination across thresholds and precision-recall performance, respectively. VAU semantically matches generated descriptions against references and computes event-level and video-level F1 separately. The text counts a match as correct above a predefined semantic-similarity threshold but does not specify that threshold or the complete matching algorithm. F1 follows the usual definition; the equation below expands the paper's harmonic-mean description rather than introducing a new loss:
Here \(P\) and \(R\) are precision and recall under that semantic-matching protocol, not a token-by-token matching score. The VAU scores consequently support model comparisons within the same protocol but cannot directly be interpreted as probabilities of fully understanding real events.
Key Experimental Results¶
Main Results¶
The following selection comes from Table 2, page 10, reporting Overall frame-level metrics on the Pistachio test set in %. Trainable methods use the Pistachio training set; VADTree is evaluated without training and does not have the same in-domain training conditions.
| Method | Feature Backbone | AUC | AP |
|---|---|---|---|
| RTFM | I3D | 82.9 | 69.3 |
| DR-DMU | I3D | 81.5 | 71.5 |
| MULDE | I3D | 63.4 | 34.9 |
| CLIP-TSA | ViT | 80.9 | 57.3 |
| PEL4VAD | ViT | 83.7 | 71.0 |
| Fed-WSVAD | ViT | 83.2 | 71.9 |
| VADTree | Not specified | 72.52 | 25.15 |
PEL4VAD has the highest overall AUC, whereas Fed-WSVAD has the highest overall AP; the rankings should not be conflated. The authors attribute some low scores to complex normal behavior and fine-grained interactions, but these explanations are not causal evidence from isolated-variable experiments.
Ablation Study¶
The paper provides no quantitative ablation of generation components; the following uses the data-fusion analysis in Table 4, page 13. The detector is fixed to CLIP-TSA, with all results evaluated on the UCF-Crime test set in %.
| Training Data Configuration | AUC | AP |
|---|---|---|
| Pistachio only | 61.58 | 10.48 |
| Full UCF-Crime only | 82.56 | 21.62 |
| Pistachio + 10% UCF-Crime | 83.96 | 24.25 |
| Pistachio + 30% UCF-Crime | 86.19 | 28.44 |
| Pistachio + 50% UCF-Crime | 86.79 | 33.04 |
Adding 10% real training data exceeds training on the complete real dataset alone by 1.40 percentage points in AUC and 2.63 percentage points in AP. However, the table describes data combination while the prose also uses fine-tuning terminology; the cache does not resolve the exact schedule of joint training versus sequential pretraining and fine-tuning.
Key Findings¶
Table 5, page 14, reports event-level and video-level VAU F1; the following retains representative models on the original numerical scale.
| Model | Parameters | Event-level F1 | Video-level F1 | Average |
|---|---|---|---|---|
| InternVL3-1B | 1B | 28.25 | 23.04 | 25.65 |
| Qwen2.5-VL-7B | 7B | 30.57 | 25.58 | 28.08 |
| Qwen3-VL-8B | 8B | 28.14 | 26.65 | 27.40 |
| InternVL3-8B | 8B | 32.56 | 25.65 | 29.11 |
| InternVL3-14B | 14B | 32.29 | 26.49 | 29.39 |
InternVL3-8B leads at the event level, Qwen3-VL-8B leads at the video level, and InternVL3-14B has the highest average, illustrating differences across temporal granularities. In Table 3, page 12, Pistachio-trained CLIP-TSA achieves 72.17/75.89 AUC on Cosmos/Hailuo. The cross-generator paragraph on page 13 instead gives 85.22/82.34 AUC without clearly identifying the same model and configuration; these prose values must not replace Table 3. Joint training is also not best on every metric: its MSAD AP in Table 3 is 25.69, below the 54.24 of Pistachio-only CLIP-TSA.
Highlights & Insights¶
- Using storylines for both generation control and initial annotations reduces the cost of writing long-video descriptions from scratch. Visual refinement also acknowledges that plans and observations need not agree.
- Enriching normal behavior is as important as enriching anomaly categories. The former tests whether a detector merely treats unusual or complex actions as abnormal, while the latter broadens event coverage.
- Data-fusion results are more persuasive than claiming that synthetic data completely replaces real footage. Limited target-domain data can correct photographic and viewpoint differences while retaining the coverage provided by synthetic samples.
Limitations & Future Work¶
- The authors explicitly acknowledge the real-domain gap: Pistachio-only CLIP-TSA trails target-domain training substantially on UCF-Crime and cannot directly replace low-resolution fixed-camera surveillance data.
- Local final-frame conditioning does not guarantee long-term identity consistency, and generated characters can still change appearance; this limits tasks dependent on persistent identity tracking.
- Balance should not be interpreted as absence of bias. Readers should examine generator styles, textual-annotation model preferences, and whether visual refinement favors related model families.
- Novel-anomaly counts are described inconsistently: Section 3.1 explicitly lists 10 types, while other passages say nearly half or more than half; multi-anomaly video counts also conflict at 35/36.
- The VAU similarity threshold, complete data splits, and model training schedule cannot be recovered from this cache, limiting exact reproducibility; supplementary materials or an official implementation require further verification.
- Independent human semantic checks, category-wise transfer reporting, and controlled ablations of storyline segmentation, final-frame conditioning, and the Refiner would be useful next steps; these are reader suggestions, not completed experiments in the paper.
Related Work & Insights¶
- Compared with UCF-Crime and XD-Violence: real datasets provide real imaging distributions, while Pistachio allows proactive event-coverage control; the relationship is more complementary than substitutive.
- Compared with NWPU Campus and UBnormal: Pistachio extends generation control to more scenes and long-form narratives but still faces generative artifacts and the need for real-domain validation.
- Compared with Holmes-VAU/HIVAU-70k: the paper adopts hierarchical event-level and video-level understanding evaluation, connecting storyline-driven synthesis to annotations at those granularities.
- Research direction: hold scenes and normal activities fixed while changing anomaly events or photographic styles to build paired tests that separate semantic recognition from domain bias; the paper does not validate this proposal.
Rating¶
- Novelty: 4/5. Combines controllable long-form video generation, richer normal behavior, and dual-task annotation in an anomaly benchmark.
- Experimental Thoroughness: 3/5. Covers multiple methods, real domains, and cross-generator tests, but lacks component ablations and some protocol details.
- Writing Quality: 3/5. The pipeline and scope are reasonably clear, but some counts, narrative conclusions, and table values disagree.
- Value: 4/5. Useful as a complementary resource for broad anomaly coverage and real-domain adaptation, not standalone evidence of deployment reliability.