O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Project: https://o-vad.github.io/
Area: Video Understanding
Keywords: Industrial Video Anomaly Detection, Object-Centric Tracking, Training-Free Agent, Vision-Language Models, Chain-of-Thought Reasoning

TL;DR

Addressing the failure of holistic vision-language models to capture subtle physical transformations and process violations in industrial video anomaly detection, O-VAD proposes a training-free agentic framework that tracks per-object spatiotemporal state evolution and conducts a six-step cognitive reasoning process, achieving state-of-the-art anomaly localization and root-cause analysis without domain-specific fine-tuning.

Background & Motivation

Industrial Video Anomaly Detection (IVAD) is essential for modern automated manufacturing and quality assurance, aiming to flag irregular objects and operational failures across continuous video feeds. Unlike typical surveillance or action recognition setups, industrial environments exhibit three defining characteristics: first, workpieces undergo substantial, legitimate physical deformations through pressing, cutting, bending, and assembly; second, operations are governed by strict spatial, kinematic, and physical constraints where slight misalignments signal severe process breakdowns; and third, on-site operators require transparent, causal explanations for diagnostic action rather than opaque scalar anomaly scores.

Traditional video anomaly detection methods predominantly follow the "one-class-one-model" paradigm, relying either on reconstruction error or feature embedding deviations trained solely on extensive collections of normal samples. This setup requires retraining for every new product line and fails to generalize across flexible production lines, while providing zero semantic rationale for detected defects. While recent frontier vision-language models (VLMs) have demonstrated zero-shot open-ended reasoning in natural domains, their direct deployment to industrial inspection suffers severe degradation. Because they evaluate videos through whole-frame features or coarse temporal captions, they lack localized physical representations of individual workpieces over time, frequently falling prey to a strong "normal" bias that misinterprets destructive fractures or leaks as expected mechanical actions.

Rather than relying on tedious manual prompt engineering, external technical manuals, or costly domain-specific fine-tuning, O-VAD mirrors the cognitive inspection process of human quality auditors by shifting the foundational analytical unit from whole video frames to individual objects across time. Core idea: construct a training-free "ground→track→reason" agentic framework that establishes spatiotemporal object tubelets, recovers broken tracks via physical and semantic priors to log open-ended state changes, and applies a six-step cognitive chain-of-thought with post-visual verification for grounded anomaly diagnosis.

Method

Overall Architecture

Given an industrial input video \(V = \{I_t\}_{t=1}^T\), O-VAD decomposes inspection into three distinct, interpretable stages: Stage 1 inventories and segments all candidate task-relevant objects across multiple candidate frames; Stage 2 constructs spatiotemporal tubelets, maintains continuity through severe object transformations via spatial proximity and semantic consistency, and generates structured state change tuples; Stage 3 executes a six-step cascaded chain-of-thought reasoning process to untangle expected process responses from actual defects, followed by a multi-tiered post-verification gating scheme to filter false alarms.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Industrial Video<br/>V = {I_t}"] --> B["Automated Object Grounding<br/>Multi-frame inventory + SAM3"]
    B --> C["Object-Centric State Tracking<br/>CropFormer + SAM2 tubelet build"]
    C --> D["Track Continuation Recovery<br/>Spatial proximity + Semantic consistency"]
    D --> E["Open-Ended State Change Detection<br/>Extract state event tuple e"]
    E --> F["Cognitive Anomaly Reasoning<br/>6-step CoT: Process vs Failure"]
    F --> G["Post Visual Verification<br/>Multiplicative confidence gating"]
    G --> H["Structured Anomaly Report<br/>Type / Object / Frame / Causal analysis"]

Key Designs

1. Automated Object Grounding: eliminating single-frame blind spots Industrial operations often introduce workpieces that are occluded, absent, or shadowed during the initial frame. To avoid missing critical equipment, O-VAD samples candidate frames \(\{I_{t_j}\}_{j=1}^{N_f}\) across the video. A VLM inventories all distinct entities per frame, producing structured metadata covering object names \(n_k\), physical attributes and initial condition \(d_k\), and bounding hints \(b_k\). Merged and deduplicated across time into a global asset inventory \(\mathcal{O}\), each object \(o_k\) is segmented on a clean reference frame using SAM3 with concept prompts, yielding robust initial masks \(\mathcal{M}_1 = \{m_k^{(1)}\}_{k=1}^{|\mathcal{O}|}\) to seed temporal tracking.

2. Transformation-Aware Spatiotemporal Tubelet Tracking: preserving identity across deformation and fragmentation Under mechanical forces such as cutting, crushing, or burst leaks, objects undergo dramatic topological alterations that defeat standard multi-object trackers. O-VAD employs CropFormer to segment per-frame entities \(E_t = \text{CF}(I_t)\) and propagates initial masks forward with SAM2 into spatiotemporal tubelets \(\mathcal{P}_{\text{init}}\). Whenever emerging image regions remain uncovered by existing tubelets (below coverage threshold \(\tau_{\text{coverage}}\)), new candidate tubelets are initiated. To link fragments back to their source workpiece, candidate track \(C\) starting at frame \(s\) is evaluated against prompt track \(P\) using dual physical criteria: First, initial frame spatial overlap via SAM2 multi-mask outputs: $\(S_{\text{prox}}(C, P) = \max_{j \in \{1,2,3\}} \frac{|c_s \cap m_s^j|}{|c_s|}\)$ Second, deep masked CLIP feature cosine similarity evaluating material and appearance consistency: $\(S_{\text{sem}}(C, P) = \max_{i < s,\, j \ge s} f(p_i, I_i) \cdot f(c_j, I_j)^\top\)$ Candidates satisfying both thresholds constitute the valid continuation set \(\mathcal{V}\), capturing the workpiece continuously from pre-stress through catastrophic structural failure.

3. Open-Ended State Change Detection: structuring physical events in natural language Across key transition frame pairs, the VLM receives masked visual crops and annotates physical state dynamics. Critically, instead of forcing observations into a restricted dictionary, the system extracts free-form natural language capturing deformation, material release, surface abrasion, and relative mechanics. Each verified transition is parameterized as a structured event tuple: $\(e = (t_{\text{start}},\, t_{\text{end}},\, \textit{type},\, \textit{cause},\, \textit{desc},\, \textit{sev},\, k)\)$ specifying the start and end frame indices, free-form change type, attributed causal factor, detailed narrative, severity rating \(\textit{sev} \in \{\text{none}, \text{slight}, \text{moderate}, \text{severe}\}\), and the affected object index \(k\), converting raw pixels into an auditable evidence trajectory.

4. Cognitive Anomaly Reasoning with Post Visual Verification: disentangling operational actions from failure Equipped with complete state trajectories and global video context, the VLM executes an inspection-inspired six-step chain-of-thought: - Process Understanding: clarifies the overall manufacturing step and triages events into expected baseline actions, potential anomalies, and sensor noise; - Observation: references exact object indices, frame windows, and state shifts flagged as suspicious; - Expectation: articulates explicit physical pass/fail thresholds (e.g., elastic bending is permissible, permanent wall rupture is not); - Comparison: decides whether observed deformations match expected machine tolerances or represent true damage; - Causation: deduces plausible root causes (excessive clamping force, tool misalignment, thread stripping) without taxonomy constraints; - Classification & Severity: assigns open-ended defect descriptors and rates operational risk. To prevent hallucinations on ambiguous visual cues, candidate anomalies with initial confidence \(c_{\text{orig}}(a)\) undergo tiered verification: predictions exceeding \(\tau_{\text{hi}}=0.8\) bypass verification, those below \(\tau_{\text{lo}}=0.2\) are discarded, and intermediate candidates are re-verified against focused evidence frames \(\{I_t\}_{t \in \text{evidence}}\) using multiplicative gating: $\(c_{\text{final}}(a) = \begin{cases} c_{\text{orig}}(a) \cdot c_{\text{ver}}, & \text{if verified is true} \\ c_{\text{orig}}(a) \cdot (1 - c_{\text{ver}}), & \text{if verified is false} \end{cases}\)$ Only anomalies maintaining \(c_{\text{final}}(a) \ge \tau_{\text{conf}}=0.3\) enter the final diagnostic report.

A Worked Example

Consider an industrial robotic manipulation trial evaluating plastic water bottle capping and inversion: 1. Object Grounding: the model inventories the primary workpiece \(k=1\) (thin-walled transparent PET bottle, capped, filled with water) and the manipulator (metallic mechanical clamp). 2. State Tracking: across frames 0–60, the clamp closes on the bottle; the tracker registers moderate elastic deformation. At frames 60–70, a localized puncture appears near the contact edge and fluid escapes, triggering a new tubelet tagged as severe material release; by frames 70–80, fluid drainage leaves the container largely collapsed. 3. Cognitive Reasoning: the reasoning engine notes that while gripper-induced elastic indentation is normal during transport, fluid escaping from a supposedly sealed container violates fundamental containment criteria. 4. Causation and Verification: the system isolates stress concentration caused by clamp edge misalignment or excessive gripping torque as the primary root cause. The verifier re-evaluates frames 60–80, confirms fluid loss, and issues a high-severity report for material_anomaly/loss_of_containment_leakage.

Key Experimental Results

Main Results

O-VAD was evaluated against leading traditional baselines, direct-prompted frontier VLMs, and current agentic frameworks across three benchmarks: Phys-AD (physical interaction dynamics), LiquidAD (multi-channel automated pipetting), and IPAD (manufacturing reference deviation).

Dataset	Level	Metric	O-VAD* (Ours)	GPT-5*	S3R† (Trad. Best)	VERA† (Workflow)
Phys-AD	Video	AUROC (%)	58.4	50.2	55.5	45.6
Phys-AD	Video	F1-Score	0.621	0.630	0.689	0.091
Phys-AD	Type	BERTScore	0.803	0.878	—	—
Phys-AD	Type	LLM-judge	0.595	0.580	—	—
LiquidAD	Video	AUROC (%)	69.2	46.5	65.1	53.4
LiquidAD	Video	Recall	0.948	0.462	0.616	0.000
LiquidAD	Frame	F1-Score	0.507	0.176	0.679	0.000
IPAD	Video	AUROC (%)	56.5	51.9	53.9	51.9
IPAD	Frame	Recall	0.477	0.119	0.166	0.000

Note: * indicates training-free methods; † indicates methods trained or fine-tuned on target datasets.

On the 22 object categories in Phys-AD, O-VAD ranked first or second in AUROC on 16 categories, achieving outstanding performance on complex multi-step mechanical assemblies: Gear (87.9%), Lock (78.0%), Screw (70.4%), and Sticky Roller (81.1%), establishing the highest overall training-free video AUROC of 58.4%.

Ablation Study

A component-wise ablation was performed across four representative categories from Phys-AD covering distinct defect profiles: Sticky Roller (continuous surface defects), Liquid (fluid dispensing/overflow), Screw (localized fastener assembly), and Rubber Band (elastic detachment).

Config	Sticky Roller AUC	Liquid AUC	Screw AUC	Rubber Band AUC	Notes & Observations
O-VAD (Full Model)	0.811	0.776	0.704	0.721	Highest balanced sensitivity and localization precision
(a) w/o Caption	0.564	0.742	0.731	0.590	Severe drops on Roller (-0.247) and Rubber Band (-0.131); macro-context is vital for behavioral tracking
(b) w/o State Tracking	0.484	0.667	0.400	0.587	Catastrophic collapse: P/R/F1 plummet to 0.000 on three classes as predictions default to all-normal
(c) w/o CoT Reasoning	0.720	0.738	0.664	0.681	Replacing structured steps with generic prompting drops AUC by 0.038–0.091 and hurts calibration
(d) w/o Post-verifier	0.633	0.718	0.622	0.714	Eliminating verification incurs a 0.178 AUC drop on Roller due to unchecked borderline false alarms

Key Findings

• Object state tracking is indispensable: Removing the tracking module collapses precision, recall, and F1 to zero across three of the four evaluation classes. Without explicit tracking of workpiece boundaries and transformations across time, the VLM simply declares every sequence "normal," confirming that industrial anomaly reasoning fails at evidence collection rather than language reasoning.
• Caption context exhibits scale dependency: High-level video captions provide crucial situational anchors for holistic behaviors (e.g., rubber band slippage or irregular roller sweep), but introduce minor noise for highly localized defects (e.g., fastener cross-threading), where object-level tubelets already contain sufficient signal.
• Hierarchical reasoning and verification ensure diagnostic reliability: Structured six-step chain-of-thought enforces principled physical boundaries between legitimate strain and destructive failure, while multiplicative post-verification suppresses ambiguous false alarms along the decision boundary.

Highlights & Insights

• Decoupling video understanding into physical state trajectories: Rather than treating industrial video as an undifferentiated stream of frames, O-VAD converts the inspection problem into discrete object-centric event graphs, providing natural robustness against complex camera motion and background clutter.
• Surpassing trained models without training: By coordinating foundation models (SAM3, CropFormer, SAM2, GPT-5) in an agentic loop, O-VAD outperforms specialized supervised models such as S3R and MNAD.p on the demanding Phys-AD benchmark.
• Generalizable agentic architecture: The combination of multi-frame discovery, transformation-tolerant tubelet tracking, and confidence-gated verification offers a blueprint for safety-critical inspection in surgical robotics, laboratory automation, and aerospace maintenance.

Limitations & Future Work

• Computational overhead and latency: Running multiple visual segmentation models alongside multi-turn LLM reasoning incurs substantial inference latency, presenting hurdles for millisecond-level high-speed manufacturing lines.
• Inability to detect visually imperceptible defects: O-VAD relies strictly on visible physical manifestations (cracks, leaks, deformations, disconnections); internal electrical faults, latent material stress, or magnetic degaussing that exhibit no surface change remain outside the model's perceptual horizon.
• Future directions: Integrating few-shot in-context learning with technical CAD specifications and engineering drawings, coupled with event-driven adaptive frame sampling, represents a promising path toward real-time deployment.

Related Work & Insights

• vs. Traditional Video Anomaly Detection (e.g., MNAD.p, S3R): Traditional methods require extensive in-domain normal data and output uninterpretable scalar scores; O-VAD is training-free, highly generalizable, and produces grounded causal explanations alongside frame-level masks.
• vs. Direct VLM Prompting (e.g., GPT-5, Qwen3-VL-32B): Unconstrained frame-level prompting lacks local geometric grounding, leading to widespread false negatives where subtle failures are overlooked; O-VAD's per-object state extraction provides the requisite microscopic evidence.
• vs. Existing VAD Agentic Frameworks (e.g., URF-ZS-HVAA, VERA): Prior agents rely heavily on static QA prompting without tracking object deformation across time; O-VAD explicitly resolves object identity through topological transformations and fluid releases.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Transforms industrial video inspection into object-centric physical state tracking and multi-stage agentic reasoning without retraining]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluations across Phys-AD, LiquidAD, and IPAD benchmarks spanning video, frame, and type metrics alongside extensive ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous formulation of physical deformation challenges, and comprehensive qualitative comparisons]
• Value: ⭐⭐⭐⭐⭐ [Delivers an interpretable, highly effective paradigm for industrial quality control and autonomous embodied inspection systems]