Bounding-Box Trajectories Matter for Video Anomaly Detection

Conference: ECCV2026
Authors: Inpyo Song and Jangwon Lee (Sungkyunkwan University)
Paper: Official paper page · PDF
Project: TrajVAD
Area: Video Understanding
Keywords: Video anomaly detection, multi-class tracking, bounding-box trajectories, normalizing flows, pose reliability gating

TL;DR

TrajVAD promotes bounding-box trajectories already produced by detection and tracking to the primary anomaly signal, learning normal motion with a class-aware normalizing flow; its trajectory-only variant reaches 87.7 AP on ShanghaiTech, while reliability-gated pose fusion reaches 88.6 AUROC and 90.9 AP, although pose does not help on every dataset.

Background & Motivation

Video anomaly detection commonly trains on normal videos and asks whether test events depart from the learned distribution. RGB reconstruction or prediction can mistake lighting, texture, and background changes for anomalies. Human-pose methods remove much of this appearance variation and model joint motion directly, making them effective on benchmarks dominated by running or fighting. However, pose estimation requires sufficiently visible, well-resolved people: distant views and occlusion corrupt keypoints, while vehicles and other non-human objects have no skeleton that these models can score.

There is a specific information-use problem here. Before a pose pipeline extracts skeletons, detection and tracking have already produced object centers, dimensions, confidence scores, and identities over time. Existing systems often use these boxes only for identity association, or retain just person-center trajectories. This discards scale, shape, and motion dynamics while restricting coverage to people. The paper is not the first study of trajectory anomaly detection; it systematically tests whether full multi-class bounding-box dynamics can serve as a standalone modality on modern benchmarks.

Core idea: establish normality from bounding-box motion statistics shared across detectable classes, then add joint information only for human tracks with valid, reliable poses, making pose an optional refinement rather than the mandatory entry point for anomaly detection.

Method

Overall Architecture

The system takes video, uses YOLOX, OSNet, and ByteTrack to produce multi-class object tracks, and applies smoothing and short-gap interpolation before dividing tracks into fixed-length overlapping windows. Each frame's 27-dimensional motion descriptor is concatenated with a class embedding and passed through a trajectory normalizing flow. Low likelihood under the learned normal distribution yields a high anomaly score. TrajVAD-T stops here and does not run pose estimation.

TrajVAD-P additionally extracts AlphaPose keypoints for human windows, conditions a pose flow on the trajectory latent, and combines the two likelihoods through a gate based on class, validity, and keypoint confidence. The variants share the representation and scoring principle, but use different trajectory-flow depths in the main experiments, so their difference cannot be interpreted as adding pose alone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    video["Video and multi-class<br/>detection and tracking"] --> features["Class-aware<br/>trajectory representation"]
    features --> trajectory["Trajectory<br/>normalizing flow"]
    video -->|Human windows| pose["AlphaPose keypoints<br/>and confidence"]
    trajectory -->|Trajectory latent| fusion["Reliability-gated<br/>pose fusion"]
    pose --> fusion
    trajectory -->|TrajVAD-T| score["Window anomaly score"]
    fusion -->|TrajVAD-P| score

Key Designs

1. Class-aware trajectory representation: extracting more than center locations from boxes

Center coordinates and dimensions are first normalized to \([0,1]\) by frame resolution. The temporal sequence then produces six groups totaling 27 features. The 6 state features describe center, dimensions, area, and aspect ratio. The 10 temporal-dynamics features describe velocity components and magnitude, acceleration components and magnitude, sine and cosine of direction, jerk, and curvature. The 2 geometric-dynamics features capture box expansion and aspect-ratio change. Another 2 pseudo-physical quantities capture area times squared speed as a kinetic-energy proxy and path efficiency. The 6 perspective-normalized features divide speed and acceleration by box height, width, and area. The final scalar is detector confidence. Together, these describe where an object is, how it moves, how its shape changes, and how trustworthy the measurement is, without requiring any internal joint structure.

This does not recover physical three-dimensional velocity or mass: area times squared speed is an image-space proxy, and box-size normalization only mitigates distance effects. The same pixel displacement means different things for a small distant target and a large nearby one, motivating scale-related features. Features are standardized using training-split statistics. A learned 3-dimensional class embedding covering the 80 COCO classes is appended at every frame, allowing a shared flow to distinguish category-specific normality. The complete input has \(27+3=30\) dimensions per frame, rather than requiring a separate model per category. Exact feature definitions are assigned to supplementary material that is absent from the cache, so discrete formulas for jerk, curvature, and path efficiency are not invented here.

2. Trajectory normalizing flow: measuring normal motion with a tractable density

Feeding handcrafted features into a conventional classifier would still require anomaly labels; the paper instead learns a density from normal data. ActNorm initializes channel-wise affine parameters from the first training batch. Alternating affine coupling layers then split channels into two halves, leave one half unchanged, and invertibly transform the other using predicted scale and translation. The prediction subnet uses causal one-dimensional temporal convolutions with increasing dilation to connect changes within a window. Alternating the transformed half allows all channels to interact. Invertibility matters because it permits evaluation of both the Gaussian latent density and the change-of-volume correction, rather than mistaking latent-space distance for the complete likelihood.

Let \(\ell_{\mathrm{traj}}\) denote the unnormalized log-likelihood, with \(D=27\) trajectory features and \(E=3\) embedding dimensions. The surrounding prose supports the following formulation:

\[ \ell_{\mathrm{traj}}=\log p(z_{\mathrm{traj}})+\log\left|\det J_{\mathrm{traj}}\right|, \qquad \mathcal L_{\mathrm{traj}}=-\frac{\ell_{\mathrm{traj}}}{T(D+E)}. \]

The Gaussian prior is \(p(z_{\mathrm{traj}})=\mathcal N(z_{\mathrm{traj}};\mu_0\mathbf 1,I)\) with fixed offset \(\mu_0=3\), and the Jacobian term accumulates contributions from ActNorm and all coupling layers. Training minimizes negative log-likelihood normalized by the number of window elements; testing uses the same quantity as the score. An anomaly is therefore an unlikely trajectory relative to learned normal motion, not a predefined action class. Cached Equations (2) and (3) are incompletely extracted, so the precise coupling-update order and numerical stability constant are not guessed here. The equation above follows the textual definitions of Equations (4) and (5).

3. Reliability-gated pose fusion: adding joints without requiring skeletons for every object

Boxes capture global translation and scale changes but cannot reliably distinguish different body configurations along the same path. TrajVAD-P therefore adds an independent pose flow for human tracks, using 17 aligned joint coordinates and their confidence scores per window. Coordinates enter the pose branch with its own ActNorm and coupling layers. The trajectory latent conditions the scale and translation subnets in every pose coupling layer, so pose normality depends on overall motion. Keypoint confidence controls the strength of this evidence instead of blindly trusting the estimated skeleton. Conditioning and reliability gating serve different purposes: the former interprets body configuration in its motion context, whereas the latter suppresses unreliable measurements.

The gate is \(g=g_{\mathrm{cls}}g_{\mathrm{valid}}\bar q\), where the first two factors indicate a human class and an available valid pose, and \(\bar q\in[0,1]\) is mean keypoint confidence. Non-human or invalid-pose windows have \(g=0\); lower confidence weakens the pose contribution. Equation (8) combines the unnormalized likelihoods using effective dimensionality:

\[ \mathcal L_{\mathrm{total}} =-\frac{\ell_{\mathrm{traj}}+\lambda g\,\ell_{\mathrm{pose}}} {T\left(D_{\mathrm{traj}}+gD_{\mathrm{pose}}\right)}. \]

Here \(D_{\mathrm{traj}}\) includes the trajectory class embedding, \(D_{\mathrm{pose}}\) is the pose feature dimension, and \(\lambda\) weights pose likelihood. The denominator changes with the gate to keep score magnitudes comparable when pose is enabled or disabled. At \(g=0\), the expression reduces to trajectory-only scoring. This is a reduction of the scoring formula, not a claim that separately trained T and P models with different flow depths must produce identical scores for every sample. The main text does not provide the value of \(\lambda\) or a pose-validity threshold, so neither is fabricated. Gating also does not guarantee an accuracy improvement: the MSAD results show that additional human-pose evidence can be unhelpful.

A Worked Example

Consider the paper's qualitative example of a vehicle entering a pedestrian area. The detector produces a vehicle class and successive bounding boxes, and the tracker maintains an object identity. Under the ShanghaiTech configuration, a 16-frame window becomes 27 motion features plus a 3-dimensional class embedding per frame. The trajectory flow evaluates whether the vehicle's motion and scale changes resemble the normal training distribution. Because the vehicle is not a person, the pose gate is zero. Both variants provide evidence through their trajectory paths, whereas a skeleton-only method has no vehicle score. No window coordinates or numerical likelihoods are invented for this illustration.

For a person whose legs are occluded, the bounding-box track may remain usable while lower pose confidence weakens the joint branch. Short detection gaps are first handled by interpolation; long gaps split the track instead of forcing velocity calculations across extended occlusion. Thus the representation and missing-data preprocessing must be understood together: missed-detection robustness cannot be attributed entirely to the normalizing flow.

Loss & Training

Training uses only normal data and minimizes the negative log-likelihood above. YOLOX performs detection, OSNet supplies re-identification information, and ByteTrack supports multi-class tracking. A Kalman filter smooths tracked coordinates. Gaps of at most 10 consecutive frames are linearly interpolated; longer gaps split the track. Sequences shorter than a full window are discarded. Window lengths are \(T=12,16,24\) for UBnormal, ShanghaiTech, and MSAD respectively. Besides setting the local temporal scale, this means that very short tracks may never enter the scoring model.

The main experiments use \(K=6\) trajectory coupling layers for TrajVAD-T and \(K=18\) for TrajVAD-P. The text introduces an independent pose depth \(K_p\), but the cache does not provide its explicit value. The optimizer, learning rate, training duration, and timing hardware are also not fully specified in the readable main text and are not filled in from convention. Although the coupling subnet uses causal convolutions, interpolation and window formation may require waiting, so causality alone does not establish zero-latency operation for the complete pipeline.

Key Experimental Results

Main Results

All results below are percentages. The authors concatenate frame-level scores across test videos and compute micro-averaged AUROC and AP, rather than averaging per-video metrics. ShanghaiTech contains 330 normal training videos, 107 test videos, and 13 fixed cameras. UBnormal is synthetic, with 29 scenes and 268, 64, and 211 training, validation, and test videos. MSAD contains 720 videos and 55 anomaly categories. HR means Human-Related, not that the input contains only people: the 7 MSAD-HR categories include vehicle-involving events such as traffic accidents.

Dataset	Method	AUROC	AP
ShanghaiTech	STG-NF	85.9	77.6
ShanghaiTech	SeeKer	85.5	80.0
ShanghaiTech	TrajVAD-T	84.9	87.7
ShanghaiTech	TrajVAD-P	88.6	90.9
UBnormal	STG-NF	71.8	62.7
UBnormal	SeeKer	77.9	80.3
UBnormal	TrajVAD-T	68.0	63.2
UBnormal	TrajVAD-P	73.8	68.3
MSAD-HR	STG-NF	55.7	56.5
MSAD-HR	SeeKer	61.1	60.1
MSAD-HR	TrajVAD-T	69.7	60.4
MSAD-HR	TrajVAD-P	68.5	58.5
MSAD	STG-NF	53.8	37.5
MSAD	TrajVAD-T	57.2	42.7
MSAD	TrajVAD-P	55.5	41.5

These are selected rows from Tables 2, 3, and 4. On ShanghaiTech, T trails STG-NF by 1.0 AUROC point but exceeds it by 10.1 AP points, so any claim of improvement must identify the metric. P leads the compared methods on both metrics there. On UBnormal, P improves over T by 5.8 AUROC points but still trails SeeKer by 4.1 points. On MSAD-HR, T exceeds SeeKer by 8.6 AUROC points but only 0.3 AP points; the large AUROC margin is not a large improvement on every metric.

Ablation Study

The following columns come from Table 6 for the full ShanghaiTech set. Dimensionality counts box features only, excluding the 3-dimensional class embedding. Parentheses give AUROC percentage-point changes relative to the full feature set.

Removed feature group	Remaining dimensions	TrajVAD-T AUROC	TrajVAD-P AUROC
None	27	84.9	88.6
State	21	84.6 (-0.3)	87.5 (-1.1)
Temporal dynamics	17	83.9 (-1.0)	88.2 (-0.4)
Geometric dynamics	25	84.0 (-0.9)	88.0 (-0.6)
Pseudo-physical	25	84.1 (-0.8)	88.0 (-0.6)
Perspective-normalized	21	83.4 (-1.5)	88.6 (+0.0)
Detector confidence	26	82.5 (-2.4)	85.9 (-2.7)

Removing the single confidence scalar hurts more than removing any other feature group. The model therefore uses more than pure geometric motion: it also exploits the detector's response to occlusion, truncation, and appearance. Perspective normalization matters for T but provides no additional benefit for P; the authors attribute this to pelvis normalization in the pose branch already encoding body-relative scale. This ablation was conducted on ShanghaiTech, so the -2.4-point result should not be presented as a directly measured confidence ablation on UBnormal.

Key Findings

Table 5 separates end-to-end cost. Detection, tracking, and pose estimation are measured per frame, whereas inference is measured per window. The authors combine them under stride-1 sliding-window evaluation, where a new window is produced for each frame; this accounting should not be generalized to arbitrary batching settings.

Method	Detection and tracking ms/frame	Pose ms/frame	Inference ms/window	Total ms/window
STG-NF	105.8	31.0	8.8	145.6
MoCoDAD	105.8	31.0	2857.0	2993.8
TrajVAD-T	104.4	Not used	2.8	107.2
TrajVAD-P	104.4	31.0	8.6	144.0

T saves time primarily by removing pose estimation, not by making detection free. Its 107.2 ms total does not support an unqualified high-frame-rate real-time claim. P is only slightly faster overall than STG-NF, so the trajectory-only efficiency advantage should not be assigned wholesale to the fused variant.

In Table 7, randomly deleting 10% of each track's detections reduces T from 84.9 to 82.4 AUROC, a 2.5-point loss. STG-NF falls from 85.9 to 80.7, losing 5.2 points, while TSGAD falls from 80.6 to 76.1, losing 4.5 points. This supports robustness to short detection gaps, not comprehensive robustness to sustained occlusion or identity switches. Figure 4 sweeps \(K=2\) through \(22\): T ranges from 83.8 to 84.9 AUROC on ShanghaiTech and peaks at \(K=6\), while P peaks at \(K=18\). The cached curves do not clearly expose every point, so no complete numerical series is reconstructed.

Highlights & Insights

• A low-cost representation need not be a weak representation. Detection boxes already encode scale, shape, and temporal motion, so using them only for cropping wastes information. The paper's value lies in treating this intermediate output as an independently testable modality.
• Anomaly coverage matters as much as anomaly modeling. A human-only model cannot provide vehicle evidence when vehicles never enter its representation. The MSAD-HR gains suggest that covering the anomalous entity can matter more than improving a network restricted to one entity type.
• Reliability belongs in score fusion. The gate changes both pose weighting and effective dimensionality, limiting score-scale changes when a modality is absent. The transferable idea is this treatment of unreliable or missing modalities, not an assumption that extra modalities always improve accuracy.

Limitations & Future Work

• Author-stated dependence on detector quality. Synthetic UBnormal imagery causes class flips and low, noisy confidence scores; missed detections, unstable identities, and miscalibrated confidence enter the trajectory features. Confidence calibration and track-uncertainty modeling deserve evaluation; random-dropout robustness does not solve every upstream failure.
• Author-stated class and event coverage limits. The 80 COCO classes offer broader coverage than human skeletons, but fire, explosions, or out-of-vocabulary objects may not yield usable tracks. T drops from 69.7 AUROC on MSAD-HR to 57.2 on full MSAD, a 12.5-point gap reflecting harder event coverage. This is not a controlled same-distribution ablation, so the entire gap cannot be causally assigned to the class vocabulary alone.
• Note analysis: fusion needs finer controls. T and P use different flow depths, so the main results do not isolate added capacity, trajectory conditioning, gating, and pose information. The readable main text does not include separate no-gate or no-conditioning comparisons, or multi-seed confidence intervals.
• Note analysis: geometric tracks lack semantics and complete reproduction details. Similar motion can correspond to normal carrying or theft, and camera motion beyond fixed-camera settings can contaminate image-space trajectories. The cache does not fully specify window-to-frame score aggregation, supplementary feature definitions, or all training hyperparameters. This note is therefore not a complete reproduction recipe; open-vocabulary detection and global semantic evidence remain directions to test.

Related Work & Insights

• vs STG-NF / SeeKer: They primarily score human poses, while TrajVAD starts from multi-class boxes. ShanghaiTech and MSAD support that choice, but SeeKer is stronger on UBnormal, showing that upstream detector domain shifts can still change the ranking.
• vs TSGAD / TrajREC: Earlier methods already use human trajectories or temporal reconstruction, so this is not the first use of trajectories. Its distinction is integrating multi-class full-box dynamics, scale, and detector confidence into standalone density-modeling inputs.
• vs RGB / optical-flow methods: Trajectories remove much background appearance and associate evidence with objects, at the cost of missing events without trackable foreground entities. A useful extension would test complementary object-trajectory and global-event evidence rather than replace all visual information with tracks.

Rating

• Novelty: 4/5. Trajectory detection and normalizing flows are established, but full multi-class box dynamics with optional pose form a clearly motivated combination.
• Experimental Thoroughness: 4/5. Three benchmarks, feature ablations, missed detections, depth, and runtime are covered; some fusion-specific controls and statistical variation remain unreported.
• Writing Quality: 4/5. The operating regimes and negative results of the two variants are clear; cached equations contain extraction damage and several implementation details require the supplement.
• Value: 4/5. Useful for systems already running detection and tracking, with benefits depending on object coverage, detector quality, and the semantic requirements of the anomaly.