A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

Conference: ECCV 2026
Paper: ECCV official page
PDF: Full paper
Project: CrashTwin
Code: https://github.com/phai-lab/CrashTwin
Area: Autonomous Driving
Keywords: World models, physical consistency, multi-vehicle dynamics, calibration-free reconstruction, video generation evaluation

TL;DR

CrashTwin combines traffic-interaction data, calibration-free reconstruction, and separate physical diagnostics to show that visually coherent world-model videos can still contain incorrect momentum exchange and kinetic-energy increases; its contribution is an evaluation framework, not a new video generator.

Background & Motivation

Autonomous-driving world models could supply rare interactions that are difficult to collect at scale, expanding the coverage of training and evaluation data. However, a convincing video is not necessarily a valid simulation: vehicles may preserve their texture and identity while their motion changes in ways unsupported by the interaction. FVD, image-quality scores, and general video benchmarks primarily assess distributions or visible appearance rather than directly measuring momentum transfer. The paper therefore uses multi-vehicle collisions as a stress test, where brief contact couples the participants' states and provides more specific constraints than a generic judgment of plausibility.

The main obstacle is observability rather than a lack of physical laws. Generated videos usually provide neither camera calibration nor metric depth or ground-truth velocities; camera motion is mixed with the motion of other vehicles, while occlusion and tracking failures concentrate around contact. General video collections do not necessarily cover these interactions, and generators conditioned on complete future trajectories may receive the dynamics that an evaluator is supposed to test. CrashTwin addresses these gaps through controlled simulation, curated real videos, and a reconstruction pipeline built from existing vision models. Core idea: make the physical state in generated video measurable, then separately assess geometry, instance integrity, and pre-/post-contact conservation residuals instead of treating visual quality as evidence of correct physics.

Method

Overall Architecture

The evaluation inputs include video examples with first-frame actor annotations and textual descriptions, together with the rollouts produced by the world models being tested. CrashTwin estimates the participating vehicles' three-dimensional trajectories, orientations, and motion, and reports seven grouped metrics rather than one aggregate score that could conceal different failures. Its main elements are scenario and input constraints, calibration-free dynamics reconstruction, and parallel checks for geometry and identity and for collision conservation. The first diagnostic branch uses images, optical flow, tracking identities, and appearance features; the second uses reconstructed states over a short contact window.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Scenario and input constraints"] --> B["World-model video generation"]
    B --> C["Calibration-free<br/>dynamics reconstruction"]
    C --> D["Geometry and identity diagnostics"]
    C --> E["Collision conservation diagnostics"]
    D --> F["Separate diagnostic scores"]
    E --> F

Key Designs

1. Scenario and input constraints: evaluate dynamics rather than replayed trajectory conditions

The synthetic portion uses CARLA to instantiate seven representative intersection interaction types, varying initial positions, velocities, orientations, encounter timing, and contact geometry. It records vehicle states, three-dimensional boxes, segmentation, depth, and optical flow alongside the rendered video. These 30 Hz records support comparisons between recovered trajectories and known states. The real portion consists of curated public traffic videos, segmented around contact and captioned with vehicle motion, environmental context, and timing. Table 1 reports 38,349 clips and 7.04M frames overall; the approximate 25K synthetic and 12K real figures in the prose should not be used to infer exact subset counts.

The protocol fixes the two actors to evaluate before measuring their behavior. Synthetic clips use simulator identities, whereas real clips require manual actor annotations released with the test data. This reduces the chance that incorrect participant selection is attributed to the generator, but means the evaluation is not entirely annotation-free. First-frame context and descriptions specify the initial situation and intended event without supplying a complete future trajectory. This distinction matters because a system given all future boxes can reproduce externally specified motion without predicting the underlying dynamics itself. The paper consequently distinguishes unconstrained world-model evaluation from specialized trajectory-conditioned generation, rather than claiming that all generation settings share a fair leaderboard.

CrashTwin-Eval contains 300 synthetic and 44 real held-out videos. For proprietary systems with access or inference-cost constraints, CrashTwin-Eval-Mini contains a random sample of 100 synthetic and 16 real videos. Both are drawn from the test split, while the remaining data support training. Since the two model groups use different evaluation samples, their reported scores should not be combined into a single directly comparable ranking.

2. Calibration-free dynamics reconstruction: separate vehicle motion from camera motion

MapAnything estimates camera intrinsics, CenterTrack supplies initial three-dimensional track fragments, and SAM2 video instance segmentation supports reconnecting fragments. Relinking is essential because an occluded vehicle that reappears as a new track can otherwise introduce discontinuities that look like abrupt physical motion. Metric3D V2 supplies metric-depth correction to improve geometry and restore scale, producing relative trajectories in camera coordinates. Here, calibration-free means that users need not provide calibration beforehand; it does not mean that the system avoids estimating camera parameters or scale.

DROID-SLAM estimates ego-motion, with metric-scale correction, so that relative actor trajectories can be expressed in a common global coordinate system. Without this step, camera translation or rotation would contaminate the measured velocity changes of other vehicles. A Kalman filter smooths positions and orientations to stabilize the resulting dynamics. These are components of the evaluator, not newly inserted modules inside each world model. The first frame at which the distance between the selected vehicles falls below a predefined safety threshold establishes the contact window, including short pre-contact and post-contact segments.

Some generated sequences never produce the intended contact. For these cases, the protocol assigns a fixed penalty to collision-based momentum and energy metrics, avoiding unstable calculations and preventing a missing event from appearing physically successful. The cached main paper does not specify that penalty, the distance threshold, the window length, or all implementation parameters. Its pipeline description therefore explains the measurement logic but is not a complete parameter-level reproduction recipe; those details require the referenced supplementary material or implementation.

3. Geometry and identity diagnostics: distinguish visual continuity from persistent objects

Temporal consistency is measured by flow-warping error, \(E_{\mathrm{warp}}\). Pixels within a vehicle mask are transported to the next frame using optical flow, and their appearance discrepancies are averaged. A low error indicates coherent propagation of appearance, not physically correct changes in velocity. Spatial consistency uses flow divergence, \(E_{\mathrm{div}}\), to measure local expansion or compression within the foreground region. Lower values indicate fewer apparent deformations, but this remains a projected-image diagnostic rather than a complete test of three-dimensional rigidity.

Instance stability uses the Simpson index. For a vehicle, tracked frames are grouped by assigned identity; if \(f_{k,i}\) is the fraction associated with identity \(k\), the score is:

\[ S_{\mathrm{ID}}^{(i)}=\sum_k f_{k,i}^{2}. \]

A vehicle that retains one identity scores 1, while fragmentation typically lowers the score. This measures concentration of tracking identity, not every possible failure of object permanence. Appearance drift, \(D_{\mathrm{ad}}\), instead uses normalized CLIP features from SAM2-masked vehicle crops. It averages the angles between feature vectors in consecutive visible frames, then forms a video-level average weighted by each instance's frame count. Lower drift indicates smoother appearance evolution. Stable identity and stable appearance are thus distinct properties, and neither establishes conservation of momentum.

4. Collision conservation diagnostics: test state transitions over a short impulse window

The mechanics-based checks assume a sufficiently short contact window that tire friction and aerodynamic drag contribute negligible impulse relative to inter-vehicle contact. Under this approximately closed-system assumption, pre-/post-contact momentum differences become useful indicators of inconsistency. The linear-momentum residual, \(J_p\), compares total momentum vectors before and after contact, where each vehicle contributes its mass times velocity. The norm of the difference is normalized by the sum of pre-contact mass times speed across vehicles. It tests the combined system's momentum balance rather than requiring either vehicle to maintain its individual velocity.

Angular-momentum residual \(J_H\) is evaluated about the contact point. It includes both yaw moment of inertia times yaw rate and the orbital contribution formed by position relative to contact crossed with linear momentum. The norm of the pre-/post-contact difference is divided by the pre-contact angular-momentum norm plus a small stabilizing constant. This tests whether changes in yaw and translational motion are mutually consistent, which simply checking orientation smoothness would miss. Reliable mass, inertia, and contact-point estimates are consequently important, but their acquisition for all real videos is not fully specified in the cached main text.

The energy check does not require an inelastic collision to preserve kinetic energy. Equation 5 uses the sum of translational kinetic energies and penalizes only an increase:

\[ E_k=\frac{1}{2}\sum_i m_i\|v_i\|^2,\qquad J_E=\max\left(0,\frac{E_k^{+}-E_k^{-}}{E_k^{-}}\right). \]

Superscripts \(-\) and \(+\) denote pre-contact and post-contact states. Any non-increasing kinetic energy receives zero penalty, allowing dissipation but also leaving excessive dissipation undetected by this metric alone. Momentum and the other diagnostics must therefore be considered alongside it. The stated energy expression also omits rotational kinetic energy and should not be described as a strict conservation test for total mechanical energy. The metrics are complementary diagnostics, not independent certificates of physical validity.

Loss & Training

The evaluator primarily applies existing perception models to recover states; it does not train a new collision generator. The paper additionally reports post-training on the CrashTwin training split, using physical-consistency objectives for flow-based temporal coherence, spatial rigidity, and short-window momentum and energy residuals. The cached main text does not provide a complete objective, weights, optimizer, training budget, or an account of gradient propagation through all state-extraction components. The results therefore support the usefulness of these signals for adaptation, but should not be expanded into an invented fully specified training algorithm.

Key Experimental Results

Main Results

The following selected columns from Table 2 compare four open-source models on the same CrashTwin-Eval split: 300 synthetic and 44 real held-out videos. These are protocol scores, not success rates; arrows indicate the preferred direction. Proprietary models evaluated on Mini are deliberately excluded from this table.

Model	\(E_{\mathrm{warp}}\) ↓	\(J_p\) ↓	\(J_E\) ↓	\(S_{\mathrm{ID}}\) ↑
Wan 2.1-14B	0.0179	0.8235	0.7864	0.6760
Wan 2.2-5B	0.0145	0.8899	0.8649	0.7254
Cosmos-Predict2-2B	0.0240	0.8890	0.8590	0.6129
Cosmos-Predict2-14B	0.0117	0.6828	0.6047	0.6737

Wan 2.2-5B improves warping error and identity stability relative to Wan 2.1-14B, yet has worse momentum and energy residuals. This directly illustrates why one dimension cannot substitute for another. The two Cosmos variants also differ across dimensions, but this is not a controlled parameter-scaling experiment and does not establish that model size alone resolves physical inconsistency.

Ablation Study

Table 3 incrementally adds reconstruction components on simulated collisions and reports three-dimensional absolute trajectory error, ATE, in meters; lower is better. The selected columns show scene-level SE(3), instance-level SE(3), and instance-level Sim(3) results. SE(3) alignment retains scale error, whereas Sim(3) permits a global scale adjustment, so these accuracies must not be conflated.

Cumulative configuration	Scene ATE, SE(3) ↓	Instance ATE, SE(3) ↓	Instance ATE, Sim(3) ↓
Basic 3D tracking	11.71	3.89	1.98
Add instance relinking	5.96	3.73	1.89
Also add Kalman filtering	5.61	3.29	1.47
Also add metric-depth correction	5.48	2.63	0.91

The sub-meter result is specifically the final instance-level Sim(3) ATE of 0.91 m; final instance-level SE(3) ATE remains 2.63 m. This ablation validates reconstruction quality, not the effect of removing components from a video generator.

Key Findings

• In Table 4, post-training reduces the baseline's \(J_p\) from 0.8890 to 0.6479 and \(J_E\) from 0.8590 to 0.5534. These improvements do not imply that conservation is solved.
• The same table reports ground-truth reference scores of \(J_p=0.3089\) and \(J_E=0.2502\) after evaluation. Nonzero residuals therefore also reflect measurement error and modeling approximations, not only generator failures.
• Relinking produces the first large reduction in scene-level drift, while filtering and depth correction further improve instance trajectories. Because the ablation is cumulative, it does not fully isolate each component's independent contribution.

Highlights & Insights

• Establishing physical observability before scoring connects each diagnostic to measurable state. Compared with a generic vision-language judgment, the decomposition makes identity, geometry, and dynamics failures easier to distinguish.
• Allowing energy dissipation while penalizing increases better matches inelastic contact than enforcing equal kinetic energy. Its one-sided blind spot is explicit, motivating joint interpretation with momentum metrics.
• Reporting nonzero ground-truth reference residuals is valuable. The evaluator has its own error floor, so generated-video scores need a measurement baseline rather than an assumption that every valid sequence scores zero.

Limitations & Future Work

• The authors acknowledge that automated actor selection is unreliable, requiring manual annotations for real clips. Non-contact outputs also receive special penalties, making boundary-case handling part of the score definition.
• Reader assessment: the short-window approximation, monocular scale, occluded tracking, and smoothing can all affect estimated momentum. Filtering may alter rapid transitions near contact, motivating explicit uncertainty analysis.
• Reader assessment: flow divergence and CLIP continuity remain visual proxies. Legitimate changes in viewpoint, occlusion, or vehicle deformation can affect them; diagnostic evidence is not a proof of physical correctness.
• Evidence boundary: the local cache contains the complete 19-page main paper but not its referenced supplementary material. Mass/inertia assignments, contact thresholds, non-contact penalties, post-training hyperparameters, and human-preference study details cannot be confirmed locally, so unverifiable details and preference numbers are omitted.
• Comparison boundary: open-source and proprietary models use different evaluation subsets. A unified ranking would require rerunning them on matched samples and generation conditions.

Related Work & Insights

• Compared with VBench / FVD: VBench covers visible video-quality dimensions, while FVD assesses distributional differences. CrashTwin adds contact-mechanics residuals from estimated states, complementing rather than replacing visual evaluation.
• Compared with VideoPhy-style physical commonsense evaluation: CrashTwin focuses on measurable three-dimensional motion and short-window conservation. This offers more specific diagnosis but introduces reconstruction error and parameter assumptions.
• Compared with Ctrl-Crash-style trajectory-conditioned generation: complete temporal boxes can determine future motion externally. CrashTwin primarily probes evolution not already prescribed by those trajectories, so the evaluation targets and difficulty differ.

Rating

• Novelty: 4/5. A focused combination of traffic interactions, state recovery, and grouped physical diagnostics.
• Experimental Thoroughness: 3/5. Multiple models, reconstruction ablations, and post-training analysis are useful, but different evaluation subsets and measurement uncertainty limit conclusions.
• Writing Quality: 3/5. The motivation and diagnostic structure are clear, while important implementation details depend on supplementary material unavailable in the local cache.
• Value: 4/5. Useful for investigating physical failures in autonomous-driving world models, but no individual score should serve as simulation certification.