Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/fanhong-li/model-human-divergence
Area: LLM Reasoning / Multimodal VLM
Keywords: Physical Reasoning, Human-AI Comparison, Video Understanding, Distributional Evaluation, Intuitive Physics
TL;DR¶
Addressing the question of whether video foundation models matching human-level accuracy genuinely simulate intuitive physics, this paper proposes a population-level distributional evaluation framework comparing model probe seeds against human rater distributions, revealing that models form a coherent yet distinct non-human strategy driven by visible statistical cues rather than explicit forward simulation.
Background & Motivation¶
Predicting unobserved physical dynamics—anticipating how objects slide, collide, topple, or remain stable under partial observation—is a cornerstone of human intelligence that emerges early in infancy. Cognitive science posits an "intuitive physics engine" wherein humans perform approximate, probabilistic forward mental simulations over future trajectories. Recent self-supervised video foundation models such as V-JEPA2, VideoMAEv2, and DINOv2 have posted impressive results on physical reasoning benchmarks, with V-JEPA2 coming within approximately one percentage point of average human accuracy on contact prediction tasks. However, this convergence raises a fundamental question: have these models truly acquired human-like forward simulation of physical dynamics, or are they merely exploiting statistical co-occurrences in the visible scene without tracking unobserved trajectories?
Standard evaluation protocols rely exclusively on aggregate accuracy, contrasting a single model's point estimate against mean human performance. This coarse metric masks profound strategic divergences. Two systems can achieve identical overall scores while failing on completely disjoint sets of stimuli or reversing their relative advantages across physical domains. When a benchmark mixes geometric scenarios (decidable from spatial relationships alone) with long causal-chain scenarios (requiring forward temporal integration), aggregate accuracy conflates these capabilities and creates a false impression of cognitive alignment.
To overcome this blind spot, this paper departs from aggregate scalar evaluation and adopts a distributional perspective drawn from psychometrics. By treating multiple random initializations of linear/attentive probes as a model population and crowdsourced annotators as a human reference population, one can analyze inter-rater consensus, entropy-based uncertainty, and feature-level attribution. Core idea: construct bilateral population distributions from model seeds and human raters to establish a three-level agreement hierarchy (κHH ≫ κMM ≫ κMH), entropy quadrant decomposition, and layered physical feature fingerprinting, demonstrating that models rely on surface-level visual regularities rather than forward causal simulation despite matching human-level accuracy.
Method¶
Overall Architecture¶
The proposed distributional evaluation framework shifts benchmark assessment from isolated point estimates to population-level statistical distributions. Taking physical benchmark videos and their ground-truth physical simulation trajectories (from Physion) as input, the framework processes predictions across \(K=8\) probe initializations on frozen vision encoders and \(\sim 100\) independent human raters per stimulus. The pipeline progresses through three key stages: population representation aggregation, three-level agreement and entropy decomposition, and physical strategy fingerprinting coupled with multi-layer feature attribution.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Data<br/>Physion 1200 test videos + HDF5 simulation states"] --> B["Population Representation<br/>Frozen backbone + 8 probe seeds vs ~100 human raters"]
B --> C["Three-Level Agreement & Uncertainty Decomposition<br/>κHH ≫ κMM ≫ κMH & four-quadrant entropy analysis"]
C --> D["Strategy Fingerprinting & Layered Physical Attribution<br/>37-dim fingerprint cosine similarity + 89-dim A/B/C attribution"]
D --> E["Evaluation Diagnostics<br/>Revealing cue-based heuristic without forward simulation"]
Key Designs¶
1. Population Representation: Converting seeds and raters into consensus distributions
Conventional probing trains a single linear or attentive classification head, conflating probe initialization variance with encoder capability. To ensure a rigorous and fair comparison across V-JEPA2, VideoMAEv2, and DINOv2 (standardized to ViT-Large backbones, \(\sim 307\text{M}\) parameters, 8 input frames with fractional step 5.75), the framework freezes the vision encoders and trains 8 randomly initialized end-to-end attentive probes (depth 4, 16 heads) per backbone. For sample \(i\), the model population consensus is \(p_m^{(i)} = \frac{1}{K}\sum_{k=1}^K M_{ik}\), and the human consensus is \(p_h^{(i)} = \frac{1}{H_i}\sum_{j=1}^{H_i} R_{ij}\), with respective majority-vote decisions \(\hat{y}_m^{(i)}\) and \(\hat{y}_h^{(i)}\) obtained by thresholding at 0.5. This transforms discrete binary labels into continuous consensus distributions, establishing a common mathematical space for fine-grained error consistency analysis.
2. Three-Level Agreement Hierarchy & Uncertainty Decomposition: Quantifying irreducible human-model divergence
To evaluate whether models converge to human cognitive strategies, the framework establishes a three-level Cohen's \(\kappa\) hierarchy evaluated via 1,000 split-half bootstrap iterations. It computes human–human consensus agreement \(\kappa_{HH}\) by splitting raters into two halves, model–model agreement \(\kappa_{MM}\) by splitting probe seeds into two halves, and model–human agreement \(\kappa_{MH}\) between the overall majority votes. If models genuinely simulate human-like reasoning, one expects \(\kappa_{MM} \approx \kappa_{MH} \approx \kappa_{HH}\); conversely, if models form an internally consistent yet non-human solution, the ordering collapses to \(\kappa_{HH} > \kappa_{MM} \gg \kappa_{MH}\). In parallel, binary Shannon entropy \(H(p) = -p\log_2 p - (1-p)\log_2(1-p)\) partitions samples into four quadrants using threshold \(\tau = H(0.25) \approx 0.81\text{ bits}\): - Q1 (both certain, easy stimuli); - Q2 (model uncertain, human certain, model blind spots); - Q3 (model certain, human uncertain, confident divergence); - Q4 (both uncertain, intrinsically ambiguous). The asymmetry ratio \(A = |Q2| / |Q3|\) diagnoses the directionality of information access. An asymmetry of \(A \ll 1\) indicates that models do not possess superior knowledge, but rather make overconfident commitments on ambiguous stimuli where human uncertainty is naturally elevated.
3. Strategy Fingerprinting & Layered Attribution: Uncovering the locus of decision drivers
To uncover the physical mechanisms governing model decisions, the framework extracts full physical state trajectories from the Physion HDF5 simulator files, structured into three temporal layers: Layer A (static scene properties: mass, friction, object count, geometry), Layer B (observable dynamics prior to cutoff: velocities, kinetic energy, collisions), and Layer C (unobservable outcome features: post-cutoff contact timing, causal chain length, prediction horizon). For strategy fingerprinting, a core 37-dimensional subset (6A + 24B + 7C) is regressed onto each population's per-sample correctness vector via Ridge regression, yielding coefficient vector \(\beta \in \mathbb{R}^{37}\) as a behavioral strategy fingerprint. Pairwise cosine similarity between fingerprints quantifies strategy alignment across architectures and humans. For feature attribution, the full 89-dimensional feature library (19A + 47B + 23C) is utilized within a 5-fold cross-validation setup to predict the divergence magnitude \(|p_m - p_h|\), empirically testing whether divergence stems from visible scene encoding or from unobserved future dynamics.
Key Experimental Results¶
Main Results¶
Under the unified ViT-Large attentive probe protocol on Physion, cross-model comparison against human raters yields the following metrics (Table 1 of the paper):
| Model / Population | Seeds | Test Acc. (%) | Train Acc. (%) | Disagreement Rate (%) | \(\kappa\) Agreement |
|---|---|---|---|---|---|
| V-JEPA2 | 8 | 73.16 ± 0.95 | 90.60 ± 0.56 | 26.4 | 0.484 (\(\kappa_{MH}\)) |
| VideoMAEv2 | 8 | 67.65 ± 0.72 | 85.41 ± 0.47 | 30.0 | 0.397 (\(\kappa_{MH}\)) |
| DINOv2 | 8 | 66.90 ± 0.73 | 91.96 ± 0.38 | 30.2 | 0.399 (\(\kappa_{MH}\)) |
| Human | ~100/sample | 74.2 | — | 4.8 (within human) | 0.906 (\(\kappa_{HH}\)) |
Note: Model–model agreement is \(\kappa_{MM} = 0.796\) [0.77, 0.82]. Despite V-JEPA2 trailing human accuracy by only \(\sim 1.0\text{ pp}\), the model–human disagreement rate (26.4%) is 5.5× higher than human–human disagreement (4.8%).
Scenario Breakdown & Attribution Ablation¶
Table 2 from the paper breaks down per-scenario accuracy across all 8 Physion sub-domains (mean ± std across seeds, \(\Delta = \text{V-JEPA2} - \text{Human}\)):
| Scenario | V-JEPA2 (%) | VideoMAEv2 (%) | DINOv2 (%) | Human (%) | \(\Delta\) (V-JEPA2 vs Human) |
|---|---|---|---|---|---|
| Collision | 83.8 ± 1.7 | 77.8 ± 2.2 | 70.6 ± 1.8 | 80.3 | +3.5 |
| Containment | 80.8 ± 2.6 | 73.4 ± 1.9 | 72.2 ± 2.1 | 76.0 | +4.8 |
| Towers | 80.3 ± 2.1 | 75.8 ± 1.9 | 72.2 ± 1.9 | 75.5 | +4.8 |
| Linking | 75.6 ± 1.8 | 72.7 ± 1.4 | 60.7 ± 3.4 | 63.8 | +11.8 |
| Roll/Slide | 75.9 ± 2.1 | 74.8 ± 2.4 | 74.6 ± 1.8 | 87.7 | -11.8 |
| Drop | 69.7 ± 2.8 | 66.4 ± 1.0 | 66.2 ± 2.5 | 74.0 | -4.3 |
| Clothiness | 60.9 ± 3.0 | 55.5 ± 2.6 | 61.1 ± 3.1 | 67.7 | -6.8 |
| Dominoes | 58.2 ± 2.3 | 44.8 ± 3.6 | 57.5 ± 2.2 | 68.7 | -10.5 |
| Overall | 73.2 ± 0.95 | 67.7 ± 0.72 | 66.9 ± 0.73 | 74.2 | -1.0 |
In the layer-wise feature attribution ablation predicting divergence \(|p_m - p_h|\) via 5-fold cross-validation: - Observable Features (Layers A + B, 66 dims): yields negative explanatory power (\(R^2 = -0.32\), \(\text{AUROC} = 0.63\)), demonstrating that visible scene properties before cutoff do not account for where models diverge from humans. - Unobservable Outcome Features (Layer C, 23 dims): achieves significant predictive power (\(R^2 = 0.29\), \(\text{AUROC} = 0.70\)); for samples where both model and humans are highly confident yet disagree, AUROC reaches 0.85.
Key Findings¶
- Breakdown of Agreement Hierarchy: The hierarchy confirms \(\kappa_{HH} (0.906) \gg \kappa_{MM} (0.796) \gg \kappa_{MH} (0.484)\). Internal model variability gap (\(\kappa_{HH} - \kappa_{MM} = 0.110\)) is small, but the cross-population gap (\(\kappa_{MM} - \kappa_{MH} = 0.312\)) is \(2.8\times\) larger, highlighting a profound divergence in solution strategies.
- Tightly Clustered Non-Human Strategies: All three architectures form a tight cluster of strategy fingerprints (pairwise cosine similarities 0.60–0.75, all \(p < 0.001\)), yet none significantly aligns with humans (cosine similarities 0.27–0.34, all \(p > 0.1\)).
- Geometric vs. Causal-Chain Domain Divergence: Models hold strong advantages on spatial/geometric tasks (linking: +11.8 pp), while humans retain large advantages on multi-step causal-chain scenarios (dominoes: -10.5 pp; roll/slide: -11.8 pp).
- Difficulty Inversion & Overconfidence: On the hardest stimuli for humans (human accuracy <50%, mean 30.7%), V-JEPA2 scores 50.4% (+19.8 pp higher, near chance). Entropy analysis reveals an asymmetry ratio of \(A = 0.32\) (\(|Q3| \approx 3\times |Q2|\)), indicating that models make confident commitments on ambiguous cases rather than mastering deeper physical concepts.
Highlights & Insights¶
- From Point Metrics to Population Distributions: The paper elegantly applies psychometric population modeling to machine learning benchmarking, revealing that surface-level score parity can conceal orthogonal behavioral paradigms.
- Temporally Decoupled Physical Feature Attribution: Isolating observable pre-cutoff dynamics from post-cutoff unobserved outcomes provides concrete empirical grounding that model failures stem from unsimulated futures rather than visual encoding deficiencies.
- Rigorous Controlled Probing Across Architectures: Eliminating confounders through standardized ViT-L backbones, temporal sampling rates, and attentive probe designs confirms that heuristic shortcutting is a shared characteristic across major self-supervised paradigms.
Limitations & Future Work¶
- High Crowdsourcing Dependency: The evaluation pipeline requires large-scale per-item human distributions (\(\sim 100\) annotations per video), which is expensive to scale to open-world video benchmarks.
- Correlational Rather Than Causal Attribution: Layer C features correlate with task outcomes by definition; establishing strict causal proof would require interventional physics engine manipulations.
- Frozen Representation Scope: The empirical scope is constrained to frozen ViT-L features with supervised attentive probes, leaving open whether 10B+ scale fine-tuned multimodal models or generative diffusion simulators exhibit genuine mental simulation.
Related Work & Insights¶
- vs. Physion Benchmark (Bear et al., 2021): While Physion established the binary contact task and aggregate accuracy comparison, this work repurposes its full human response matrix to expose previously hidden inter-rater disagreements and heuristic shortcuts.
- vs. Beyond-Accuracy Error Consistency (Geirhos et al., 2020): Extends image-classification error consistency frameworks into the dynamic physical reasoning regime, introducing temporal physical feature attribution and entropy quadrant profiling.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering distributional evaluation framework dissecting intuitive physics reasoning beyond aggregate accuracy]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous multi-seed, multi-architecture ViT-L setup across 1,200 videos and 120,000+ human response records]
- Writing Quality: ⭐⭐⭐⭐⭐ [Flawless narrative arc, lucid mathematical framing, and highly informative visual/tabular synthesis]
- Value: ⭐⭐⭐⭐⭐ [Crucial diagnostic framework for embodied AI, cognitive vision benchmarks, and video foundation model assessment]