Skip to content

title: >- [Paper Note] Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos description: >- [ECCV 2026][Image Generation][Video Evaluation] Proposes a learned action manifold combining 3D SMPL skeletal geometry, 2D keypoints, visual appearance, and first-order temporal derivatives to assess action consistency and temporal coherence in synthesized human videos, outperforming SOTA MLLMs by up to 68% on TAG-Bench-v0. tags: - ECCV 2026 - Image Generation - Video Generation Evaluation - Human Action Consistency - Temporal Coherence date: 2026-09-19 content_hash: 735ab69b71ca2152

Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

Conference: ECCV 2026
Paper: ECCV Official
Code: https://xthomasbu.github.io/video-gen-evals/
Area: Image Generation
Keywords: Video Generation Evaluation, Human Action Consistency, Temporal Coherence, SMPL Skeletal Geometry, TAG-Bench-v0

TL;DR

Addressing strong appearance bias and a fundamental lack of temporal kinematic awareness in existing vision encoders and MLLMs, this paper introduces a learned latent manifold of real-world human actions fusing 3D/2D geometry with first-order motion derivatives, proposing reference-free action consistency and temporal coherence metrics that outperform SOTA MLLMs by over 68% in alignment with human judgments on the new TAG-Bench-v0 benchmark.

Background & Motivation

Recent generative video models—such as Wan2.1/2.2, Runway Gen-4, and HunyuanVideo—have made striking strides in photorealism and visual quality, and are increasingly regarded as emergent world models. However, when synthesizing complex human activities, these models frequently produce severe physical anomalies, including anatomical distortions, limb morphing, unnatural joint dislocations, and sudden temporal freezing. Standard automated metrics (e.g., PSNR, SSIM, LPIPS, CLIP-sim, and FVD) fundamentally evaluate single-frame perceptual appearance or coarse distribution statistics, remaining completely blind to fine-grained kinematic violations and temporal coherence over time.

Even recent attempts to employ Multimodal Large Language Models (MLLMs) as evaluators (such as VideoScore and VideoPhy) struggle significantly in this regime. These models suffer from heavy appearance biases and are primarily trained or prompted to verify general object-level physics (e.g., gravity or fluid buoyancy), rather than nuanced human biomechanics and joint coordination. Furthermore, existing benchmarks such as EvalCrafter and VBench-2.0 do not explicitly probe whether human actions are executed plausibly across time. This evaluation gap leaves the video synthesis community without reliable automated diagnostic tools for human motion realism.

This work addresses this challenge by positing that human motion realism can be formalized by measuring the distance to a learned compact manifold of natural, physically plausible human actions. Core idea: fuse appearance-invariant 3D SMPL geometry, unconstrained 2D keypoints, visual appearance features, and their first-order temporal derivatives to learn a latent real-world action manifold via supervised contrastive and temporal hard-negative objectives, deriving reference-free metrics for action consistency and temporal coherence.

Method

Overall Architecture

The framework quantifies the deviation of generated human actions from the distribution of natural real-world actions. The pipeline consists of human-centric multi-source feature extraction, static-motion dual-branch temporal convolution encoding, per-frame adaptive attention fusion, long-range temporal aggregation via a Transformer encoder, and manifold distance scoring. Input videos are sliced into overlapping temporal windows of \(T=32\) frames, mapped into the learned latent space, and evaluated using two complementary quantitative metrics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Video Clips<br/>Temporal window T=32 frames"] --> B["Multi-Source Human-Centric Feature Extraction<br/>3D SMPL + 2D Keypoints + ViT + First-Order Derivatives"]
    B --> C["Dual-Pathway Temporal Convolution Encoding<br/>Static feature stream + motion residual additive fusion"]
    C --> D["Adaptive Attention Per-Frame Fusion<br/>Dynamic channel weighting across feature streams"]
    D --> E["Temporal Transformer Aggregation & Dual Optimization<br/>Supervised contrastive loss + temporal hard-negative loss"]
    E --> F["Quantitative Metric Evaluation<br/>Action Consistency S_cons + Temporal Coherence S_temp"]

Key Designs

1. Multi-source static and first-order derivative feature representation

To circumvent the blind spots of individual representations, the model defines a static feature set \(S = \{\theta, \beta, go, kp2D, f_{vis}\}\). TokenHMR extracts 3D joint rotations \(\theta\), body shape parameters \(\beta\), and pelvis global orientation \(go\) from the SMPL model to provide appearance-invariant anatomical constraints. To detect anomalies that SMPL priors might mask (e.g., elongated limbs), DWPose extracts unconstrained 2D keypoints \(kp2D\). In addition, ViT-H/16 appearance embeddings \(f_{vis}\) capture clothing, texture, and contextual cues. Crucially, to catch unnatural morphing and temporal jitter, the method computes first-order temporal differences across consecutive frames, yielding motion features \(U = \{m_\theta, m_\beta, m_{go}, m_{kp2D}, m_{fvis}\}\) (relative rotations for angles; \(\ell_2\) differences for shape, keypoints, and appearance), making the representations acutely sensitive to dynamic generative artifacts.

2. Dual-pathway temporal convolutions and adaptive attention fusion

To process features with disparate temporal dynamics, separate 1D temporal convolution blocks \(\phi_{static}^k\) and \(\phi_{motion}^k\) with dilation rates {1, 2, 4} are applied to each static and motion stream. The motion pathway is injected as a residual into the static pathway: $\(\mathbf{e}_{k,t} = \phi^k_{\text{static}}(s_{k,t}) + \phi^k_{\text{motion}}(u_{k,t})\)$ To aggregate the heterogeneous encoded features into a single frame-level representation \(\mathbf{f}_t\), a scaled dot-product attention mechanism dynamically computes importance weights \(\alpha_{k,t}\) via a learnable query vector \(\mathbf{q} \in \mathbb{R}^d\) and projection matrix \(W_a\): $\(\mathbf{f}_t = \sum_k \alpha_{k,t} \mathbf{e}_{k,t}, \quad \alpha_{k,t} = \text{softmax}_k\left(\frac{\mathbf{q}^\top W_a \mathbf{e}_{k,t}}{\sqrt{d}}\right)\)$ This enables the model to focus adaptively on the most informative anatomical or visual channels for any given action.

3. Manifold learning with action semantics and temporal hard negatives

A learnable \([CLS]\) token is prepended to the sequence of frame tokens \(\{\mathbf{f}_t\}_{t=1}^T\) and fed into a 4-layer Transformer encoder to model long-range temporal dependencies, yielding a window embedding \(\mathbf{z}_{CLS}\) and frame embeddings \(\{\mathbf{z}_t\}_{t=1}^T\). The network is optimized via a dual-objective loss: $\(\mathcal{L} = \mathcal{L}_{\text{supcon}} + \lambda \mathcal{L}_{\text{hard-negative}}\)$ where \(\mathcal{L}_{\text{supcon}}\) clusters real videos belonging to the same action class while separating different actions, and \(\mathcal{L}_{\text{hard-negative}}\) explicitly enforces physical plausibility by pushing away self-supervised temporal perturbations—namely frame shuffling (breaking temporal continuity), first-frame repetition (simulating frozen motion), and frame reversal (violating causal arrow of time).

4. Dual quantitative metrics: Action Consistency and Temporal Coherence

From the structured latent action manifold, two complementary reference-free metrics are formulated: - Action Consistency (\(S_{cons}\)): Measures the Euclidean distance between the average window-level \([CLS]\) embedding of the synthesized video \(\mathbf{z}_{genvideo}\) and the centroid of real-world training instances for that specific action class \(\mathbf{c}_k\): $\(S_{cons} = \|\mathbf{z}_{genvideo} - \mathbf{c}_k\|_2\)$ A lower distance indicates closer adherence to authentic real-world kinematic semantics. - Temporal Coherence (\(S_{temp}\)): Measures the trajectory smoothness across consecutive frame embeddings within the latent space: $\(S_{temp} = \frac{1}{T-1} \sum_{t=1}^{T-1} \|\mathbf{z}_{t+1} - \mathbf{z}_t\|_2\)$ A lower score indicates smooth, biomechanically plausible movement transitions, penalizing sudden visual or skeletal jerks.

Key Experimental Results

Main Results

Evaluation is performed on TAG-Bench-v0, comprising 300 videos generated across 5 frontier models (Wan2.1, Wan2.2, Hunyuan, OpenSora, Runway Gen-4) evaluated by 246 Amazon Mechanical Turk human raters. Performance is measured via Spearman's rank correlation (\(\rho\)) against human scores.

Table 1: Spearman's rank correlation (\(\rho\)) with human ratings on TAG-Bench-v0

Category Method / Metric Action Consistency \(\rho \uparrow\) Temporal Coherence \(\rho \uparrow\)
Baseline Random -0.07 -0.11
Feature-based Metrics (Top-3) VideoMAE(UCF101)-cls 0.18 0.17
DINO-sim 0.08 0.21
CLIP-sim 0.03 0.16
MLLM-based Fine-tuned (Top-3) VideoPhy-2 (Physical Commonsense) 0.28 0.37
VideoPhy-2 (Semantic Adherence) 0.19 0.16
VideoScore2 (Physical Consistency) 0.18 0.17
MLLM Prompting (Top-3) GPT-4o 0.34 0.31
Gemini-2.5-Pro (4x10 grid) 0.31 0.26
GPT-5 0.45 0.38
Ours Action Consistency \(S_{cons}\) (Ours) 0.61 0.45
Temporal Coherence \(S_{temp}\) (Ours) 0.53 0.64
Gain over Best Baseline \(\Delta\) over best baseline +0.16 (+35.6%) +0.26 (+68.4%)
Human Agreement Upper Bound Human vs Human (Inter-rater) 0.72 0.71

Ablation Study

Ablation studies analyze the impact of individual loss objectives (Table 2) and input feature modalities (Table 3) on Spearman correlation.

Table 2 & 3: Ablation analysis of loss functions and input feature components (\(\rho\))

Configuration Feature / Loss Components Action Consistency \(\rho\) Temporal Coherence \(\rho\) Note
Full Model \(\mathcal{L}_{supcon} + \mathcal{L}_{hard-neg}\) (all features) 0.61 0.64 Best overall performance
Loss Ablation w/o \(\mathcal{L}_{supcon}\) (hard-negative only) 0.26 0.38 Semantic collapse without cluster structure
Loss Ablation w/o \(\mathcal{L}_{hard-neg}\) (supcon only) 0.54 0.57 Degraded temporal violation sensitivity
Feature Ablation Zeroing 3D Pose (\(\theta\)) 0.56 0.57 Missing kinematic rotation constraints
Feature Ablation Zeroing 2D Keypoints (\(kp2D\)) 0.56 0.59 Missing unbiased distortion cues
Feature Ablation Zeroing all Motion Derivatives (\(U\)) 0.46 0.50 Largest degradation across all features
Visual-Only Comparison CLIP / DINOv2 without human features 0.12 – 0.14 0.19 – 0.26 Appearance-only features fail to evaluate dynamics

Key Findings

  • Motion derivatives are critical anomaly indicators: Masking out first-order temporal derivatives causes the sharpest performance drop (Action Consistency falls from 0.61 to 0.46, Temporal Coherence drops from 0.64 to 0.50), proving that frame-to-frame change velocity is the single most informative cue for catching generative defects.
  • Generative model performance comparison: Wan2.2 achieves the highest win ratios on TAG-Bench-v0 (0.77 on Action Consistency, 0.72 on Temporal Coherence), outperforming closed-source Runway Gen-4 (0.65 and 0.61, respectively).
  • Rotational full-body dynamics remain universal failure points: Highly rotational actions involving complex momentum (e.g., Shotput and ThrowDiscus) suffer from severely degraded scores across all five evaluated generative models, while reciprocating motions (e.g., PushUps, PullUps) are synthesized with substantially higher fidelity.

Highlights & Insights

  • Complementary geometric and unconstrained keypoint pairing: By coupling SMPL 3D mesh parameters (which provide strong biomechanical priors) with 2D keypoints (which are free of anatomical constraints and readily reveal distorted or elongated limbs), the model effectively captures both anatomical plausibility and generation-specific artifacts.
  • First-order temporal derivatives as artifact amplifiers: Computing temporal differences magnifies subtle spatial glitches into noticeable dynamic discontinuities, offering an elegant and computationally lightweight signal for detecting temporal jitter and sudden body morphing.
  • Self-supervised temporal distortion perturbations: Frame shuffling, freezing, and reversal synthesize realistic hard negatives without requiring manual error annotations, instilling an acute sensitivity to the physical arrow of time.

Limitations & Future Work

  • Dependency on human mesh recovery and pose estimation backbones: The pipeline's accuracy relies on upstream detectors (TokenHMR, DWPose). Severe occlusion, multi-person crowds, or heavy rendering blur that causes tracking failures may degrade feature quality.
  • Centroid requirement for action consistency: The \(S_{cons}\) metric relies on real video centroids for each action class, limiting its plug-and-play application on open-ended or arbitrary text-prompted actions (whereas \(S_{temp}\) generalizes seamlessly in a zero-shot manner).
  • Future directions: The authors plan to integrate human physics representations into modern MLLMs to enable open-vocabulary prompt evaluation, and extend the benchmark to multi-person interactions and long-form video synthesis.
  • vs VideoPhy-2 / VideoScore2: While VideoPhy-2 assesses physical commonsense via fine-tuned MLLMs, it focuses on general object-level physics (gravity, collision) rather than biological kinematics. The proposed metric outperforms VideoPhy-2 by +35.6% on Action Consistency and +68.4% on Temporal Coherence.
  • vs VBench-2.0: VBench-2.0 measures static human appearance and prompt adherence per frame. On the VBench-2.0 Human Anatomy subset, the proposed \(S_{temp}\) metric yields model win-ratio rankings that match human rater preferences with 100% agreement.

Rating

  • Novelty: ⭐⭐⭐⭐ [Introduces a principled action manifold combining 3D/2D geometry and temporal derivatives for video generation evaluation]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Introduces TAG-Bench-v0 with 300 videos, 246 annotators, extensive ablations, and external benchmark validation]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated methodology, structured empirical analysis]
  • Value: ⭐⭐⭐⭐ [Provides an essential evaluation toolkit and diagnostic benchmark for physically plausible human video synthesis]