KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding¶

Conference: ECCV 2026
Paper: ECCV page
Code: https://github.com/minecraft-zzz/KineBench
Area: Robotics & Embodied AI
Keywords: embodied world model, closed-loop evaluation, kinematic grounding, manipulability, complexity scaling

TL;DR¶

KineBench is an IDM-free closed-loop benchmark for embodied world models (EWMs): an explicit kinematic grounding pipeline (instance segmentation → monocular metric depth → CAD-constrained 6D pose tracking) recovers end-effector poses from generated videos and injects them into ManiSkill3 for physical execution, while two classical robot-centric metrics, SPARC and the Maruyama Manipulability Index, provide robot-side diagnostic signals; 7 frontier video models are evaluated over 20 manipulation tasks in four progressive suites.

Background & Motivation¶

Adopting video generation models as embodied world models has become a main line of Embodied AI: trained on internet-scale data, these video foundation models exhibit strong spatiotemporal predictive capability and are therefore treated as predictive priors for physical intelligence — either as pre-training backbones for vision-language-action models or as model-based planners that supply predictive rollouts for downstream execution. This shift exposes an evaluation gap: what we actually need to know is whether the motions these models generate can be executed and whether they are physically admissible, not whether the frames look good.

Existing evaluations largely cannot answer that. Early video generation benchmarks (VBench, FVD/PSNR-style metrics) measure pixel-level fidelity and spatiotemporal consistency; recent embodied benchmarks such as EWMBench and RBench go further and use vision models or large vision-language models to score scene consistency, motion correctness and semantic alignment, but they remain open-loop, pixel-space assessments that cannot verify physical feasibility. Another line probes physical commonsense with curated prompt suites and multimodal evaluators, yet still measures visual plausibility rather than actual physical feasibility. A few frameworks (World-in-World, WoW-World-Eval) ground generated sequences into a physics simulator for closed-loop execution and have already shown that visual quality does not correlate reliably with task success — the right direction, but it immediately hits the same methodological bottleneck: robot actions must be recovered from the generated RGB frames.

Almost every closed-loop framework has to do this with a learned inverse dynamics model (IDM), either an end-to-end neural mapping or an explicit tracking-based approach. The problem is that the mapping from 2D pixel space to 3D kinematic space is ill-posed and strongly dependent on the training distribution: an IDM overfits the action distribution of its training data and degrades markedly on the novel objects, scenes and trajectories produced by generative models (in the controlled study of Sec. 4.2, the IDM's pose error on unseen tasks reaches the order of ten centimeters). When closed-loop execution then fails, nobody can tell whether the world model drew a physically impossible scene or the IDM simply extracted the wrong actions — an attribution ambiguity that makes the conclusions of the whole closed-loop evaluation suspect.

This paper's angle is to replace implicit learned action inversion with explicit geometric grounding. It is feasible now because segmentation, monocular metric depth and CAD-based 6D pose tracking have become reliable enough, the gripper CAD model is available directly from manufacturers (so no manual annotation is needed at all), and — most importantly — every intermediate output of an explicit pipeline can be inspected and ablated independently, localizing the error to a specific stage. Core idea: replace the IDM entirely with cascaded visual foundation models that perform explicit 6D end-effector pose grounding, connect generated videos to a physics simulator for closed-loop execution, and complement execution success with two robot-centric kinematic metrics (SPARC and the Maruyama Manipulability Index) that explain why a model succeeds or fails, systematically probing EWM capability boundaries over 20 tasks and four progressive suites.

Method¶

Overall Architecture¶

KineBench decomposes "evaluating an embodied world model" into three things: first recover the end-effector's real motion in 3D space from the generated video, then actually execute that motion in a physics simulator, and finally use the robot's own kinematic metrics to explain why it succeeded or failed. The input is a language instruction plus an initial observation: the evaluated EWM autoregressively generates a predictive video; KineBench estimates the 6D end-effector pose frame by frame with cascaded visual foundation models; the pose sequence is injected into ManiSkill3, which executes the actions and renders the next observation, which is fed back to the EWM for the next generation cycle. The loop requires no modification to the evaluated model and no access to its action head, so open-weight models and closed-source API models are treated identically.

The decisive difference from existing closed-loop benchmarks is where actions come from: they are not inferred by an inverse dynamics model but computed by a geometric pipeline. The three stages inside that pipeline (mask, metric depth, CAD pose) can each be inspected in isolation, which makes it a more stable and more diagnosable evaluation interface than an IDM.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["language instruction + initial observation"] --> B["embodied world model<br/>autoregressive video generation"]
    B --> C["IDM-free explicit kinematic grounding<br/>instance segmentation → metric depth → CAD 6D pose"]
    C --> D["ManiSkill3 closed-loop execution<br/>pose injection → render next observation"]
    D -->|closed-loop feedback| B
    C --> E["robot-centric kinematic metrics<br/>SPARC smoothness + Maruyama manipulability"]
    D --> F["four-suite evaluation protocol<br/>S0 basic / S1 transfer<br/>S2 visual OOD / S3 complexity scaling"]
    E --> F

Key Designs¶

1. IDM-free explicit kinematic grounding: replacing learned action inversion with an inspectable geometric pipeline

This design targets exactly that attribution ambiguity: once action extraction errs in a closed loop, it becomes impossible to say whether the fault lies with the world model or with the extractor. KineBench performs extraction with three explicit geometric stages. First, a fine-tuned YOLOv11 segments the end-effector mask in every frame — and because the precise CAD model of the gripper is provided by the manufacturer, this stage requires no manual annotation. Second, a domain-fine-tuned MoGeV2 produces high-precision absolute metric depth maps that anchor the 2D mask in 3D space. Third, the mask, CAD model and depth map are handed to FoundationPose for 6D pose tracking, whose core mechanism is render-and-compare: the pose in SE(3) is iteratively refined by minimizing the photometric and geometric residuals between rendered CAD templates and the generated frames; the recovered pose is then injected into ManiSkill3, which executes it and renders the next observation.

The payoff is twofold. The first is diagnosability: the segmentation, depth and pose intermediates can be read and ablated individually, so errors can be localized — the paper's ablation attributes most of the pipeline's translational error to the depth estimation stage, while CAD-constrained pose tracking stays comparatively stable; an end-to-end IDM, by contrast, offers no way to separate perception errors from action-inference errors when it fails. The second is a geometric regularization effect that comes for free with the rigid-body constraint: generated videos inevitably contain high-frequency temporal jitter and local non-rigid deformations, and FoundationPose's enforced 3D rigid alignment absorbs such small pixel-level noise (the paper likens the effect to the low-pass filtering of an admittance controller), while remaining acutely sensitive to genuine physical hallucinations — a vanishing gripper or a spatially discontinuous jump fundamentally violates rigid-body mechanics and yields unstable or infeasible poses. Note that the paper claims only that this pipeline reduces IDM-specific attribution ambiguity; it does not claim error-free pose recovery, since it still depends on segmentation, depth recovery and sufficient visibility of the end effector.

2. Robot-centric dual-metric diagnosis: SPARC for "is it jittery", Maruyama manipulability for "is it reachable"

Metrics such as PSNR and FVD operate only in the 2D pixel domain, and a visually smooth video does not mean the robot can actually perform that motion. Because KineBench explicitly recovers 6D poses, evaluation can move from visual appearance to kinematic fidelity, so it adopts two classical metrics validated in clinical and robotics settings, characterizing trajectory smoothness and kinematic feasibility respectively.

SPARC (Spectral Arc Length) addresses a known defect of smoothness measures: traditional ones rely on high-order temporal derivatives of the position signal, which disproportionately amplify the minute high-frequency pose noise of generated video and invalidate the metric. SPARC shifts to the frequency domain and defines the arc length of the normalized velocity magnitude spectrum:

\[\text{SPARC} \triangleq -\int_0^{\omega_c}\left[\left(\frac{1}{\omega_c}\right)^2+\left(\frac{d\hat V(\omega)}{d\omega}\right)^2\right]^{1/2} d\omega\]

where \(\hat V(\omega)\) is the normalized velocity magnitude spectrum and \(\omega_c\) is the adaptive cutoff frequency. The physical meaning is direct: smooth actions obeying the minimum-energy principle concentrate their energy at low frequencies, giving a flat spectrum and a SPARC value approaching 0; conversely, if the model generates stuttering or teleportation, high-frequency harmonics increase, the curve becomes convoluted and the SPARC value drops markedly. The metric therefore probes temporal consistency of the world model (⚠️ the formula's typesetting is corrupted in the cached text; it is reconstructed here from the original SPARC definition — refer to the paper).

The Maruyama Manipulability Index covers a different failure mode: smoothness alone does not guarantee manipulation success, since the robot must persistently avoid singular configurations. Manipulability is defined as \(w=\sqrt{\det(J(q)J^{T}(q))}\), the volume of the end-effector velocity ellipsoid determined by the Jacobian; feeding the 6D poses recovered from the generated video into an inverse kinematics solver, poses that exceed the robot's physical joint and structural constraints make the index decline sharply, so integrating manipulability along the whole trajectory directly reflects the world model's implicit cognition of this robot's kinematic boundaries.

The two metrics do not replace success rate but act as complementary failure-mode probes: the paper finds their association with execution success to be task- and model-dependent, with no single monotonic score-to-success mapping. That is precisely where their diagnostic value lies — once smoothness saturates, failures come mainly from semantic misalignment, whereas low manipulability points to generative randomness pushing the robot toward singularities or kinematically unreachable points.

3. Four-suite progressive protocol: isolating one generalization axis at a time, then controlled complexity scaling

A single success-rate number cannot distinguish "the model memorized the training trajectories" from "the model learned physical causality", so KineBench designs 20 manipulation tasks in ManiSkill3 — covering basic reaching, rigid-body grasping with and without obstacles, articulated object manipulation along the X/Y/Z axes (hinges and sliders), and long-horizon compositional tasks, each with highly randomized object instances and background textures and stratified into difficulty levels (some assets come from the HumanoidGen project) — and organizes them into four suites of increasing diagnostic depth.

Suite 0 (basic execution) keeps training and evaluation strictly identically distributed (I.I.D.) over all tasks, object instances and backgrounds, establishing the baseline capability to reproduce demonstrated dynamics without domain shift. Suite 1 (task transfer) measures zero-shot transfer to conceptually related but disjoint tasks: a model trained on CloseBox, OpenBoxEasy-v1 and CloseLaptopEasy is evaluated on unseen spatial recombinations such as OpenBoxEasy-v2 (novel box orientations) or a harder variant such as CloseLaptopHard, probing instruction understanding, spatial extrapolation and physical causality rather than trajectory memorization. Suite 2 (visual OOD generalization) splits the randomized asset library (object instances and background textures) in half, restricting training to 50% and evaluating entirely on the disjoint remaining 50%, thereby decoupling visual representation robustness from physical reasoning. Suite 3 (complexity scaling) follows the I.I.D. task distribution of Suite 0 and systematically varies the training data volume and compute budget (with training duration as a proxy for compute), examining how intrinsic task complexity — degrees of freedom, contact dynamics — changes the gains obtained from more data and compute.

Together the four suites turn basic execution capability, transfer capability, visual robustness and scaling behavior into four separately readable quantities instead of one blended score.

Loss & Training¶

KineBench is itself an evaluation framework with no unified training objective, but two learned components in the pipeline require in-domain fine-tuning: a fine-tuned YOLOv11 for 2D mask extraction, and a two-stage fine-tuned MoGeV2 whose second stage amplifies the loss on fine-grained geometric details to obtain high-fidelity metric depth; pose tracking uses FoundationPose zero-shot with no additional training. The evaluated EWMs are fine-tuned with their respective recipes: Wan 2.1-1.3B, Wan 2.2-5B and CogVideoX-2B are fully fine-tuned, while LoRA variants exist for CogVideoX-5B and Wan 2.2-5B; the default is 7.5k training steps, and all experiments are constrained by a budget of four NVIDIA A100 GPUs. The Suite 3 scaling study sweeps optimization steps between 1.5k and 15k and expert trajectory counts between 10 and 50 (per the Table 1 column labels), using training duration as the compute proxy. The extraction-and-rollout pipeline is deliberately decoupled to keep closed-loop evaluation efficient and stable.

Key Experimental Results¶

Main Results¶

The primary metric is closed-loop execution success rate in ManiSkill3, covering base capacity (S0), cross-task transfer (S1) and visual OOD robustness (S2); values are taken directly from Table 1 of the paper:

Suite	Wan2.2-5B	CogVideoX-2B	Wan2.1-1.3B	Wan2.2-5B (LoRA)	CogVideoX-5B (LoRA)	Wan2.6 (API)	Hailuo-V2 (API)
Suite 0 basic execution (I.I.D.)	56.32	46.17	43.96	19.00	18.33	14.08	6.56
Suite 1 task transfer	11.90	57.78	20.00	16.67	3.81	–	–
Suite 2 visual OOD (seen)	58.00	56.11	57.19	25.33	–	–	–
Suite 2 visual OOD (unseen)	55.50	51.67	52.83	26.33	–	–	–
Suite 3 complexity scaling	–	–	43.96	–	–	–	–

(All values are percentages; – marks a configuration not evaluated in that suite. ⚠️ The column headers of Table 1 are scrambled in the PDF text extraction, so the column-to-model mapping above is reconstructed from the consistency between the values and the paper's statements — full fine-tuning of Wan2.2-5B beats its LoRA variant in S0 while the relation reverses in S1, and the Suite 3 cell uses Wan2.1-1.3B (43.96 matches the same configuration in S0) — refer to Table 1 of the original paper for the exact per-column mapping.)

The controlled scaling study in Suite 3 (Wan2.1-1.3B architecture, training duration as compute proxy):

S3 configuration	Closed-loop success rate	Note
optimization steps 1.5k	44.83	smallest compute budget
optimization steps 4.5k	47.88	modest gain
optimization steps 7.5k (default)	43.96	matches the same configuration in S0
optimization steps 15k	73.33	the largest compute gain
trajectories scale=10 (7.5k steps)	53.67	least training data
trajectories scale=25	52.00	more data, slightly lower
trajectories scale=50	51.17	decreasing further

(The Suite 3 values are taken from the right half of Table 1; the column labels follow the header order "steps 1.5k / 4.5k / 15k / Scale 10 / 25 / 50" — ⚠️ refer to the original. The main text describes the sweep as 10 → 100 expert trajectories and 1,500 → 15,000 optimization steps, which differs slightly from the Table 1 column labels.)

Diagnostics from the two robot-centric metrics (Figures 6 and 7 of the paper): the association between SPARC and standardized success rate depends strongly on a model's generation capability — for zero-shot and under-optimized models SPARC correlates strongly with success rate and effectively filters out catastrophic physical hallucinations caused by poor motion fluency, whereas fully fine-tuned models already generate kinematically smooth trajectories that saturate SPARC, so their failures stem mainly from semantic misalignment rather than local jitter. On the manipulability side, fully fine-tuned models consistently maintain low manipulability costs, indicating that they internalize the structural boundaries of this 7-DoF robot; zero-shot closed-source models and capacity-limited LoRA models do not, generating visually plausible motion while their inherent stochasticity frequently pushes the robot toward singularities or kinematically unreachable points, leading to execution failures. The Manip score is computed from trajectory costs obtained via PyRoki and normalized with robust min–max scaling over the 10th–90th percentiles; lower is better.

Ablation Study¶

The first ablation targets the action-extraction stage, comparing three configurations on simulator-rendered trajectories from tasks excluded from IDM training (the IDM baseline is Dino3DFlowIDM from WoW-World-Eval, trained on all trajectories of the ten training tasks in Suite 1, 100 trajectories per task; evaluation randomly samples five trajectories per unseen task, 35 in total):

Action-extraction configuration	Translation error	Rotation error	Note
FoundationPose + simulator ground-truth depth	centimeter level	not reported separately	upper bound of the pipeline
FoundationPose + MoGeV2-estimated depth	roughly 1.5–3 cm	roughly 10°	the actual KineBench pipeline
Dino3DFlowIDM (prior closed-loop benchmark)	order of ten centimeters	occasionally lower mean, but many more long-tail failures	degrades markedly on unseen tasks

The second ablation examines the influence of upstream depth quality. Under controlled visual perturbations (gripper deformation, 20%/50% occlusion, reduced contrast, motion blur, image noise), the KineBench pipeline achieves substantially lower translation error than the IDM in every condition; its rotation mean error is slightly higher in some cases, but the distribution is far more concentrated with far fewer long-tail failures. The paper further compares "well-trained" and "clearly underfitting" MoGe quality and finds the resulting pose errors nearly identical (Figure 5), showing that pose recovery in this pipeline is insensitive to depth quality.

Key Findings¶

The attribution ambiguity is measured, not asserted: on unseen tasks the explicit grounding pipeline reaches centimeter-level translation error (roughly 1.5–3 cm) whereas the IDM sits at the order of ten centimeters — a gap caused by the IDM's overfitting to its training action distribution, not by the simulator or the tasks. This suggests prior IDM-based closed-loop evaluations may under-estimate a world model's true capability under distribution shift, or charge the extractor's errors to the world model.
Scale is no panacea and its payoff is tightly coupled to task complexity: in S0 the larger Wan2.2-5B clearly beats the smaller models (56.32 vs 46.17 for CogVideoX-2B and 43.96 for Wan2.1-1.3B), yet all models drop sharply on contact-rich tasks (StackCube, PickFruits) — evidence that video models capture kinematic priors while friction and collision dynamics remain an open problem.
Full fine-tuning versus LoRA is a specialization-versus-transfer trade-off: S0 favors full fine-tuning (56.32 vs 19.00), but S1 reverses the relation (11.90 vs 16.67). One possible explanation is that LoRA preserves more pretrained visual and motion priors while full-parameter fine-tuning specializes harder to the training tasks — the paper explicitly treats this as a hypothesis rather than a causal conclusion.
Under visual OOD, models rely on appearance-specific features rather than 3D affordances: simple tasks such as CloseBox stay relatively robust on unseen assets, but tasks requiring precise affordance localization degrade substantially — Wan 2.2 drops from 60.0% to 30.0% on OpenBoxHard (task-level numbers from the main text).
Scaling curves are nonlinear and task-dependent: across 1.5k → 4.5k → 7.5k → 15k optimization steps the success rate moves 44.83 / 47.88 / 43.96 / 73.33, i.e. non-monotonically overall, while increasing trajectories from 10 to 50 actually lowers it from 53.67 to 51.17. At task level the contrast is starker: on the simpler OpenBoxEasy, raising homogeneous training trajectories from 10 to 100 drops success from 100% to 83.33% (overspecialization risk), whereas on the more complex StoreCube-v2 a larger optimization budget raises it from 13.34% to 69.52%. The marginal benefit of data and compute depends on task complexity and trajectory diversity.

Highlights & Insights¶

It puts a hidden confounder of closed-loop evaluation on the table and quantifies it. Most closed-loop benchmarks treat the IDM as a harmless engineering detail; KineBench argues it is the largest source of attribution ambiguity and measures a ten-centimeter versus centimeter-level gap on unseen tasks. This "first prove the old evaluation interface is unreliable, then replace it" argument generalizes to any evaluation pipeline that uses a learned model as a component.
CAD plus rigid-body constraints buy both annotation-free operation and noise immunity. Because the gripper CAD comes from the manufacturer, mask annotation cost is zero; and render-and-compare's rigid-body assumption acts as a low-pass filter that absorbs pixel-level jitter yet stays sensitive to genuine physical hallucinations — one constraint solving two problems is an economical modeling choice.
Classical robotics and clinical metrics are imported into generative model evaluation. SPARC and manipulability were originally used for human movement rehabilitation assessment and robot configuration quality; KineBench moves them onto generated video, giving two interpretable, recomputable axes for "does this generated motion look like something the robot could actually do." The same move — borrowing mature domain metrics — transfers to any generation evaluation that needs physical feasibility (e.g. turning energy conservation or contact impulse constraints into metrics).
Reusable trick: expose intermediates to make ablation possible. Even though the pipeline remains a cascade of learned modules, readable intermediate outputs enable clean ablations such as swapping in ground-truth depth, which answers "which stage does the error come from." Any work that uses a learned model as an evaluation interface can copy this.

Limitations & Future Work¶

The authors acknowledge that the expert trajectories in the benchmark are generated by motion planning, so the action distribution is narrow; future iterations will integrate human teleoperation data to enrich it. They also note that the 3D kinematic metrics capture only low-level physical coherence, with future work integrating them with existing 2D pixel-based evaluations for a more holistic diagnosis.
Residual pipeline error is acknowledged rather than hidden: pose recovery depends on segmentation, depth and end-effector visibility, and the remaining rotation error is about 10°, which is not small for precision manipulation; the authors therefore limit the claim to "reducing IDM-specific attribution ambiguity" rather than error-free recovery.
Limitations I would add: the scaling study covers only the Wan 2.1-1.3B architecture, so "complexity-conditioned nonlinear scaling" needs care when extrapolated; the exact meaning of scale (trajectory count versus a data-size multiplier) is not fully consistent between the Table 1 labels and the main text; the closed-source API models appear only in Suite 0 and thus contribute nothing to the transfer or OOD diagnostics; a single simulator (ManiSkill3) and a single 7-DoF arm leave cross-simulator and cross-embodiment generalization untested; and the manipulability score still depends on the inverse kinematics solver (PyRoki) and the chosen normalization.
Improvement directions: adopt a more occlusion-robust depth estimator (or multi-view input) to cut the rotation error; fuse IDM and explicit grounding complementarily (falling back to an IDM when the end effector is fully occluded, with explicit uncertainty reporting); and repeat the scaling study across architectures and embodiments to test whether the trend is architecture-independent.

vs World-in-World / WoW-World-Eval (IDM-based closed-loop benchmarks): they also ground generated sequences into a simulator for closed-loop execution, but actions must be extracted by an IDM (WoW-World-Eval even proposes an IDM-based Turing test); this paper removes the IDM dependency entirely, replacing implicit inversion with explicit geometric grounding and showing empirically that IDM pose error on unseen tasks is an order of magnitude larger. The advantage is a more stable, more diagnosable interface; the cost is a residual dependence on segmentation and depth quality, which still fails when the end effector is heavily occluded.
vs EWMBench / RBench (open-loop visual and VLM evaluation): they score scene consistency, action completeness and semantic alignment in pixel and semantic space with broad coverage at low cost, but cannot verify physical feasibility; KineBench measures directly whether the robot can execute the motion, so the two are complementary rather than substitutes.
vs physical-commonsense evaluations such as VideoPhy: those judge whether physical commonsense holds via prompt suites and multimodal evaluators (or diffusion likelihoods and optical-flow proxies), which is visual plausibility; KineBench argues physical feasibility can only be verified by execution.
vs video generation quality benchmarks (VBench / FVD / PSNR): they do not correlate reliably with task success, which the main results here reconfirm.

Rating¶

Novelty: ⭐⭐⭐⭐ First work to eliminate the IDM dependency entirely in closed-loop embodied world model evaluation, and the first to bring SPARC and the Maruyama Manipulability Index into generative video assessment.
Experimental Thoroughness: ⭐⭐⭐⭐ 7 models × 20 tasks × four suites plus controlled visual-perturbation and depth-quality ablations; the scaling study, however, is limited to a single architecture by the 4×A100 budget.
Writing Quality: ⭐⭐⭐⭐ The problem is sharply framed and the argument chain is complete, with experimental support for the IDM critique; a few table headers disagree slightly with the main text.
Value: ⭐⭐⭐⭐ It offers an executable, diagnosable and recomputable evaluation protocol for embodied world models, and its "explicit grounding instead of implicit inversion" idea transfers directly to other evaluation pipelines that use models as components.