Schroedinger’s Cat: Probabilistic Representation and Prediction of Potential Scene Kinematics

Conference: ECCV 2026
Paper: ECCV Official
Project: https://compvis.github.io/schroedingers_cat/
Area: Video Generation
Keywords: Scene Kinematics, Motion Planning, Probabilistic Latent Representation, Sampling-free Density Estimation, World Models

TL;DR

GARFIELD models the distribution over plausible future scene kinematics via structured spatio-temporal latents, supporting both coherent multi-trajectory generation and instantaneous sampling-free probability density estimation at 97× faster trajectory sampling than video models.

Background & Motivation

Anticipating how a physical scene may evolve under partial observability is inherently stochastic and under-determined, where a single current frame often admits multiple diverging future trajectories. Conventional video world models treat future prediction as pixel-level synthesis, generating dense frame sequences. However, a major portion of their model capacity and compute budget is squandered on hallucinating fine appearance details and static background textures rather than dynamics, leading to prohibitive inference latency (often tens to hundreds of seconds) that hampers fast exploration of possibilities. Direct motion prediction models based on optical flow or point tracking alleviate this computational bottleneck, yet prior works remain confined to dense flow grids or sample individual realizations without explicitly modeling the underlying probability density and uncertainty.

The core tension stems from an intrinsic data and inferential limitation: real-world video annotations provide only a single factual realization of motion, leaving the true distribution over potential futures unobservable. While generative diffusion models can sample trajectories, estimating motion probabilities typically requires drawing extensive Monte-Carlo samples, which introduces intolerable latency in real-time robotic planning and interactive guidance. Furthermore, existing trajectory conditioning schemes demand rigid, dense inputs such as complete goal frames, text prompts, or continuous temporal tracks, failing to handle flexible, spatio-temporally sparse goal conditioning.

Inspired by Schrödinger's thought experiment in quantum mechanics where unobserved systems exist in superposition until measurement collapses the wave function, this work models scene futures as latent probability distributions that progressively collapse as observations or sparse constraints are incorporated. GARFIELD addresses this challenge through a unified probabilistic representation that jointly supports multi-trajectory sampling and deterministic density decoding. Core idea: represent the distribution of potential future scene kinematics via spatio-temporally disentangled latent distributions conditioned on sparse constraints, enabling both coherent joint trajectory sampling and sub-millisecond deterministic density decoding in a single forward pass.

Method

Overall Architecture

GARFIELD structures its pipeline into three core stages: spatio-temporal latent encoding, deterministic density estimation, and coherent trajectory generation. The model receives a start frame image \(I\) alongside spatio-temporally sparse constraints \(b_T\) (summarized as condition \(C = (I, b_T)\)). A joint transformer encoder \(E_\phi\) extracts visual and motion features to construct a factorized set of latents \(z_{i\tau}\), each corresponding to a specific scene element \(i\) at future timestep \(\tau\). Crucially, the identical latent space serves two downstream decoding pathways: a deterministic density estimator \(D_\omega^d\) that produces non-parametric spatial probability heatmaps via a single ViT forward pass, and a generative full decoder \(D_\theta\) that captures cross-temporal and inter-object dependencies to sample complete, mutually consistent trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Conditioning C<br/>(Start Image I + Sparse Goals bT)"] --> B["Joint Kinematics Encoder: Global Feature Extraction & Spatio-temporal Factorization"]
    B --> C["Structured Latent Distribution {ziτ}"]
    C --> D["Sampling-free Density Decoder: Deterministic Heatmap Prediction"]
    C --> E["Full Trajectory Decoder: Joint Generative Trajectory Sampling"]
    D --> F["Interactive Uncertainty Exploration: Shannon Entropy Guided Intervention"]
    F -->|Progressively Refined Constraints C| A
    E --> G["Output Coherent Future Trajectories T"]

Key Designs

1. Structured Spatio-temporal Latent Factorization: Disentangling Local Kinematics from Scene Context Global sequence vectors fail to localize uncertainty to particular objects and timesteps, preventing precise inspection and local constraint refinement. GARFIELD addresses this by learning a joint transformer encoder \(E_\phi\) combined with DINOv2R-B vision features and 3D Axial RoPE, mapping input condition \(C\) into factorized latent vectors \(z_{i\tau}\) for each element \(i\) and time \(\tau\). During encoder pretraining, a lightweight point-wise generative MLP decoder \(D_\psi^p\) (34M parameters) is jointly trained to reconstruct track points \(t_{i\tau} \sim D_\psi^p(z_{i\tau}) \approx p(t_{i\tau} \mid C)\). Goal sparsity \(b_T / T\) is linearly decayed from 0.5 to 0.01 over the first 50k steps to enforce robust inference across varying degrees of partial knowledge, ensuring that each local latent component faithfully encodes the localized future kinematics distribution.

2. Sampling-free Deterministic Density Estimation: Constant-time Non-parametric Uncertainty Decoding Estimating probability densities via generative Monte-Carlo sampling demands dozens of sampling steps, making real-time feedback impossible. GARFIELD introduces a deterministic density decoder \(D_\omega^d\) operating over a regular \(g \times g\) spatial grid \(\mathcal{G}\) on the image plane. Taking only the point-wise latent \(z_{i\tau}\) as input, a single Vision Transformer forward pass directly outputs the discrete spatial probability distribution: $$ p_\omega^{(u,v)}(z_{i\tau}) = \text{P}(t_{i\tau} \in \mathcal{B}{u,v} \mid z) $$ Trained under a grid cross-entropy objective \(\mathcal{L}_{\text{grid}}\) with one-hot discretized bin targets, this density estimator computes accurate spatial heatmaps in under 0.8 ms. This provides a speedup of more than two orders of magnitude compared to iterative Monte-Carlo estimation while closely preserving non-parametric multi-modal densities.

3. Full Generative Decoder: Eliminating Temporal Incoherence and Inter-object Mode Collisions Sampling trajectories point-wise across timesteps under multi-modal uncertainty often causes mode collapse or mode flickering, where sample points jump erratically between diverging paths across consecutive frames. To resolve mutual dependencies between points within and across tracks, GARFIELD freezes the pretrained encoder and trains a joint full decoder \(D_\theta\). Conditioning on the full latent set \(\{z_{i\tau}\}_{i,\tau}\), \(D_\theta\) generates coherent trajectories \(T \sim p(T \mid C)\) in a single generative rollout, resolving spatial collision constraints and temporal continuity through global self-attention across all elements and timesteps.

4. Entropy-guided Active Conditioning: Accelerating Possibility Collapse To minimize user intervention during interactive planning, GARFIELD leverages the Shannon entropy of the predicted spatial density as an explicit measure of kinematics uncertainty: $$ H(t_{i\tau}) = - \sum_{(u,v) \in \mathcal{G}} p_\omega^{(u,v)}(t_{i\tau} \mid z_{i\tau}) \log p_\omega^{(u,v)}(t_{i\tau} \mid z_{i\tau}) $$ High entropy flags ambiguous branch points in the future space, whereas low entropy indicates deterministic physics or constrained flow. By actively querying or placing constraints at the spatio-temporal coordinates with maximal entropy, the system rapidly collapses the possibility distribution with minimal sparse input.

Loss & Training

The framework adopts a two-stage training scheme. First, the encoder \(E_\phi\) and point-wise decoder \(D_\psi^p\) are trained jointly for 460k steps with goal sparsity decaying from 0.5 to 0.01. Latent vectors \(z_{i\tau}\) are squashed with \(\tanh\) and injected with Gaussian noise (\(\sigma = 10^{-5}\)) to regularize the latent manifold. In the second stage, \(E_\phi\) is frozen, while the density estimator \(D_\omega^d\) (optimized via \(\mathcal{L}_{\text{grid}}\)) and the full decoder \(D_\theta\) (optimized via trajectory regression and flow matching) are trained independently for approximately 100k steps. All models use AdamW with learning rate \(10^{-4}\), a global batch size of 256, and bfloat16 mixed precision.

Key Experimental Results

Main Results

On the public OpenVid-1M benchmark, GARFIELD is compared against leading video generation models and trajectory forecasting baselines for open-set motion planning. Five samples are drawn per model and the best trajectory is evaluated using normalized Endpoint Error (EPE), Percentage of Correct Keypoints (PCK@10%, PCK@1%), Final Distance Error (FDE), and Latency.

Method	Type	Sparse Goals	EPE ↓	PCK@10% ↑	PCK@1% ↑	FDE ↓	Latency (s) ↓
CogVideoX [49]	Video Generation	-	0.025	0.936	0.601	0.046	178
LTX-Video [15]	Video Generation	-	0.019	0.960	0.648	0.023	39
Motion-I2V [37]	Motion Prior	4	0.033	0.927	0.412	0.044	13.2
Motion-I2V [37]	Motion Prior	16	0.018	0.976	0.587	0.025	13.2
Track2Act [7]	Trajectory	- (Req. Goal RGB)	0.033	0.967	0.100	0.038	4.10
FPT [5]	Explicit Distribution	4	0.112	0.669	0.109	0.169	0.60
FPT [5]	Explicit Distribution	16	0.103	0.671	0.150	0.113	0.60
GARFIELD (Ours)	Explicit Kinematics	4	0.014	0.969	0.795	0.018	0.40
GARFIELD (Ours)	Explicit Kinematics	16	0.012	0.977	0.821	0.015	0.40

Ablation Study

The ablation evaluations highlight the critical role of structured latent factorizations, latent dimension capacity, and decoder configurations across sparse and dense constraints.

Config	Conditioning	EPE ↓	FDE ↓	Note
Global Latent	\(b_T/T = 1\%\)	0.479	0.482	Missing spatial-temporal factorization collapses localization
Entangled Latent	\(b_T/T = 1\%\)	0.509	0.510	Unconstrained representation degrades trajectory fidelity
GARFIELD (Structured)	\(b_T/T = 1\%\)	0.011	0.015	Factorized point-wise supervision enables accurate dynamics
Latent dim=16	\(b_T/T = 90\%\) (OOD)	0.009	-	Compact latent generalizes well under dense constraints
Latent dim=64 (Default)	\(b_T/T = 90\%\) (OOD)	0.008	-	Optimal balance between expressive capacity and generalization
Latent dim=256	\(b_T/T = 90\%\) (OOD)	0.019	-	Large latents overfit under extreme dense conditioning
Point-wise Decoder \(D_\psi^p\)	\(\|b_T\| = 2\)	0.031	0.037	Degrades under severe multi-modality due to point independence
Full Decoder \(D_\theta\)	\(\|b_T\| = 2\)	0.028	0.032	Cross-attention resolves inter-point temporal dependencies
Point-wise Decoder \(D_\psi^p\)	\(\|b_T\| = 16\)	0.013	0.017	Comparable once constraints collapse distribution
Full Decoder \(D_\theta\)	\(\|b_T\| = 16\)	0.013	0.017	Preserves high precision across collapsed regimes

Execution latencies across individual modules: Encoder \(E_\phi\) takes 3.0 ms, deterministic density estimator \(D_\omega^d\) takes only 0.8 ms, point-wise decoder \(D_\psi^p\) takes 37.0 ms, and the generative full decoder \(D_\theta\) takes 330.4 ms.

Key Findings

• Trajectory generation latency drops from 178 seconds (CogVideoX) to 0.4 seconds (GARFIELD), delivering a 97× to 445× inference speedup while improving tracking precision (EPE drops from 0.025 to 0.012).
• Spatial-temporal factorization is indispensable: omitting structured factorization (Global or Entangled) triggers complete degradation, with EPE worsening forty-fold (from 0.011 to 0.509).
• The deterministic density estimator predicts distributions in 0.8 ms that closely match 100-step Monte-Carlo sample heatmaps in Jensen-Shannon divergence and Wasserstein distance, enabling instant interactive feedback.
• Active conditioning guided by Shannon entropy outperforms ground-truth error oracles, reaching target planning precision with roughly half the required constraints.

Highlights & Insights

• Grounding the Schrödinger Wavefunction Collapse in Kinematics: Translating quantum state superposition and measurement-driven collapse into conditional generative modeling yields an intuitive, mathematically grounded foundation for scene dynamics under uncertainty.
• Dual-branch Decoder Harmonizes Latency and Fidelity: Combining a 0.8 ms deterministic classification head for density estimation with a joint transformer generator for coherent trajectory rollout overcomes the classic dilemma between inference speed and multi-modal consistency.
• Entropy-driven Active Exploration: Utilizing non-parametric spatial entropy to automatically select informative spatio-temporal control points provides a general paradigm transferable to interactive robotic demonstration and autonomous vehicle trajectory intervention.

Limitations & Future Work

• 2D Planar Discretization: Density estimation is currently parameterized over a \(20 \times 20\) 2D grid on the image plane, exhibiting reduced fidelity when handling significant out-of-plane depth motion or abrupt camera ego-rotations.
• Absence of Explicit 3D Rigid-body Constraints: Because training supervision derives primarily from 2D optical trackers, the model occasionally outputs unphysical trajectories during complex object collisions or fluid interactions.
• Future Directions: Extending the latent formulation into 3D Gaussian Splatting or voxelized occupancy representations, combined with differentiable physics simulators, will establish fully embodied 3D world dynamics.

Related Work & Insights

• vs CogVideoX / LTX-Video (Video World Models): Video diffusion models implicitly couple appearance rendering with dynamics, incurring massive compute overhead (39-178 seconds per sequence); GARFIELD models kinematics directly, achieving superior planning accuracy with 97× faster sampling.
• vs Track2Act: Track2Act requires a full RGB goal image that is seldom available prior to task completion; GARFIELD accommodates arbitrary spatio-temporally sparse goal points.
• vs Flow Poke Transformer (FPT): FPT restricts motion outputs to parametric Gaussian Mixture Models across single-step horizons; GARFIELD implements non-parametric density estimation across 32 continuous temporal steps for up to 64 coordinated entities.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant integration of sampling-free density decoding and joint trajectory sampling within a unified probabilistic latent space.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning open-set video planning, robotic manipulation, pedestrian forecasting, and calibration metrics.
• Writing Quality: ⭐⭐⭐⭐⭐ Lucid formulation with compelling physics analogies and rigorous experimental validation.
• Value: ⭐⭐⭐⭐⭐ Establishes an efficient, uncertainty-aware foundation for robotic planning, video generation guidance, and interactive autonomous navigation.