Learning Video Dynamics with Predictive Differentiable Rendering¶

Conference: ECCV2026
arXiv: 2606.31050
Code: To be confirmed
Area: 3D Vision
Keywords: Video Prediction, Differentiable Rendering, 2D Gaussian Splatting, Continuous Representation, Plug-and-Play Adapter

TL;DR¶

This paper proposes the Predictive Differentiable Rendering (PDR) framework. By appending a lightweight PredGS adapter to existing pixel-space predictors, it maps coarse predictions to 2D Gaussian parameters. A CUDA-accelerated differentiable renderer then generates continuous-space predictions, which are fused with pixel predictions. This significantly alleviates the over-smoothing issue in deterministic video prediction with minimal computational overhead.

Background & Motivation¶

Video prediction—predicting future frames from observed frames—is fundamental to many practical applications such as weather forecasting, traffic flow modeling, and human motion analysis. In recent years, deterministic forecasting methods based on architectures like ConvLSTM, SimVP, and TAU have made significant progress. However, almost all deterministic methods operate in the discrete pixel space and use pixel-wise Mean Squared Error (MSE) as the optimization objective. The MSE loss naturally encourages the model to output the expectation over all possible futures, leading to severe over-smoothing in long-range scenarios: blurred motion boundaries, lost texture details, and degraded inter-frame consistency. This bottleneck is not due to a lack of power in the architectures, but is rather an inherent limitation of the MSE loss in the discrete pixel space.

Meanwhile, 3D Gaussian Splatting (3DGS) has demonstrated strong capabilities in representing visual information continuously using a small number of Gaussian primitives in scene reconstruction and novel view synthesis. Inspired by this, 2D Gaussian representation has been successfully applied to continuous image representation and super-resolution, but its introduction into video prediction—a sequence modeling task—remains unexplored. The key challenges lie in: (i) how to incorporate continuous space modeling without making architectural changes to existing predictors; (ii) how to achieve efficient differentiable rendering from Gaussian parameters to pixels under real-time inference requirements; and (iii) how to co-design the continuous representation and the loss for maximum benefit.

The core idea of this paper is to explicitly decompose deterministic video prediction into a dual-branch paradigm (PDR) consisting of "coarse prediction in discrete pixel space + fine rendering in continuous 2D Gaussian space". Through a plug-and-play lightweight adapter, PredGS, the coarse predictions are mapped to a set of Gaussian parameters for each frame. These are then projected into continuous-space predictions via a CUDA-accelerated differentiable renderer, predgsplat, and adaptively fused with the pixel predictions. Finally, L1+SSIM is used to replace MSE to fully unleash the potential of the continuous representation.

Method¶

Overall Architecture¶

The design of PDR does not replace or modify existing pixel-space predictors. Instead, it connects a lightweight continuous-space branch in series at the output end to form co-prediction across dual domains. Given an observed frame sequence $X_{1:T}$, the pixel-space predictor $F_p$ (such as TAU) first generates a set of coarse prediction frames $\tilde{X}_{T+1:T+\tau}$. Then, the lightweight adapter PredGS takes the coarse predictions as input to predict a set of 2D Gaussian parameters (position $\mu$, scale $s$, rotation $\theta$, amplitude $a$) for each frame. These parameters are projected back to the pixel grid via a CUDA-accelerated differentiable renderer, predgsplat, obtaining continuous-space prediction maps $Y_{T+1:T+\tau}$. Finally, a learnable fusion function $\Phi$ (channel concatenation followed by $1\times 1$ convolution) fuses the coarse pixel predictions with the continuous-space predictions into the final output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Observed Frames<br/>X_{1:T}"] --> B["Pixel-Space Predictor F_p<br/>Any backbone like TAU / SimVP"]
    B --> C["Coarse Prediction Frames<br/>X~_{T+1:T+τ}"]
    C --> D["PredGS Adapter<br/>Avg Pooling + 4× MLP Heads"]
    D --> E["N 2D Gaussians per frame<br/>μ, s, θ, a"]
    E --> F["predgsplat Differentiable Renderer<br/>CUDA Accelerated · Supports Arbitrary Channels"]
    F --> G["Continuous-Space Prediction Maps<br/>Y_{T+1:T+τ}"]
    C --> H["Learnable Fusion Φ<br/>Channel Concat + 1×1 Conv"]
    G --> H
    H --> I["Final Prediction<br/>X̂_{T+1:T+τ}"]

Key Designs¶

1. PredGS Adapter: A Plug-and-Play Discrete-to-Continuous Bridge Module

The output of the pixel-space predictor is a conventional four-dimensional tensor ($B\times T\times C\times H\times W$). PredGS first applies spatial average pooling on it to lower the resolution, reducing subsequent computational cost. It then regresses four sets of parameters for the $N$ Gaussian primitives of each frame via four independent, lightweight MLP heads: $MLP_\mu$ outputs the 2D position of each Gaussian center in normalized coordinates $[-1,1]$, $MLP_s$ outputs the anisotropic radius, $MLP_\theta$ outputs the planar rotation angle, and $MLP_a$ outputs a $C$-channel amplitude vector. Activation functions are selected for each parameter according to their physical meaning: $\mu$ and $\theta$ use tanh to restrict the range within $[-1,1]$ and $[-\pi/2, \pi/2]$, while $s$ and $a$ use sigmoid to ensure non-negativity. The hidden layers of the MLPs are typically only 64-256 dimensions, making the parameters and FLOPs introduced by the entire adapter negligible. The core ingenuity lies in the fact that it does not touch any internal structure of the backbone; at inference time, it can even be turned off, degrading to a pure pixel predictor. Therefore, it can be seamlessly attached to any architecture such as TAU, SimVP, and ConvNeXt.

2. predgsplat: Efficient Differentiable 2D Gaussian Renderer for Arbitrary Channels

Projecting the predicted Gaussian parameters back to the pixel space requires a differentiable renderer. Existing implementations (such as GaussianSR) are slow due to repeated affine transformations and inefficient memory access. predgsplat introduces two key optimizations: First, it eliminates the expensive per-Gaussian affine transformation—by predefining a normalized reference grid and directly applying the predicted coordinates $\mu$ as offsets on this grid, completely bypassing matrix multiplication. Second, it designs a fully vectorized CUDA kernel, parallelizing rasterization across both Gaussian instances and channels. The core operation accumulates the contribution of all Gaussians at each pixel position $p$: $$x(p)=\sum_i a_i\cdot f_i(p|\mu_i,\Sigma_i)$$. The renderer supports an arbitrary number of channels $C$ (not limited to RGB), which is crucial for multivariate meteorological/traffic field data. Together, these two optimizations make the rendering speed 10× faster than the baseline, reaching 1149 FPS at a resolution of $256\times256$, which fully satisfies real-time inference requirements.

3. Gaussian Coordinate Initialization: Spatially Uniform Prior for Stable Training

Directly initializing the weights of the coordinate prediction $MLP_\mu$ randomly causes the early stage training gradients to be dominated by random noise, leading to unstable convergence. This paper systematically investigates four initialization strategies: uniform grid sampling, Harris corner sampling, brightest pixel sampling, and random uniform sampling. Ablation experiments show that uniform grid initialization yields the most stable results—it provides a spatially uniform Gaussian prior distribution, allowing each Gaussian primitive to cover the entire image plane evenly in the early stages, and later move individually to regions that require fine-grained modeling based on gradient signals. Training without explicit initialization can still converge and outperform TAU, but the grid initialization provides faster convergence and lower final errors.

4. Hybrid L1+SSIM Loss: The Key to Activating Continuous Representations

This is the most counter-intuitive finding of the paper. Using the continuous Gaussian representation alone (PDR + L2 loss) not only yields no improvement but actually performs worse than the TAU baseline (with MSE increasing by 9.4% on TaxiBJ). Using the L1+SSIM loss alone (TAU + L1+SSIM) also yields only minor improvements (0.6% reduction in MSE). However, combining both (PDR + L1+SSIM) produces a striking synergistic effect: MSE drops by 9.7% on TaxiBJ, and LPIPS drops by 29.0% on KTH. This indicates that continuous Gaussian representation is not a panacea—it requires a loss like L1+SSIM that preserves high-frequency structures to guide the parameter space to converge toward sharp details. Conversely, the MSE loss forces continuous representations to regress to the mean, erasing the expressiveness of Gaussian primitives. Both must be used simultaneously to unleash their full potential. The loss formulation is $$\mathcal{L} = \lambda\mathcal{L}_1 + (1-\lambda)\mathcal{L}_{SSIM}$$, with $\lambda$ set to 0.5 across all experiments.

Loss & Training¶

The overall system is trained end-to-end, jointly optimizing the pixel predictor, the PredGS adapter, and the differentiable renderer. The objective is the hybrid L1+SSIM loss ($\lambda=0.5$), optimized using AdamW (weight decay $1\times10^{-2}$). The number of training epochs matches the corresponding TAU baseline for a fair comparison. The number of Gaussians $N$ and the kernel size $K$ are configured per dataset: low-resolution datasets (TaxiBJ $32\times32$, WeatherBench $32\times32$) use $N=300$, $K=15$ and $2\times$ downsampling; high-resolution datasets (KTH $128\times128$, Human3.6M $256\times256$) use $N=400$, $K=51$ and $8\times/16\times$ downsampling.

Key Experimental Results¶

Main Results¶

Dataset	Metric	TAU (Baseline)	PDR (Ours)	Gain
TaxiBJ (4→4)	MSE↓	0.3108	0.2807	-9.7%
TaxiBJ (4→4)	MAE↓	14.93	14.29	-4.3%
WeatherBench T2m (12→12)	MSE↓	1.162	1.071	-7.8%
WeatherBench T2m (12→12)	MAE↓	0.6707	0.6353	-5.3%
Human3.6M (4→4)	SSIM↑	0.9839	0.9857	+0.2%
Human3.6M (4→4)	LPIPS↓	0.02783	0.02192	-21.2%
KTH (10→20)	SSIM↑	0.9086	0.9145	+0.6%
KTH (10→20)	LPIPS↓	0.22856	0.16237	-29.0%

Ablation Study¶

Configuration	TaxiBJ MSE	KTH LPIPS	Description
TAU+L2+reg (Baseline)	0.3108	0.22856	Original TAU
TAU+L1+SSIM	0.3088 (-0.6%)	0.20216 (-11.6%)	Only changing loss, limited gain
PDR+L2+reg	0.3400 (+9.4%)	0.21850 (-4.4%)	Continuous representation + L2 gets worse instead
PDR+L1+SSIM	0.2807 (-9.7%)	0.16237 (-29.0%)	Continuous + correct loss = maximum gain
PredGS w/o init	0.2893	—	Still outperforms TAU without init
PredGS w/ grid init	0.2807	—	Grid init is the best
Fusion: Add	0.2998	—	Simple addition has limited effect
Fusion: Concat+1×1 Conv	0.2807	—	Learnable fusion is the best

Key Findings¶

The synergy between loss and representation is the most core conclusion: Continuous Gaussian representation is not a silver bullet on its own and requires L1+SSIM loss to unleash its full potential; training PDR with MSE actually performs worse than the pure pixel baseline, which is an important engineering insight.
PredGS consistently brings improvements across 11 different backbones (e.g., MLP-Mixer MSE decreases by 8.05%, ConvNeXt MAE decreases by 6.72% on WeatherBench), proving the architecture-agnostic generality of the adapter.
Comparison with diffusion-based methods: On KTH $64\times64$, PDR's SSIM (0.825) outperforms all diffusion methods and achieves the highest PSNR (26.69), while the inference speed is 33 to 1460 times faster than diffusion methods.
The rendering efficiency is extremely high, reaching 1149 FPS at $256\times256$, and the comprehensive optimizations (base_grid + CUDA) bring up to $16\times$ acceleration.
Distribution analysis shows that PDR recovers 56% of the kurtosis gap and 57% of the diversity gap between TAU and GT, demonstrating that continuous representation selectively preserves prominent motion patterns.
In terms of temporal consistency, PDR's TWE is situated between TAU (over-smoothed $\rightarrow$ TWE is too low) and GT (real noise), suggesting that it recovers sharpness without introducing flickering artifacts.

Highlights & Insights¶

Dual-Domain Collaborative Paradigm: Without modifying backbones or significantly increasing inference latency, a dramatic improvement in sharpness is achieved simply by appending a lightweight Gaussian branch at the backend. This "plug-in" concept can be generalized to other pixel-level prediction tasks such as super-resolution and optical flow estimation.
Discovery of the "Continuous Representation + Correct Loss" Double Negative Effect: PDR+L2 is poor, TAU+L1+SSIM is mediocre, and PDR+L1+SSIM is optimal. This finding provides direct guidance for subsequent research utilizing continuous representations in video prediction.
predgsplat's Coordinate Offset Trick: Eliminating per-Gaussian affine transformations and directly adding offsets onto a normalized grid is simple, but contributes 1.3-3.4× acceleration, which is a valuable reference for other differentiable renderers.
The Renderer Supports Arbitrary Channels: This design is crucial for non-RGB modalities such as multivariate meteorological and traffic fields, greatly expanding the application scope of the method.
Velocity Comparison with Diffusion Methods (PDR 146 FPS vs ARFree 4.41 FPS) shows that deterministic prediction still holds irreplaceable value in real-time scenarios, and PDR pushes the perceptual quality of deterministic prediction to a new level.

Limitations & Future Work¶

The number of Gaussians $N$ and the kernel size $K$ are currently dataset-level fixed hyperparameters, requiring manual tuning. Designing an adaptive Gaussian allocation mechanism (such as density-guided dynamic pruning/splitting) could further enhance generalization and efficiency.
The experiments are conducted on only 4 datasets, lacking validation on large-scale, high-resolution, long-horizon video prediction benchmarks (such as Cityscapes, Kinetics-400, VEDAI).
PDR is still a deterministic prediction method and cannot model multimodal future distributions like diffusion-based methods. It has inherent limitations in scenarios requiring diversity (such as future weather sampling or pedestrian trajectory prediction).
The analysis of inter-frame temporal consistency relies on only one metric (TWE), and the authors acknowledge that TWE needs to be interpreted in conjunction with sharpness. A more systematic evaluation of temporal jitter or flickering is still lacking.
Currently, PredGS uses a fixed number of Gaussian primitives, without differential treatment for sparse and dense content regions, which may lead to representation redundancy.

vs. Pixel-Space Predictors like TAU / SimVP: They only optimize with MSE in the discrete pixel space, inevitably generating blurry predictions. PDR appends a continuous Gaussian branch to their outputs, significantly enhancing details without architectural modifications.
vs. Diffusion-Based Prediction Methods (PreDiff, DiffCast, ARFree): Diffusion methods generate high-quality predictions through iterative denoising but suffer from extremely slow inference (e.g., ARFree runs at only 4.41 FPS on KTH). PDR is a single-forward deterministic prediction method (146 FPS) that significantly narrows the gap in perceptual metrics like LPIPS.
vs. 3DGS/2DGS (GaussianImage, GaussianSR): Existing 2DGS work focuses on static image representation or super-resolution. PDR is the first to extend 2DGS to modeling temporal dynamics in video prediction, achieving differentiable rendering that supports arbitrary channels with CUDA acceleration.
vs. STCGS: STCGS uses 3DGS to optimize each sequence independently and then perform temporal prediction, which is computationally expensive and not end-to-end. PDR is a single-forward end-to-end framework without sequence-by-sequence optimization cost.

Rating¶

Novelty: ⭐⭐⭐⭐ The idea of introducing 2DGS into video prediction is natural yet previously unexplored. The "plug-and-play adapter + differentiable renderer" design is clean and elegant, and the discovery of the loss-representation synergy is an important insight.
Experimental Thoroughness: ⭐⭐⭐⭐ The main experiments cover 4 datasets, and the ablations are comprehensive (losses, number of Gaussians, kernel size, initialization, fusion strategies, backbone generalization). The comparison with diffusion methods is also solid. However, validation on large-scale high-resolution benchmarks is still missing.
Writing Quality: ⭐⭐⭐⭐⭐ The structure is clear, and the motivation progresses logically. The most critical conclusion (the synergy between loss and representation) is cross-verified across multiple tables and ablations, presenting a smooth and compelling narrative.
Value: ⭐⭐⭐⭐ PDR provides a detail-enhancement scheme for deterministic video prediction with almost zero extra latency. Its plug-and-play design makes it easy to integrate into existing systems. The drawback is the lack of validation on large-scale benchmarks and the absence of an adaptive Gaussian mechanism.