Learning Video Dynamics with Predictive Differentiable Rendering

Conference: ECCV2026
arXiv: 2606.31050
Code: To be confirmed
Area: 3D Vision
Keywords: Video Prediction, Differentiable Rendering, 2D Gaussian Splatting, Continuous Representation, Plug-and-Play Adapter

TL;DR

This paper proposes the Predictive Differentiable Rendering (PDR) framework, which appends a lightweight PredGS adapter to existing pixel-space predictors to map coarse predictions to 2D Gaussian parameters. These parameters are rendered into continuous-space predictions via a CUDA-accelerated differentiable renderer and then fused with pixel predictions, significantly alleviating the over-smoothing issue of deterministic video prediction with minimal computational overhead.

Background & Motivation

Video prediction—predicting future frames based on observed frames—is fundamental to many practical applications such as weather forecasting, traffic flow modeling, and human motion analysis. Recently, deterministic prediction methods based on architectures like ConvLSTM, SimVP, and TAU have made significant progress. However, almost all deterministic methods operate in the discrete pixel space and use pixel-wise Mean Squared Error (MSE) as the optimization objective. The MSE loss naturally encourages the model to take the expectation over all possible future scenarios, leading to severe over-smoothing in long-term predictions: blurry motion boundaries, lost texture details, and degraded inter-frame consistency. This bottleneck is not due to weak architectures, but rather the intrinsic limitation of MSE loss in the discrete pixel space.

Meanwhile, 3D Gaussian Splatting (3DGS) has demonstrated a powerful ability to represent visual information continuously using a small number of Gaussian primitives in scene reconstruction and novel view synthesis. Inspired by this, 2D Gaussian representation has been successfully applied to continuous image representation and super-resolution, but its integration into video prediction—a sequential modeling task—remains unexplored. The key challenges lie in: (i) how to introduce continuous-space modeling without modifying the architecture of existing predictors; (ii) how to efficiently perform differentiable rendering from Gaussian parameters to pixels under real-time inference constraints; (iii) how the continuous representation and loss design should collaborate to maximize benefits.

The Core Idea of this work is to explicitly decompose deterministic video prediction into a two-branch paradigm consisting of "coarse prediction in discrete pixel space + fine-grained rendering in continuous 2D Gaussian space" (PDR). A plug-and-play lightweight adapter, PredGS, maps coarse predictions to Gaussian parameter sets for each frame. These are projected back via a CUDA-accelerated differentiable renderer, predgsplat, to generate continuous-space predictions, which are then adaptively fused with the pixel predictions. Finally, \(L_1\) + SSIM is used instead of MSE to fully unleash the potential of the continuous representation.

Method

Overall Architecture

The design concept of PDR is to avoid replacing or modifying existing pixel-space predictors. Instead, a lightweight continuous-space branch is concatenated to their outputs, forming a dual-domain collaborative prediction. Given an observed frame sequence \(X_{1:T}\), a pixel-space predictor \(F_p\) (e.g., TAU) first generates a set of coarse prediction frames \(\tilde{X}_{T+1:T+\tau}\). Then, the lightweight adapter PredGS takes the coarse predictions as input to predict a set of 2D Gaussian parameters (position \(\mu\), scale \(s\), rotation \(\theta\), amplitude \(a\)) for each frame. These parameters are projected back to the pixel grid via a CUDA-accelerated differentiable renderer predgsplat, producing continuous-space predictive maps \(Y_{T+1:T+\tau}\). Finally, a learnable fusion function \(\Phi\) (channel concatenation followed by \(1\times 1\) convolution) integrates the coarse pixel predictions and continuous-space predictions into the final output.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Observed Frames<br/>$X_{1:T}$"] --> B["Pixel-Space Predictor $F_p$<br/>Any backbone such as TAU / SimVP"]
    B --> C["Coarse Predictions<br/>$\tilde{X}_{T+1:T+\tau}$"]
    C --> D["PredGS Adapter<br/>Avg Pooling + 4× MLP Heads"]
    D --> E["$N$ 2D Gaussians per frame<br/>$\mu, s, \theta, a$"]
    E --> F["predgsplat Differentiable Renderer<br/>CUDA-accelerated, supports any channel"]
    F --> G["Continuous-Space Predictions<br/>$Y_{T+1:T+\tau}$"]
    C --> H["Learnable Fusion $\Phi$<br/>Channel Concat + $1\times 1$ Conv"]
    G --> H
    H --> I["Final Predictions<br/>$\hat{X}_{T+1:T+\tau}$"]

Key Designs

1. PredGS Adapter: A Plug-and-Play Discrete-to-Continuous Bridge Block

The output of the pixel-space predictor is a conventional 5D tensor (\(B\times T\times C\times H\times W\)). PredGS first applies spatial average pooling to reduce the resolution and decrease subsequent computational overhead. Then, four independent lightweight MLP heads regress the four parameters for \(N\) Gaussian primitives per frame: \(\text{MLP}_\mu\) outputs the 2D position of each Gaussian center on the normalized coordinates \([-1,1]\), \(\text{MLP}_s\) outputs the anisotropic radius, \(\text{MLP}_\theta\) outputs the planar rotation angle, and \(\text{MLP}_a\) outputs the intensity vector of \(C\) channels. Activation functions are chosen based on the physical meaning of each parameter: \(\tanh\) is used for \(\mu\) and \(\theta\) to constrain them within \([-1,1]\) and \([-\pi/2, \pi/2]\) respectively, while \(\text{sigmoid}\) is used for \(s\) and \(a\) to ensure non-negativity. The MLP hidden layers are usually only 64-256 dimensions, making the parameters and FLOPs introduced by the adapter negligible. The core elegance is that it does not touch any internal structure of the backbone, and can even be disabled during inference to degenerate into a pure pixel predictor, allowing it to seamlessly latch onto any architecture like TAU, SimVP, or ConvNeXt.

2. predgsplat: Efficient Differentiable 2D Gaussian Renderer for Arbitrary Channels

Re-projecting the predicted Gaussian parameters back to the pixel space requires a differentiable renderer. Existing implementations (such as GaussianSR) are slow due to repeated affine transformations and inefficient memory access. predgsplat introduces two key optimizations: First, it eliminates expensive per-Gaussian affine transformations by pre-defining a normalized baseline grid and directly offsetting the predicted coordinates \(\mu\) on this grid, completely avoiding matrix multiplications. Second, it designs a fully vectorized CUDA kernel to parallelize rasterization across both Gaussian instances and channel dimensions. The core operation accumulates the contribution of all Gaussians at each pixel position \(p\): \(x(p)=\sum_{i} a_i \cdot f_i(p|\mu_i,\Sigma_i)\). The renderer supports an arbitrary number of channels \(C\) (not limited to RGB), which is crucial for multivariable meteorological or traffic field data. Together, these two optimizations make the rendering speed 10 times faster than the baseline, reaching 1149 FPS at a \(256\times 256\) resolution, fully satisfying real-time inference requirements.

3. Gaussian Coordinate Initialization: Spatial Uniform Prior for Stable Training

Directly initializing the weights of the coordinate prediction \(\text{MLP}_\mu\) randomly causes early training gradients to be dominated by random noise, leading to unstable convergence. This paper systematically investigates four initialization strategies: uniform grid sampling, Harris corner sampling, brightest pixel sampling, and random uniform sampling. Ablation studies show that uniform grid initialization is the most stable—it provides a spatially uniform Gaussian prior distribution, allowing the Gaussian primitives to evenly cover the entire image plane in the early stages and subsequently move toward regions requiring detailed modeling guided by gradient signals. Although convergence is still achieved without explicit initialization (outperforming TAU), grid initialization provides faster convergence and lower final errors.

4. L1+SSIM Hybrid Loss: The Activation Key for Continuous Representation

This is the most counter-intuitive finding of the paper. Using the continuous Gaussian representation alone (PDR + \(L_2\) loss) does not bring improvement, but rather performs worse than the TAU baseline (\(9.4\%\) increase in MSE on TaxiBJ). Using the \(L_1\) + SSIM loss alone (TAU + \(L_1\) + SSIM) yields only minor improvements (\(0.6\%\) decrease in MSE). However, their combination (PDR + \(L_1\) + SSIM) produces a striking synergistic effect: a \(9.7\%\) decrease in MSE on TaxiBJ, and a \(29.0\%\) decrease in LPIPS on KTH. This demonstrates that continuous Gaussian representation is not a panacea—it requires losses like \(L_1\) + SSIM that preserve high-frequency structures to guide the parameter space toward sharp details. Conversely, the MSE loss encourages the continuous representation to regress to the mean, erasing the expressive power of the Gaussian primitives. Both must be used simultaneously to unlock their full potential. The loss function is formulated as \(\mathcal{L} = \lambda \mathcal{L}_1 + (1-\lambda)\mathcal{L}_{\text{SSIM}}\), where \(\lambda\) is set to 0.5.

Loss & Training

The framework is trained end-to-end, jointly optimizing the pixel predictor, PredGS adapter, and differentiable renderer. The objective is the \(L_1\) + SSIM hybrid loss (\(\lambda = 0.5\)), optimized using AdamW (weight decay \(1\times 10^{-2}\)). The number of training epochs is kept identical to the corresponding TAU baseline for a fair comparison. The number of Gaussians \(N\) and kernel size \(K\) are configured per dataset: low-resolution datasets (TaxiBJ \(32\times 32\), WeatherBench \(32\times 32\)) use \(N=300\), \(K=15\) and \(2\times\) downsampling; high-resolution datasets (KTH \(128\times 128\), Human3.6M \(256\times 256\)) use \(N=400\), \(K=51\) and \(8\times\)/\(16\times\) downsampling.

Key Experimental Results

Main Results

Dataset	Metric	TAU (Baseline)	PDR (Ours)	Gain
TaxiBJ (4→4)	MSE↓	0.3108	0.2807	-9.7%
TaxiBJ (4→4)	MAE↓	14.93	14.29	-4.3%
WeatherBench T2m (12→12)	MSE↓	1.162	1.071	-7.8%
WeatherBench T2m (12→12)	MAE↓	0.6707	0.6353	-5.3%
Human3.6M (4→4)	SSIM↑	0.9839	0.9857	+0.2%
Human3.6M (4→4)	LPIPS↓	0.02783	0.02192	-21.2%
KTH (10→20)	SSIM↑	0.9086	0.9145	+0.6%
KTH (10→20)	LPIPS↓	0.22856	0.16237	-29.0%

Ablation Study

Configuration	TaxiBJ MSE	KTH LPIPS	Description
TAU+L2+reg (Baseline)	0.3108	0.22856	Original TAU
TAU+L1+SSIM	0.3088 (-0.6%)	0.20216 (-11.6%)	Only changing loss, limited improvement
PDR+L2+reg	0.3400 (+9.4%)	0.21850 (-4.4%)	Continuous representation + \(L_2\) performs worse
PDR+L1+SSIM	0.2807 (-9.7%)	0.16237 (-29.0%)	Continuous + correct loss = maximum gain
PredGS w/o init	0.2893	—	Still outperforms TAU without init
PredGS w/ grid init	0.2807	—	Grid init performs best
Fusion: Add	0.2998	—	Simple addition yields limited gains
Fusion: Concat+1×1 Conv	0.2807	—	Learnable fusion performs best

Key Findings

• Synergy between loss and representation is the core conclusion: The continuous Gaussian representation is not a silver bullet on its own and requires the \(L_1\) + SSIM loss to unleash its potential. Training PDR with MSE yields worse results than the pure pixel baseline, presenting a vital engineering insight.
• PredGS brings consistent improvements across 11 backbones (e.g., reducing MLP-Mixer MSE by \(8.05\%\) and ConvNeXt MAE by \(6.72\%\) on WeatherBench), demonstrating the architecture-agnostic generality of the adapter.
• Comparison with diffusion methods: On KTH \(64\times 64\), PDR's SSIM (\(0.825\)) outperforms all diffusion methods, and it achieves the highest PSNR (\(26.69\)), while the inference speed is 33 to 1460 times faster than diffusion methods.
• The rendering efficiency is extremely high, achieving 1149 FPS at \(256\times 256\). Comprehensive optimizations (base_grid + CUDA) deliver up to a \(16\times\) speedup.
• Distribution analysis reveals that PDR recovers \(56\%\) of the kurtosis gap and \(57\%\) of the diversity gap between TAU and ground truth (GT), demonstrating that the continuous representation selectively preserves salient motion patterns.
• Regarding temporal consistency, PDR's TWE lies between that of TAU (over-smoothed \(\rightarrow\) overly low TWE) and GT (real noise), showing that it does not introduce flickering artifacts while restoring sharpness.

Highlights & Insights

• Dual-Domain Collaborative Paradigm: Without modifying the backbone or significantly increasing inference latency, appending a lightweight Gaussian branch at the tail end yields sharp details. This 'plug-in' concept can be generalized to other pixel-level prediction tasks like super-resolution or optical flow estimation.
• Discovery of the 'Continuous Representation + Correct Loss' synergy: While PDR + \(L_2\) is subpar and TAU + \(L_1\) + SSIM is average, PDR + \(L_1\) + SSIM achieves the best performance. This observation directly guides future research utilizing continuous representations in video prediction.
• Coordinate Offset Trick in predgsplat: Eliminating per-Gaussian affine transformations and directly applying offsets to a normalized grid is simple yet contributes a \(1.3\text{--}3.4\times\) speedup, serving as a valuable reference for other differentiable renderers.
• Renderer Supporting Arbitrary Channels: This design is critical for non-RGB modalities such as multivariable meteorological or traffic fields, vastly expanding the applicability of the method.
• Speed Comparison with Diffusion Methods (PDR at 146 FPS vs. ARFree at 4.41 FPS) shows that deterministic prediction remains indispensable for real-time scenarios. PDR elevates the perceptual quality of deterministic predictions to a new height.

Limitations & Future Work

• The number of Gaussians \(N\) and kernel size \(K\) are currently dataset-level fixed hyperparameters requiring manual tuning. Designing an adaptive Gaussian allocation mechanism (such as density-guided dynamic growth and pruning) could further improve generalization and efficiency.
• Experiments are only conducted on four datasets, lacking validation on large-scale, high-resolution, long-term video prediction benchmarks (such as Cityscapes, Kinetics-400, and VEDAI).
• PDR remains a deterministic prediction framework and cannot model multimodal future distributions like diffusion methods. It has inherent limitations in scenarios requiring diversity (e.g., future weather sampling, pedestrian trajectory prediction).
• The temporal consistency analysis only uses a single metric (TWE), and the authors acknowledge that TWE must be interpreted in conjunction with sharpness; a more systematic evaluation of temporal jitter or flickering is still lacking.
• Currently, PredGS uses a fixed number of Gaussian primitives, treating sparse and dense regions of content identically, which may result in representation redundancy.

Related Work & Insights

• vs Pixel-Space Predictors like TAU / SimVP: These methods optimize solely in the discrete pixel space using MSE, inevitably generating blurry predictions. PDR appends a continuous Gaussian branch to their outputs, substantially enhancing details without any architectural modifications.
• vs Diffusion-based Prediction Methods (PreDiff, DiffCast, ARFree): Diffusion methods generate high-quality predictions through iterative denoising but suffer from extremely slow inference (e.g., ARFree runs at only 4.41 FPS on KTH). In contrast, PDR is a single-feedforward deterministic predictor (146 FPS) that significantly narrows the gap in perceptual metrics like LPIPS.
• vs 3DGS/2DGS (GaussianImage, GaussianSR): Existing 2DGS works focus on static image representation or super-resolution. PDR is the first to extend 2DGS to temporal dynamic modeling for video prediction, implementing a differentiable renderer that supports arbitrary channels and CUDA acceleration.
• vs STCGS: STCGS optimizes 3DGS per sequence independently before performing temporal prediction, which is computationally expensive and non-end-to-end. PDR is a single-forward end-to-end framework without per-sequence optimization overhead.

Rating

• Novelty: ⭐⭐⭐⭐ The idea of introducing 2DGS to video prediction is natural yet remained unexplored. The design of 'plug-and-play adapter + differentiable renderer' is clean and elegant. The discovery of the loss-representation synergy provides a valuable insight.
• Experimental Thoroughness: ⭐⭐⭐⭐ The main experiments cover four datasets with comprehensive ablations (loss, number of Gaussians, kernel size, initialization, fusion strategies, and backbone generalization). The comparison with diffusion methods is solid, though validation on large-scale, high-resolution benchmarks is missing.
• Writing Quality: ⭐⭐⭐⭐⭐ The paper is well-structured with a logically progressive motivation. The key conclusion (the synergy between loss and representation) is cross-validated across multiple tables and ablations. The narrative is fluent and compelling.
• Value: ⭐⭐⭐⭐ It provides a detail-enhancement solution for deterministic video prediction with almost zero extra latency. The plug-and-play design is easy to integrate into existing systems. The main drawbacks are the lack of validation on large-scale benchmarks and the absence of an adaptive Gaussian mechanism.

Related Papers

• [CVPR 2026] MeshSplatting: Differentiable Rendering with Opaque Meshes
• [NeurIPS 2025] DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
• [CVPR 2026] Hermite Radial Basis Function for Surface Reconstruction via Differentiable Rendering
• [AAAI 2026] DeepRAHT: Learning Predictive RAHT for Point Cloud Attribute Compression
• [NeurIPS 2025] LinPrim: Linear Primitives for Differentiable Volumetric Rendering