Multi4D: High-Fidelity Dynamic Gaussian Splatting via Multi-Level Competitive Allocation¶

Conference: ECCV 2026
arXiv: 2606.22197
Code: https://batfacewayne.github.io/Multi4D.io/ (Project Page)
Area: 3D Vision / Dynamic Gaussian Splatting / Dynamic Scene Reconstruction
Keywords: Dynamic Gaussian Splatting, 4D Reconstruction, Competitive Allocation, Static-Dynamic Decomposition, 4D Segmentation

TL;DR¶

Multi4D explicitly decomposes a dynamic scene into three subsets of Gaussians: "static structure + persistent dynamic geometry + transient appearance", allowing them to competitively explain photometric residuals under shared rasterization. This concurrently preserves long-term motion consistency and high-frequency appearance details. Relying on 25\(\times\) fewer dynamic Gaussians than 4DGS, it achieves SOTA rendering quality and real-time frame rates, while casually enabling 4D segmentation with a 10\(\times\) speedup.

Background & Motivation¶

Background & Limitations of Prior Work: 3DGS has driven explicit Gaussians and differentiable rasterization to become a mainstay of real-time novel view synthesis. However, extending it to dynamic scenes encounters a fundamental conflict between physical plausibility (motion and correspondence) and visual fidelity (rendering quality). Prior methods are split into two opposing camps, each with severe drawbacks. The first category is the deformation-based approach: maintaining a fixed set of canonical Gaussians and warping them over time using neural deformation networks, explicit trajectories, or feature grids. Because temporal correspondence is strictly enforced, it naturally preserves Gaussian identities, making it highly suitable for downstream semantic embedding and tracking. However, its deformation field tends to group adjacent motions, over-smoothing high-frequency dynamics (motion over-factorization). It also frequently misinterprets complex appearance changes (specularities, illumination drift) as physical motion, creating spurious geometric distortions to minimize photometric errors. Crucially, querying the deformation network for every primitive at each frame leads to a linear cost scaling with the total number of Gaussians; adding primitives to capture details directly sacrifices real-time performance and VRAM. The second category is the 4D-primitive approach: modeling dynamics as 4D spatio-temporal Gaussian hyper-cylinders, which are sliced along the temporal dimension at any timestamp to yield 3D Gaussians with time-varying opacity. While this approach captures appearance details exceptionally well and accelerates rendering via temporal pre-filtering to render only active Gaussians, it conversely suffers from temporal over-parameterization. The optimization favors leveraging temporal scaling to fit photometric errors, materializing millions of extremely short-lived primitives to mimic motion. This disintegrates the geometry in fast-moving regions. Furthermore, lacking global motion priors and canonical geometry constraints, it generalizes poorly under sparse camera or monocular setups.

Key Challenge: The two schools of thought each address one side of the contradiction—deformation-based methods preserve motion consistency but lose high frequency, while 4D-primitive methods capture details but shatter geometry and lead to over-parameterization. Prior patches (coarse-to-fine deformation, frequency-aware fields, spline trajectories, temporal pruning/sensitivity analysis) merely make minor fixes within their respective camps without defying the underlying assumption that "a single monolithic representation must simultaneously explain physical kinematics and transient appearance."

Goal: The goal of this paper is to dismantle this monolithic representation assumption: as a single representation cannot achieve both, it should not be forced.

Core Idea: The core idea is to reformulate dynamic reconstruction as a competitive multi-level optimization problem—allowing multiple sets of models with different inductive biases to competitively explain the photometric residuals under a single differentiable rasterizer. Whichever bias is more suited to a certain region will reconstruct it. Specifically, the scene is decomposed into static, persistent dynamic, and transient Gaussian subsets. Their gradients are coupled via shared rasterization to achieve a residual-driven adaptive division of labor, rather than pre-assigning roles manually.

Method¶

Overall Architecture¶

Multi4D represents the scene as the union of three Gaussian subsets \(\mathcal{G}=\mathcal{G}_s\cup\mathcal{G}_d\cup\mathcal{G}_t\): the static subset \(\mathcal{G}_s\) consists of time-invariant 3D Gaussians that anchor stable structures; the persistent dynamic subset \(\mathcal{G}_d\) consists of canonical Gaussians driven by a geometry-only deformation field \(\Phi_g\) (based on HexPlane) to predict only rigid motion \((\boldsymbol{\mu}_t,\mathbf{r}_t)=(\boldsymbol{\mu},\mathbf{r})+\Phi_g(\boldsymbol{\mu},t)\), thereby maintaining long-term identity and trackability; the transient subset \(\mathcal{G}_t\) consists of short-lived 4D spatio-temporal Gaussians dedicated to modeling high-frequency appearance variations and transient geometry.

The entire pipeline is bottom-up: first, inductive biases are embedded based on the inverse relationship of "stronger expressiveness, weaker initialization" for the three subsets, followed by joint rendering in a unified differentiable rasterizer. The key lies in the fact that shared transmittance couples the gradients of the three subsets—once a subset clearly explains a certain region, the residual-driven densification of other subsets in that region is naturally suppressed. This is the physical source of "competition." Training is divided into two phases: Phase I establishes the division of labor among the three subsets "competitively" via self-supervised static-dynamic decomposition, velocity-aware periodic lifting, and mask-aware pruning; Phase II disables decomposition and lifting, using only the unified rasterizer to refine geometry and appearance. Finally, the persistent subset \(\mathcal{G}_p=\mathcal{G}_s\cup\mathcal{G}_d\) is frozen to perform efficient 4D segmentation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Multi-view/Monocular Video<br/>+ COLMAP Point Cloud"] --> B["Inverse Expressiveness Initialization<br/>+ Spatio-Temporal Rescaling"]
    B --> C["Unified Multi4D Rendering<br/>Shared Transmittance across Subsets<br/>Coupled Gradients → Competition"]
    C --> D["Self-Supervised Static-Dynamic Decomposition<br/>Learn Mask Md to Separate Foreground/Background"]
    D --> E["Velocity-Aware Periodic Lifting<br/>Gd Active Points → Gt and Inherit Momentum"]
    E --> F["Mask-Aware Utility Pruning<br/>Prune Redundancy by Visible Contribution"]
    F -->|Disable Decomp/Lifting after Phase I| G["Phase II Unified Rendering Refinement"]
    G --> H["Output: Dynamic Novel Views<br/>+ Freeze Gp for 4D Segmentation"]

Key Designs¶

1. Multi-level competitive allocation: Three subsets competing to explain residuals under shared transmittance

This is the foundation of the paper, directly targeting the conflict that "a single representation cannot handle both physical motion and transient appearance." The "competitive" process is not mystical but lands on two concrete mechanisms. First, unified rendering couples gradients: the three subsets are not rendered separately and then overlaid; instead, they are jointly sorted by depth and share transmittance accumulation in a single differentiable rasterization pass (the authors build on [Hybrid 3D-4D GS] to depth-sort all three together). Each primitive is projected to its instantaneous 3D state \(\Theta_{t,i}\) according to its active subset—\(\mathcal{G}_s\) directly uses canonical parameters, \(\mathcal{G}_d\) uses \((\boldsymbol{\mu}_t,\boldsymbol{\Sigma}_t)\) deformed by \(\Phi_g\), and \(\mathcal{G}_t\) uses time-varying geometry and opacity obtained via 4D slicing \(\sigma_t=\sigma\exp(-(t-\mu_t)^2/2\boldsymbol{\Sigma}_{4,4})\). Because transmittance is shared, once a certain subset "occupies" a sector (explaining the pixel with stable geometry), the gradients of subsequent primitives are blocked, suppressing densification and naturally curbing redundant modeling. Second, Initialization via Inverse Expressiveness creates a head start in inductive bias: the highly constrained \(\mathcal{G}_s\) is initialized with dense COLMAP points (securing the static structure first to prevent static regions from being mis-reconstructed as motion); \(\mathcal{G}_d\) starts with sparse random points, allowing dynamic geometry to grow under self-supervision; the highly expressive \(\mathcal{G}_t\), which easily overfits noise, is initialized as empty and is only populated later via the lifting mechanism. Thus, the division of labor emerges from optimization rather than human design—the deformation field only ingests rigid motion and leaves transient photometric noise to the 4D primitives, ensuring neither oversteps. Furthermore, to prevent gradient imbalance or numerical singular inversion of \(\boldsymbol{\Sigma}_{4,4}\) due to inconsistent spatial/temporal scales when jointly optimizing the three subsets, a spatio-temporal linear rescaling is performed based on camera distribution and video duration prior to training, stabilizing numerical computations.

2. Self-supervised static-dynamic decomposition: Isolating dynamics from statics without tracking labels

To ensure \(\mathcal{G}_d\) only handles "moving actors" while \(\mathcal{G}_s\) only manages the background, one must know what is moving—yet the authors do not use any ground-truth tracking labels. The approach (adopting the probabilistic mask idea from DeGauss) assigns a base mask logit \(m_i\) to each persistent dynamic Gaussian, and uses a lightweight MLP \(\mathcal{D}_m\) taking HexPlane spatio-temporal features \(\mathcal{H}(\boldsymbol{\mu}_i,t)\) to predict the time-varying offset \(m'_i(t)=m_i+\mathcal{D}_m(\mathcal{H}(\boldsymbol{\mu}_i,t))\). During Phase I, the SH colors of \(\mathcal{G}_d\) are temporarily replaced with activated mask probabilities for rasterization, yielding a continuous 2D dynamic mask \(\mathbf{M}_d\in[0,1]^{H\times W}\), where the static mask is \(1-\mathbf{M}_d\). Images are then rendered independently from the dynamic and static subsets and combined via mask composition \(\mathbf{C}_{\text{comp}}=\mathbf{M}_d\odot\mathbf{C}_d+(1-\mathbf{M}_d)\odot\mathbf{C}_s\) for early photometric supervision. Here, the clever aspect is the structural asymmetry created by initialization: \(\mathcal{G}_s\) is dense (COLMAP) and \(\mathcal{G}_d\) is sparse and random (10k points). The optimization naturally favors assigning stable contents to \(\mathcal{G}_s\), causing \(\mathbf{M}_d\) to automatically shrink to truly time-varying regions. To assist, a spatially-aware opacity penalty \(\mathcal{L}_\alpha=\lambda_\alpha\|\alpha_d-\mathbb{I}_{\mathbf{M}_d>\tau}\|_1\) suppresses persistent dynamic Gaussians projected into static regions (\(\mathbf{M}_d\approx0\)) toward zero opacity, facilitating subsequent pruning.

3. Velocity-aware periodic lifting: Injecting motion priors into transient 4D primitives rather than letting them grow blindly

\(\mathcal{G}_t\) starts empty, so how does it emerge? The answer is "lifting" from \(\mathcal{G}_d\). Once static-dynamic separation stabilizes, the deformed mask logit \(m'_i(t)\) is used as the activity score, and \(K\) candidate points are sampled from the active set \(\{g_i\in\mathcal{G}_d\mid m'_i(t)>\tau\}\) to be lifted to \(\mathcal{G}_t\) (sampling is cheap as \(\mathcal{G}_d\) remains sparse). Crucially, lifting performs Momentum Inheritance: finite differences are first used to estimate the instantaneous velocity of the parent Gaussian \(\mathbf{v}_i=(\Phi_g(\boldsymbol{\mu}_i,t+\Delta t)-\Phi_g(\boldsymbol{\mu}_i,t))/\Delta t\), which is then used to initialize the new 4D primitive. The position is set to \([\boldsymbol{\mu}_i(t)+\epsilon,\ t]^T\) (where \(\epsilon\) is slightly biased toward the camera center to prevent immediate occlusion by the parent Gaussian), and the orientation is aligned via Align to match the spatio-temporal trajectory \([\mathbf{v}_i^T,1]^T\). This step is critical: unconstrained 4D optimization is highly unstable in sparse/monocular settings; momentum inheritance provides a strong motion prior to the highly expressive transient subset, prompting it to grow along the physical motion direction right from its birth, and subsequently densify autonomously to model high-frequency appearance residuals. Ablation studies show that replacing this with random initialization drops PSNR by 0.70 dB (33.92 \(\rightarrow\) 33.22).

4. Mask-aware utility pruning: Deleting redundancy by "actual contribution to the final frame" instead of opacity

The three subsets can easily overlap and reconstruct the same region, a redundancy that opacity-based pruning fails to recognize. The authors instead use "peak visible contribution" scoring: for each Gaussian and each view \(I\), the maximum blending weight across all pixels is multiplied by a gating mask \(M(\mathbf{u})\)—using \(\mathbf{M}_d\) for \(\mathcal{G}_d\), \(1-\mathbf{M}_d\) for \(\mathcal{G}_s\), and \(1\) for \(\mathcal{G}_t\) to obtain \(w_{i,I}\). This foreground/background-aware gating ensures that persistent and static primitives count their contributions only within their respective partitions, preventing cross-set overlaps, while the transmittance of \(\mathcal{G}_t\) is sorted across the overall depth union, enabling \(w_{i,I}\) to reason through cross-set occlusions and prune transient noise hidden behind physical geometry. These are then aggregated over a time window to yield the final score:

\[s_i=\beta\cdot\max_{I\in\mathcal{I}_s}(w_{i,I})+(1-\beta)\cdot\frac{1}{|\mathcal{I}_s|}\sum_{I\in\mathcal{I}_s}w_{i,I}\]

\(\beta\) balances between "retaining occasionally highly contributing primitives" and "deleting persistently useless ones"; primitives with \(s_i<\tau_{\text{prune}}\) are directly pruned. To further suppress view-dependent overfitting, training also employs Stochastic Primitive Dropout, forcing the three subsets to collaboratively model the scene. Ablating this pruning step causes the number of dynamic Gaussians to surge from 165k to 729k, increasing storage by +145%.

Loss & Training¶

The total loss is \(\mathcal{L}_{total}=\mathcal{L}_{\text{color}}+\lambda_{sep}\mathcal{L}_{\text{sep}}+\lambda_{reg}\mathcal{L}_{\text{reg}}+\lambda_{div}\mathcal{L}_{\text{diversity}}\). Here, \(\mathcal{L}_{\text{color}}\) denotes the L1 + SSIM photometric supervision (additionally applied independently to persistent foreground and static background renders in Phase I to facilitate separation); \(\mathcal{L}_{\text{sep}}\) is the static-dynamic decomposition loss (mask composition + regional supervision, utilizing \(\gamma=0.9\) early on to weaken target signals and prevent the foreground from overfitting to the background); \(\mathcal{L}_{\text{diversity}}\) is the cross-subset diversity loss, which uses mask-weighted SSIM to penalize structural similarity between the transient rendering \(\mathbf{C}_t\) and the persistent rendering \(\mathbf{C}_p\) to discourage redundant modeling (\(\lambda_{div}=0.1\)); \(\mathcal{L}_{\text{reg}}\) aggregates mask-aware opacity, depth ordering (constraining transient geometry to lie in front of/above persistent geometry), scaling, aspect ratio, and edge-aware depth TV regularization. Training proceeds in two phases: Phase I (0–\(T_{sep}\), where \(T_{sep}=10\)k) first freezes \(\Phi_g\) for the first 2k steps to build robust canonical geometry, then unfreezes deformation + the full suite of decoupling losses + periodic lifting + subset-wise pruning; Phase II disables static-dynamic decomposition and \(\mathcal{L}_{sep}\), using only the unified rasterizer for joint refinement. The entire process takes 20k steps, converging in 1.2 hours on a single RTX 4090.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on three benchmarks: Technicolor (multi-view), Neu3D (multi-view), and NeRF-DS (monocular). Multi4D leads simultaneously in rendering quality, frame rate, and the number of dynamic Gaussians, with a particularly pronounced advantage in monocular scenes (where the 4D-primitive camp degrades severely). The table below shows the average across six Neu3D scenes (lower DSSIM is better):

Dataset	Method	PSNR\(\uparrow\)	DSSIM\(\downarrow\)	FPS\(\uparrow\)
Neu3D	4DGS (4D-primitive)	31.57	0.029	114
Neu3D	STG (4D-primitive, strongest GS baseline)	32.04	0.026	140
Neu3D	E-D3DGS (Deformation)	31.20	0.026	70
Neu3D	Multi4D (Ours)	32.30	0.026	217
Technicolor	STG	33.35	0.040	86
Technicolor	Multi4D (Ours)	34.30	0.037	161
NeRF-DS (Monocular)	STG	22.54	0.089	—
NeRF-DS (Monocular)	Def-3DGS (Deformation)	23.43	0.086	—
NeRF-DS (Monocular)	Multi4D (Ours)	23.69	0.077	—

Compared to the strongest Gaussian baseline on Technicolor, Multi4D achieves a PSNR improvement of +0.95 dB and renders at a real-time 161 FPS; it reaches 217 FPS on Neu3D. On monocular NeRF-DS, where 4DGS/STG suffer from severe degradation and floating artifacts due to the lack of overall motion priors, Multi4D consistently leads by utilizing \(\mathcal{G}_d\) to model coherent motion and \(\mathcal{G}_t\) to construct local specular highlights.

4D Segmentation (Neu3D-Mask benchmark): Multi4D achieves an average mIoU of 0.9142 and mAcc of 0.9952, outperforming TRASE (0.8932). Crucially, using only 13k dynamic Gaussians (with TRASE utilizing 624k), 32-dimensional feature rendering reaches 204 FPS, which is nearly 10\(\times\) faster than TRASE's 21 FPS.

Ablation Study¶

The table below shows the average over 4 Neu3D scenes (Cut Beef / Cook Spinach / Sear Steak / Flame Steak), with the number of dynamic Gaussians denoted as (persistent \(\mathcal{G}_d\) + transient \(\mathcal{G}_t\)):

Configuration	PSNR\(\uparrow\)	DSSIM\(\downarrow\)	Dynamic Gaussians\(\downarrow\)	Storage\(\downarrow\)
Baseline 4DGS	33.14	0.0219	4215 k	2.6 GB
w/o \(\mathcal{G}_d\) (w/o persistent dynamics)	32.78	0.0237	1139 k	727.5 MB
w/o \(\mathcal{G}_t\) (w/o transients)	32.86	0.0217	25 k	105.4 MB
w/o Periodic Lifting	33.22	0.0216	13k + 132k	184.84 MB
w/o \(\mathcal{L}_{\text{diversity}}\)	33.66	0.0203	19k + 257k	263.8 MB
w/o Mask-Aware Pruning	33.68	0.0199	70k + 659k	527.9 MB
Multi4D (Full)	33.92	0.0197	13k + 152k	214.7 MB

Key Findings¶

Both dynamic subsets are indispensable, and the division of labor holds true: Removing \(\mathcal{G}_d\) (forcing motion to be reconstructed solely by randomly initialized transient points) drops the PSNR to 32.78, disintegrates the geometry, and causes the number of primitives to explode to 1139k. Conversely, removing \(\mathcal{G}_t\) produces a highly compact representation (25k) but fails to capture high-frequency appearance, yielding a PSNR of only 32.86. This directly proves that the division of labor between \(\mathcal{G}_d\) (coherent motion) and \(\mathcal{G}_t\) (residual appearance) is genuine rather than nominal.
Momentum inheritance is the soul of the lifting mechanism: Removing velocity-awareness (replacing it with random initialization) drops performance by 0.70 dB, indicating that the motion prior provided to the 4D primitives is far more valuable than the "lifting" action itself.
Diversity loss and pruning primarily serve to suppress over-parameterization: Removing \(\mathcal{L}_{\text{diversity}}\) increases dynamic Gaussians by +67% (276k vs. 165k), and removing pruning causes them to skyrocket to 729k (+145% storage). These mechanisms serve as critical gateways to prevent temporal over-parameterization from recurring; removing them slightly inflates the PSNR artificially but creates bloated representations and degrades persistent motion quality.
Compactness = Decomposition quality, not just saving storage: Compared to 4DGS, Multi4D utilizes only 165k dynamic Gaussians (a 25\(\times\) reduction), reduces the model size from 2.6 GB to 214.7 MB, and accelerates training by 4.6\(\times\) (from 5.5 hours to 1.2 hours) while improving PSNR. The authors emphasize that excessive transient parameterization disrupts decomposition, lowering PSNR and deteriorating persistent motion.
Insensitive to hyperparameters: Sweeping parameters for \(\mathcal{L}_\alpha\), depth TV, lifting sample size \(K\), and mask threshold \(\tau\) reveals that \(\Delta\)PSNR is mostly within \(\pm0.1\) (except for extreme values, such as \(\tau=0.5\) which drops by 0.28). This demonstrates that the final allocation relies on self-correction via competition, densification, and pruning, allowing a single set of fixed parameters to work across all scenes.

Highlights & Insights¶

Flipping "who builds what" from manual assignment to optimization emergence: The most inspiring aspect is that the division of labor relies not on predefined labels, but on two physical mechanisms: "shared transmittance coupling gradients + inverse expressiveness initialization." These allow the subsets to compete and organize of their own accord. This is more elegant and robust than many hard-coded static/dynamic separation methods.
"Inverse Expressiveness Initialization" is a transferable trick: Giving highly capable modules a weak initialization and weaker modules a strong initialization creates a head start in inductive bias. This effectively prevents high-capacity modules from overfitting to noise or monopolizing details that stable modules should otherwise reconstruct. This concept is transferable to any representation learning scenario involving multi-branch architectures with uneven capacities.
Momentum inheritance provides a safeguard for unstable 4D optimization: Unconstrained 4D Gaussians are highly prone to collapsing in sparse/monocular scenarios. Using the finite-difference velocity of parent Gaussians to initialize the position and orientation of child primitives introduces physical priors for free, acting as the key to resolving monocular degradation.
Compact representations benefit downstream semantics: Because the persistent subset has only 13k highly stable Gaussians, 4D segmentation can be performed by freezing this subset and applying contrastive feature distillation. This avoids transient noise and identity drift, while yielding a 10\(\times\) speedup—proving that "doing decomposition right for reconstruction" directly translates to "saving computation for downstream semantics."

Limitations & Future Work¶

Authors' acknowledgment: Current efforts rely solely on optimization-driven compactness to reduce the number of primitives, without employing explicit attribute compression (such as Gaussian quantization). Although 165k dynamic Gaussians is 25\(\times\) fewer than 4DGS, the 214.7 MB storage footprint still has room for compression. Future work could explore post-training deformation distillation, Gaussian quantization, or lightweight deformation parameterization to further translate "structural compactness" into "storage efficiency."
Dependence on dense COLMAP initialization: The structural asymmetry of the static-dynamic decomposition relies on "\(\mathcal{G}_s\) starting with dense COLMAP points and \(\mathcal{G}_d\) with sparse random points." If COLMAP fails in textureless or violently moving scenes, the static backbone becomes unstable, loosening the static-dynamic separation prior (this degradation is not extensively tested in the paper).
Complexity of three subsets + multi-stage scheduling + multiple losses: Each subset requires its own Adam optimizer, learning rates, and densification/pruning strategies, combined with a two-phase schedule and over ten regularization terms. Although the hyperparameters are proven to be robust, the engineering implementation and tuning costs are notably higher than those of monolithic 4DGS, raising the bar for reproducibility.
Avenues for improvement: Future work could explore extending the "competitive allocation" to subsets with more diverse inductive biases (e.g., explicit non-rigid deformation subsets) or replacing the MLP for dynamic masks with feed-forward prediction to further boost speed.

vs. Deformation Camp (4DGaussian / Def-3DGS / E-D3DGS): These methods use a single canonical Gaussian set and a deformation field to model all dynamics. This enforces temporal correspondence but over-smooths high-frequency details, and the per-frame network query cost scales linearly with the number of Gaussians. Multi4D lets \(\mathcal{G}_d\)'s deformation field focus exclusively on rigid motion and offloads high-frequency appearances to \(\mathcal{G}_t\), preserving trackable identities without sacrificing details, albeit at the cost of maintaining three subsets.
vs. 4D-Primitive Camp (4DGS / STG): These methods slice 4D spatio-temporal Gaussians to capture details, yielding high quality but suffering from temporal over-parameterization, fractured geometry in fast-moving regions, and severe monocular degradation. Multi4D retains the expressiveness of 4D primitives but uses them only to model "residuals that geometry cannot explain." By leveraging momentum inheritance and mask-aware pruning, it reduces the dynamic Gaussian count from 4.2M to 165k while capturing monocular robustness—essentially yielding the best of both worlds.
vs. DeGauss: Adopts its self-supervised static-dynamic probabilistic mask decomposition, but DeGauss performs a binary static-dynamic split for distractor-free reconstruction. Multi4D extends this to a three-level split (static / persistent dynamic / transient) and lets them compete under unified rendering.
vs. TRASE (4D Segmentation): TRASE performs contrastive feature distillation on the complete dynamic representation, requiring 624k Gaussians and rendering at 21 FPS. Multi4D performs distillation solely on the compact persistent subset (13k), avoiding transient noise, achieving higher mIoU, and speeding up rendering by nearly 10\(\times\)—demonstrating that "getting decomposition right first" directly benefits downstream semantic tasks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Reformulating dynamic reconstruction as a "multi-level subset competition to explain residuals" breaks past the monolithic representation assumption. The mechanisms (shared transmittance coupling, inverse expressiveness initialization, and momentum inheritance) are self-consistent and highly insightful.
Experimental Thoroughness: ⭐⭐⭐⭐ Solid evaluation across three benchmarks and three tasks (multi-view, monocular, and segmentation) with sound ablations and sensitivity analyses. However, stress testing under COLMAP failure scenarios is missing, and metrics like LPIPS are left in the supplementary materials.
Writing Quality: ⭐⭐⭐⭐ Clear presentation of key conflicts, structured method sections, and complete figures and tables. It is somewhat notation-dense, and certain mechanisms (such as the gradient coupling in competition) require supplementary details to fully grasp.
Value: ⭐⭐⭐⭐⭐ Achieving SOTA quality + real-time performance + 4.6\(\times\) faster training with 25\(\times\) fewer dynamic Gaussians, while casually enabling a 10\(\times\) speedup for 4D segmentation, represents strong utility and impact.