ReSplat: Learning Recurrent Gaussian Splatting

Conference: ECCV2026
Paper: ECCV paper page
PDF: Full paper
Project: ReSplat
Code: https://github.com/cvg/resplat
Area: 3D Vision
Keywords: Gaussian Splatting, sparse-view synthesis, learning to optimize, rendering-error feedback, recurrent refinement

TL;DR

ReSplat predicts a compact Gaussian scene from posed images and recurrently corrects its Gaussians and hidden states using input-view rendering errors, improving eight-view DL3DV reconstruction from 26.21 dB at initialization to 27.70 dB without explicit test-time gradient computation while retaining approximately 1/16 of the Gaussian count of per-pixel baselines.

Background & Motivation

Feed-forward Gaussian Splatting models map images directly to renderable 3D Gaussians, avoiding thousands of optimization steps for every new scene. Methods such as MVSplat and DepthSplat therefore support fast sparse-view reconstruction, but a single forward pass must jointly solve geometry inference, cross-view fusion, and appearance prediction. Complex scenes or camera configurations outside the training distribution amplify prediction errors, while single-step models lack an explicit mechanism to check whether their reconstruction explains the input images. Per-scene 3DGS optimization can repeatedly correct predictions, but sparse observations invite overfitting, and gradient computation over many iterations is expensive.

ReSplat focuses on reusable correction steps rather than simply increasing network size. Input images are already available at test time, so rendering the current Gaussians at their cameras provides feedback without requiring ground-truth images for the novel views to be synthesized. A related bottleneck is representation size: predicting one Gaussian per input pixel produces millions of Gaussians from eight high-resolution images. Repeated 3D neighborhood interaction over this population would make refinement expensive, so initialization must account for the cost of subsequent updates.

The paper connects compact representation and error-driven updates: it first compensates for subsampling through 3D context, then lets a weight-sharing network inspect the reconstruction error at each iteration. Here, recurrence means repeated refinement of the same static scene, not temporal modeling of a dynamic video scene. Core Idea: turn input-view rendering residuals into observations for a Gaussian update network and repeatedly apply learned corrections in a compact Gaussian scene, instead of relying on a single prediction or solving gradient-based optimization at test time.

Method

Overall Architecture

The inputs are RGB images with camera intrinsics and extrinsics; the output is an explicit set of 3D Gaussians that can render novel camera views. The pipeline connects four designs: compact contextual initialization, dual-domain rendering error, global error propagation, and stateful recurrent updates. Initialization runs once; each subsequent iteration renders the input views and feeds the resulting errors to the same update network. Final novel-view rendering uses the refined Gaussian scene without rerunning reconstruction for every output camera.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Images and camera parameters"] --> B["Compact contextual<br/>initialization"]
    B --> C["Current Gaussians<br/>and hidden states"]
    C --> D["Dual-domain<br/>rendering error"]
    A -->|Input images as reference| D
    D --> E["Global error<br/>propagation"]
    E --> F["Stateful recurrent<br/>updates"]
    C -->|Current parameters and memory| F
    F -->|Next iteration| C
    F -->|Finish refinement| G["Gaussian scene and novel views"]

The model separates explicit Gaussian parameters from hidden states: the former determine rendering, while the latter retain context not fully expressed by positions, opacities, and appearance attributes. Inference updates these two kinds of scene state rather than network weights. Thus, gradient-free refers to the absence of explicit scene-parameter optimization gradients during inference, not to training without back-propagation.

Key Designs

1. Compact contextual initialization: strengthen 3D communication before relying on fewer Gaussians

Initialization builds on the DepthSplat depth architecture, but resizes its full-resolution depth predictions to 1/4 height and width before unprojecting them into a common 3D coordinate system with the camera parameters. For \(N\) input images of size \(H\times W\), the default multiview configuration creates \(M=NHW/16\) points rather than \(NHW\) per-pixel points. Each point also carries image features, with dimensionality 256 for Small and 512 for Base. The authors use a spatially central input camera as the reference coordinate system to moderate the distribution of transformations from different views. For a sequential trajectory, this typically means the middle frame; it is coordinate normalization rather than additional camera-pose estimation. Directly regressing Gaussians from isolated compressed point features cannot adequately recover the discarded detail, as Table 6c confirms. The regressor therefore uses six alternating blocks of kNN and global attention, giving points both local surface information and context across views and regions.

Local kNN attention uses 3D neighborhoods with \(k=16\) during initialization; global interaction connects more distant but related regions. Point positions become initial Gaussian centers, and a two-layer MLP decodes the remaining attributes, including opacity, covariance-related parameters, and spherical-harmonic appearance. The paper concatenates all parameters of each Gaussian into a 59-dimensional vector and initializes its hidden state with the context-aggregated feature. Compression here starts prediction from fewer 3D anchors rather than pruning a previously generated dense Gaussian population. The representation size is therefore controlled from the beginning, benefiting both rendering and recurrent refinement. However, this default formula applies to the multiview setting; the two-view experiments use a different decoding configuration and do not justify a universal 16-fold count reduction.

2. Dual-domain rendering error: inspect both color differences and structural feature differences

Each iteration renders the current Gaussian scene at every input camera and compares those images with the available inputs. Pixel error is rendered RGB minus input RGB, retaining the direction of color and detail discrepancies. A pixel unshuffle with spatial factor 4 then moves local pixel positions into channels, producing a quarter-resolution grid corresponding to the Gaussian anchors. Unlike ordinary average pooling, this does not immediately average away all residuals within a local block. A linear projection and Layer Normalization subsequently match the pixel branch to the feature branch's channel dimensionality.

Feature error uses the first three stages of an ImageNet-pretrained ResNet-18 at 1/2, 1/4, and 1/8 resolution. The features are bilinearly resized to quarter resolution and concatenated into a 256-dimensional representation, after which rendered-image features are subtracted by input-image features. The two errors are added element-wise: the pixel branch supplies direct color feedback, while the feature branch provides richer structural and contextual feedback. This is neither nearest-neighbor search in feature space nor a gradient of the loss with respect to Gaussian parameters; it encodes observed residuals as network inputs. In Table 6a, feature error alone substantially outperforms RGB error, but combining them remains better, suggesting complementary information.

3. Global error propagation: do not restrict feedback to a Gaussian's original pixel cell

The compressed error grid and initial Gaussians have the same number of entries, making aligned concatenation the simplest association. Yet one Gaussian can cover several rendered pixels and contribute to images from other cameras; its original pixel location is not its entire influence region. Reading only the error around that original grid position would miss constraints on geometry and appearance from other views. ReSplat therefore applies global attention to error tokens first, allowing each entry to aggregate residual information across all inputs before concatenation with its corresponding Gaussian parameters and hidden state. The order matters: errors are propagated before reaching the 3D update module, rather than explicitly back-projecting rendering gradients to Gaussians.

Direct global attention over all quarter-resolution tokens would still be expensive at high resolution. The implementation applies another pixel unshuffle with factor 4, temporarily reducing the grid to \(N\times H/16\times W/16\) for global attention over fewer tokens. Pixel shuffle then restores quarter resolution, preserving the entry-wise correspondence between error features and Gaussian states. This compresses the spatial grid used for attention computation, not the final Gaussian count a second time. Global attention in initialization uses the same efficient implementation, connecting compact representation with affordable contextual aggregation.

4. Stateful recurrent updates: accumulate corrections with a shared network instead of rebuilding the scene

The update module reads current Gaussian parameters, hidden states, and globally aggregated errors, using four kNN attention blocks to capture local 3D structure. This stage uses \(k=8\) to focus on nearby details, followed by lightweight increment decoders; the Gaussian-parameter update head is a four-layer MLP. Both explicit parameters and hidden states receive residual additions, as shown in the paper's Figure 2 and Equation (4):

\[ \mathbf{g}^{t+1}_j=\mathbf{g}^{t}_j+\Delta\mathbf{g}^{t}_j, \qquad \mathbf{z}^{t+1}_j=\mathbf{z}^{t}_j+\Delta\mathbf{z}^{t}_j. \]

Here, \(\mathbf{g}\) denotes renderable Gaussian parameters, \(\mathbf{z}\) denotes hidden state, \(j\) indexes Gaussians, and \(t\) indexes update iterations. The next iteration must render the updated Gaussians again rather than repeatedly reuse the initial error. Hidden states originate from initialization features and accumulate information through updates; Table 6d shows a substantial drop when they are removed despite retaining explicit Gaussian attributes. Iterations share the same update-network weights, so additional iterations do not introduce separate parameter sets or enlarge the initialization backbone. Performance empirically saturates after approximately four iterations, while one model can use different inference iteration counts to trade reconstruction latency for quality. The Gaussian count remains fixed throughout refinement; the gains do not come from adding or deleting primitives.

The 3D kNN operation also has an engineering limit: default global CUDA search works well at smaller point counts, but its quadratic complexity becomes a bottleneck with more views. The paper offers a local candidate alternative using same-view spatial neighbors and cross-view projections from nearby cameras, followed by selection using actual 3D distances. With fixed candidate size, search cost grows approximately linearly with point count; selection is exact within the restricted candidates, not a guarantee of finding globally nearest neighbors over the entire cloud.

A Worked Example

For the eight \(512\times960\) input images in Table 1, default initialization creates \(8\times512\times960/16=245{,}760\) Gaussians, rounded to 246K in the table. Per-pixel methods produce \(3{,}932{,}160\) Gaussians from the same inputs, reported as 3932K. The first iteration renders these 246K Gaussians at the eight known cameras, computes RGB and ResNet feature residuals, propagates them globally, and predicts Gaussian and hidden-state increments. The second iteration renders the corrected scene again, so the update module sees what remains unexplained rather than the original complete error. As an intuitive example, residuals around a window frame can guide learned changes to relevant Gaussian positions, shapes, and appearance; this illustrates the mechanism and is not a separately quantified window-frame experiment. PSNR in this setting progresses from 26.21 at initialization to 27.15 after one iteration, 27.51 after two, and 27.70 dB after four, always with 246K Gaussians. The final renderable scene then supports novel views: input images provide feedback, while novel-view ground truth does not participate in test-time correction.

Loss & Training

Training has two stages: train the compact initialization model, then freeze it and train only the recurrent model. The first stage combines target-view rendering supervision with edge-aware smoothness regularization on input depth maps; Equations (8)-(10) can be combined as:

\[ \mathcal{L}_{\mathrm{1st}} =\sum_{v=1}^{V}\left[\ell_1(\hat I_v,I_v)+0.5\,\ell_{\mathrm{perceptual}}(\hat I_v,I_v)\right] +0.01\sum_{i=1}^{N}\left[|\partial_x\hat D_i|e^{-|\partial_x I_i|}+|\partial_y\hat D_i|e^{-|\partial_y I_i|}\right]. \]

Here, \(V\) is the number of supervised target views, \(N\) is the number of input views, and the perceptual loss uses VGG feature space. Depth regularization requires no ground-truth depth: it encourages smooth depth in visually flat regions while weakening the penalty near image edges. The VGG perceptual distance used for training and the ResNet-18 feature difference used to encode feedback serve different purposes and should not be conflated. The second stage applies the same rendering loss to target-view renderings after every recurrent update, discounting earlier iterations with \(\gamma=0.9\) so that later predictions receive greater weight. Training randomly selects between 1 and 4 iterations rather than supervising only the final output of a fixed four-step process, supporting different inference budgets with one model. The optimizer is AdamW, and rendering uses the Mip-Splatting-based implementation in gsplat. Default Base has 223M parameters: 209M for initialization and 14M for recurrence; ablations use Small with 77M parameters, split into 62M and 15M. The supplied cache contains the main paper and references but not the referenced supplementary training details, so unverified learning rates, batch sizes, and training durations are not supplied here.

Key Experimental Results

Main Results

The following selection from Table 1 evaluates the 140-scene DL3DV benchmark with eight input views per scene at \(512\times960\) resolution. 3DGS optimizes each test scene, whereas feed-forward methods directly infer its representation; reconstruction time and subsequent single-view rendering time are separate metrics. Higher PSNR and SSIM are better, while lower LPIPS is better; times are in seconds, and Gaussian counts retain the paper's rounding.

Method / update iterations	PSNR	SSIM	LPIPS	Gaussians	Reconstruction time	Rendering time
3DGS / 4000	23.46	0.770	0.224	359K	70	0.0009
MVSplat / 0	22.49	0.764	0.261	3932K	0.129	0.0030
DepthSplat / 0	24.17	0.815	0.208	3932K	0.190	0.0030
ReSplat / 0	26.21	0.842	0.185	246K	0.311	0.0007
ReSplat / 1	27.15	0.859	0.169	246K	0.437	0.0007
ReSplat / 2	27.51	0.865	0.163	246K	0.563	0.0007
ReSplat / 4	27.70	0.868	0.160	246K	0.816	0.0007

The four-iteration model exceeds DepthSplat by 3.53 dB, but initialization already supplies 2.04 dB, with recurrence adding another 1.49 dB. The complete improvement over DepthSplat should therefore not be attributed entirely to recurrent updates. Rendering takes approximately one quarter of the per-pixel baselines' time, but reconstruction takes 0.816 seconds versus DepthSplat's 0.190 seconds, exchanging one-time reconstruction cost for quality and subsequent rendering efficiency. The paper summarizes the speed advantage over 3DGS as approximately 100-fold; the four-iteration table values give \(70/0.816\), approximately 85.8-fold rather than exactly 100-fold.

Table 3 uses a different DL3DV full-scene coverage protocol with 16 images at \(540\times960\), so its PSNR should not be directly compared with Table 1. Two-iteration ReSplat achieves 23.51 dB, 0.766 SSIM, and 0.284 LPIPS with 518K Gaussians and 1.7-second reconstruction; Long-LRM reports 22.66 dB, 0.740, 0.292, 2073K Gaussians, and 0.4 seconds. The Gaussian count advantage over Long-LRM is therefore approximately four-fold, but reconstruction is still slower, with kNN identified as one cause.

Table 4 uses two \(256\times256\) RealEstate10K images with four-fold spatial point subsampling and four Gaussians decoded per point, leaving the total count equal to per-pixel methods. ReSplat achieves 29.75 dB, 0.912 SSIM, and 0.100 LPIPS; LVSM decoder-only achieves 29.67 dB, 0.906, and 0.098, so ReSplat does not lead on every metric. In Table 5's zero-shot RealEstate10K-to-ACID evaluation, ReSplat reaches 29.87 dB, 0.864, and 0.135; this is cross-dataset evaluation of the separate two-view model.

Ablation Study

Both tables below come from Table 6 and use ReSplat-Small on DL3DV with eight input views at \(256\times448\) resolution. Their model and resolution differ from the main table, so 29.07 dB is not direct evidence of outperforming the high-resolution Base experiment. The first table corresponds to Table 6a and isolates the error information supplied to the recurrent network.

Feedback configuration (Table 6a)	PSNR	SSIM	LPIPS
Initialization only	26.77	0.865	0.142
Recurrent updates without rendering error	27.19	0.873	0.137
RGB error only	27.90	0.882	0.130
Feature error only	28.77	0.897	0.110
Concatenated RGB and feature errors	28.93	0.900	0.106
Added RGB and feature errors	29.07	0.902	0.105

The second table selects from Tables 6c and 6d, separating the initialization model's ability to tolerate compression from the inputs and interactions needed for recurrent refinement. Initialization rows all use 57K Gaussians; recurrent rows also keep a fixed count but report refined results, so they are not same-stage module-removal comparisons with initialization rows.

Stage and configuration	Source table	PSNR	SSIM	LPIPS
Full initialization	6c	26.77	0.865	0.142
Initialization without kNN attention	6c	25.30	0.833	0.178
Initialization without global attention	6c	26.33	0.856	0.150
Initialization without either attention type	6c	24.50	0.814	0.200
Full recurrent model	6d	29.07	0.902	0.105
Recurrent model without hidden state	6d	27.79	0.878	0.125
Recurrent model without kNN attention	6d	28.58	0.894	0.111
Recurrent model without global attention	6d	28.96	0.900	0.107

Key Findings

• Feedback is not optional auxiliary information: Table 6a gives 27.19 dB without error input versus 29.07 dB for the full model, a 1.88 dB gap; repeating a network alone does not explain all gains.
• Hidden states retain information beyond explicit Gaussian attributes: removing them in Table 6d loses 1.28 dB, while removing kNN or global attention loses 0.49 and 0.11 dB, respectively.
• Local 3D context is especially important for compression: removing initialization kNN attention in Table 6c costs 1.47 dB, and removing both attention types costs 2.27 dB.
• Coordinates also condition learning: Table 6b reports 29.07 dB with the middle-view frame and 28.14 dB with original COLMAP coordinates, a 0.93 dB difference; this does not establish invariance to arbitrary coordinate transformations.
• Parameter scaling is not an equivalent substitute: Table 2 reports 29.07 dB for four-iteration Small with 77M parameters versus 27.86 dB for single-step Large with 559M, a 1.21 dB gap; the conclusion is bounded by the compared architectures and training setups.

Highlights & Insights

• Feedback comes from observations already available at inference, not unavailable novel-view labels. This supports adaptation to the current scene without implying test-time network-weight updates.
• Compact initialization determines whether recurrence is affordable. The Figure 6 analysis reports that per-pixel MVSplat initialization can also be refined, but its larger Gaussian population makes subsequent updates 13 times slower, so initialization controls both starting quality and update cost.
• Errors are observations, whereas hidden states provide memory. This division can inform other explicit 3D representations, but transferring it requires a suitable association between feedback and representation elements, not merely more iterations.
• A persistent scene representation supports many inexpensive renders. Applications requesting numerous output views should measure reconstruction plus repeated rendering rather than only first-frame latency.

Limitations & Future Work

• The authors explicitly note that Gaussian count remains fixed during refinement. If initialization misses a structure, the updater can only modify existing primitives, without adaptive densification or pruning to reallocate capacity.
• Saturation after approximately four iterations is empirical rather than a convergence guarantee. Additional test-time computation need not improve results, and the authors propose more informative feedback as a future direction.
• Inputs currently form a static view set, not an online streaming system. Sliding-window initialization and continuous refinement are proposed but not validated as streaming reconstruction in this paper.
• Reviewer assessment: dependence on known camera parameters and a common coordinate system matters. The experiments use COLMAP poses, and coordinate choice affects quality, so comparable robustness to pose noise or unposed inputs is not established.
• Reviewer assessment: compact Gaussians and fast rendering do not imply minimum reconstruction latency. kNN, feedback rendering, and repeated updates all cost time; deployment reports should include view count, resolution, iteration count, and requested output-camera count.
• Reviewer assessment: the two-view configuration abandons the default 16-fold count reduction, showing that quality and representational redundancy depend on input coverage. Adaptive representation budgets paired with error feedback are a more targeted extension than unconditional compression.

Related Work & Insights

• vs DepthSplat / MVSplat: These methods predict Gaussians quickly in one step, while ReSplat adds feedback-driven refinement on top of compact contextual initialization. Comparisons with DepthSplat change both initialization and recurrence, so ablations are needed to separate their contributions.
• vs 3DGS / G3R: 3DGS optimizes each scene, and G3R guides learned updates with explicit gradients; ReSplat predicts updates directly from rendering residuals. It also avoids G3R's reliance on well-covered initial 3D points, although it still requires posed images.
• vs SplatFormer: SplatFormer refines optimized Gaussians primarily for object-centric data using a single-step non-recurrent network; ReSplat uses feed-forward initialization, error feedback, and weight-sharing multistep updates. The paper's scene-level transfer comparison is affected by point-cloud normalization and grid selection and should not be generalized to absolute superiority on every task.
• vs Long-LRM / LVSM: Long-LRM demonstrates efficient single-pass multiview reconstruction, while LVSM demonstrates view synthesis without explicit Gaussians. ReSplat combines an explicit representation with corrective feedback, but does not consistently lead in reconstruction latency or perceptual metrics.
• Research direction: Residual-guided decisions about where to densify and when to stop updating could allocate computation more effectively. This suggestion follows from the fixed Gaussian count and four-iteration saturation, rather than describing an implemented component.

Rating

• Novelty: 4/5. Compact initialization, dual-domain error feedback, and shared state updates form a coherent method, although learning to optimize and recurrent refinement have substantial precedent.
• Experimental Thoroughness: 4/5. Multiview, two-view, cross-dataset, scaling, and component experiments are covered, but streaming inputs, pose errors, and long-horizon iteration remain unvalidated.
• Writing Quality: 4/5. The architecture figure and ablations support the main mechanisms, while resolution, initialization configuration, and reconstruction versus rendering efficiency require careful distinction.
• Value: 4/5. The method provides a reusable error-feedback approach for fast explicit 3D reconstruction, with practical benefits depending on reconstruction budgets and subsequent rendering demand.