WilLaGS: Latent-Conditional 3D Appearance Fields for Robust Gaussian Splatting In-the-Wild

Conference: ECCV 2026
Paper: ECCV official page
Area: 3D Vision
Keywords: Gaussian Splatting, novel view synthesis, unconstrained image collections, latent-conditioned appearance fields, transient masking

TL;DR

WilLaGS learns a continuous global appearance space with a β-VAE, generates a Tri-Plane 3D appearance field from its latent code to modulate local illumination, and masks transient occlusions using perceptual differences between an EMA teacher and real images, reaching 25.84 dB on Photo Tourism's Sacre Coeur while providing 58 FPS rendering, appearance interpolation, and random sampling within the same scene.

Background & Motivation

Standard 3D Gaussian Splatting (3DGS) fuses evidence from multiple photos into one static scene, but viewpoint is not the only source of variation in internet photos: weather, exposure, and capture time alter appearance, while pedestrians and vehicles temporarily obscure buildings. Explaining all these differences as fixed colors or geometry can produce blur, ghosting, and floating primitives. The challenge is to distinguish a static object under different illumination from a transient object that should not be reconstructed, rather than simply discarding every inconsistent pixel.

NeRF-W and related methods absorb photometric differences with per-image appearance embeddings, while GS-W and others further model local appearance. Independently optimized embeddings, however, lack a natural sampling prior, and a single global modulation may insufficiently represent position-dependent shadows. Meanwhile, jointly trained uncertainty networks may lag behind 3DGS optimization, and external segmentation models impose category and distribution constraints. This paper brings continuous appearance modeling, spatial locality, and stable transient supervision into one training framework instead of repairing output images separately.

Core idea: constrain global appearance with a probabilistic latent space, translate it into local color conditions through 3D coordinate queries, and use a slowly updated teacher as a scene reference to filter suspected transient supervision in perceptual space, improving both appearance representation and static reconstruction.

Method

Overall Architecture

The input is an unconstrained multi-view photo collection of one scene; the output is a 3DGS scene that supports novel-view rendering and different appearance conditions. A β-VAE obtains an appearance latent code from an image; a hypernetwork generates three orthogonal feature planes from that code; Gaussian primitives query these planes by 3D position, then pass dynamic appearance, intrinsic features, and viewing direction to a color decoder.

During training, the student scene renders images and receives reconstruction supervision. An EMA teacher supplies a pseudo-reference for the current view, and a frozen VGG-16 compares that reference with the real photo to produce a static mask that filters unreliable supervision. The teacher and mask belong to the training branch; novel-view rendering requires only an appearance code and camera, followed by the trained appearance field and rasterization pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training reference image"] --> B["Generative Appearance Latent Space"]
    B --> C["Latent-Conditioned 3D Appearance Field"]
    X["Gaussian positions, intrinsic features<br/>and viewing directions"] --> C
    C --> R["Student rasterized rendering"]
    C -.->|EMA update from student parameters| T["Teacher scene rendering"]
    T --> M["Self-Supervised Perceptual Masking"]
    A --> M
    R --> L["Mask-weighted reconstruction loss"]
    M --> L
    A --> L
    L -.->|Training supervision| C
    Z["Specified or sampled code at rendering time"] --> C

Solid edges represent images, conditions, or loss inputs; dashed edges represent teacher updates and student supervision. The mask comes from the teacher rendering and the real image, not a direct comparison between the current student rendering and ground truth. This reduces the immediate influence of rapidly changing student predictions on the mask.

Key Designs

1. Generative Appearance Latent Space: introduce a shared prior for each photo's appearance

Rather than storing an arbitrary vector for each photo, the β-VAE encoder predicts a Gaussian posterior over appearance latents. After posterior sampling, its decoder attempts to reconstruct the input photo. The reconstruction term retains information needed to explain the image, while KL regularization constrains the posterior toward a standard Gaussian prior. The paper uses β=2, emphasizing the information bottleneck more strongly than a standard VAE in the hope of reducing image-specific details and transient noise memorized by the code, leaving more capacity for global changes such as illumination and weather.

This prior serves both reconstruction and generation. An individually trained scene can obtain codes from reference images, interpolate between two codes, or sample from the standard Gaussian distribution and pass the result to the appearance field. This does not establish identifiable physical lighting factors, nor does it imply one universal generator shared across scenes: the paper explicitly demonstrates appearance sampling for independently trained scenes. Per-image embeddings can also be numerically interpolated; the clearer distinction here is the addition of a regularized continuous distribution with a sampling interface.

2. Latent-Conditioned 3D Appearance Field: make one appearance condition act differently at different positions

Simply concatenating a global code into a color network can compress scene changes into coarse global tone adjustments. WilLaGS instead uses a hypernetwork to map the code into three orthogonal XY, XZ, and YZ feature planes, each with 64×64 resolution and 32 channels. These planes change with the appearance code but retain a scene-aligned coordinate system. The same surface location therefore queries the same appearance field across viewpoints, rather than being recolored independently in each 2D image.

For each query, a Gaussian center is normalized to a standard 3D range using the scene bounds, projected onto the three planes, and bilinearly sampled. The sampled features are concatenated and processed by a small MLP to produce a dynamic appearance feature. A separate learnable intrinsic feature is stored with each Gaussian to carry relatively stable material information. The color fusion decoder combines the dynamic feature, intrinsic feature, and viewing direction to predict view-dependent color, after which ordinary Gaussian rasterization produces the image. This binds local color modulation to 3D positions without explicitly solving light transport; learned material features should not be treated as validated albedo ground truth.

3. Self-Supervised Perceptual Masking: use a slow teacher to reduce transient contamination

Teacher and student have the same architecture, including the appearance field and color decoder. The student is optimized through losses, while the teacher is updated by an exponential moving average of student parameters to aggregate more stable predictions over training. Here, time primarily means optimization iterations, not a requirement that input photos form a temporal video. The teacher rendering is a scene consensus at the current viewpoint, not a manually supplied clean reference without pedestrians. If the teacher has not reconstructed a structure correctly, its prediction can still be wrong; stability is only a reliability cue, not a guarantee of correct static content.

Masking uses the relu1_2, relu2_2, and relu3_3 features of a frozen, pretrained VGG-16. Multi-layer L1 differences between the real image and teacher pseudo-reference are averaged over channels and summed across layers into a perceptual difference map. A threshold of 0.85 retains low-difference regions as static supervision and temporarily excludes high-difference regions. Compared with pixel differences, perceptual features aim to reduce false positives caused by normal lighting changes. This robustness is a design goal and empirical observation, not invariance to every lighting change. The method needs neither transient segmentation labels nor an external segmenter, but it explicitly depends on pretrained VGG features and therefore is not free of external priors.

Loss & Training

The student and real images are each multiplied by the static mask before L1 and D-SSIM reconstruction terms are computed, with weights 0.8 and 0.2. A VAE loss weighted by 0.01 jointly optimizes the appearance representation and scene. VAE image reconstruction and scene rasterization are different paths; the former's 2D decoded image is not the final novel-view output.

The implementation uses PyTorch and Adam, training for 30k steps on one RTX 3090. The latent dimension is 64, and both dynamic and intrinsic features are 32-dimensional. The Tri-Plane generator and color fusion decoder are three-layer ReLU MLPs. Relative to standard 3DGS, the paper reports about 933 MB of trainable VRAM overhead: approximately 66.64 MB for the β-VAE and 866.75 MB for the 3D field, plus about 528 MB of fixed overhead for frozen VGG-16. These figures are not the full system's total peak training memory.

The cached ELBO, EMA, plane-query, and total-loss equations contain missing terms or extraction damage. This note explains their mechanisms from the surrounding prose rather than reconstructing purported exact equations. The EMA decay, early-training masking policy, and hyperparameter sensitivity cannot be fully verified from the supplied main text; supplementary material was not provided.

Key Experimental Results

Main Results

Photo Tourism (PT) covers three landmarks, while NeRF-OSR covers four outdoor scenes: europa, lwp, st, and stjohann. Evaluation follows a NeRF-W-style protocol: optimize the appearance code on the left half of each test image, then compute metrics on its right half. This is not feed-forward rendering without test-image adaptation. Higher PSNR and SSIM are better; lower LPIPS is better.

The table excerpts original Tables 1 and 2, selecting three PT scenes and two NeRF-OSR scenes. Baselines are primarily selected by PSNR; this does not imply they are second-best on every metric in that scene.

Dataset / Scene	Method	PSNR ↑	SSIM ↑	LPIPS ↓
PT / Sacre Coeur	AsymGS	23.56	0.877	0.169
PT / Sacre Coeur	WilLaGS	25.84	0.891	0.164
PT / Trevi Fountain	AsymGS	23.91	0.785	0.223
PT / Trevi Fountain	WilLaGS	24.54	0.789	0.207
PT / Brandenburg Gate	AsymGS	28.49	0.928	0.139
PT / Brandenburg Gate	WilLaGS	29.94	0.939	0.139
NeRF-OSR / europa	GS-W	23.31	0.833	0.335
NeRF-OSR / europa	WilLaGS	24.38	0.843	0.216
NeRF-OSR / stjohann	GS-W	25.72	0.894	0.276
NeRF-OSR / stjohann	WilLaGS	25.55	0.906	0.161

On Sacre Coeur, PSNR exceeds AsymGS by 2.28 dB, but this does not extend to every scene and metric. The original Table 2 caption claims improvements on all three metrics across all scenes, yet stjohann PSNR is 25.55, below GS-W's 25.72. In Table 1, Trevi Fountain SSIM is also below SWAG's 0.815, and Sacre Coeur LPIPS of 0.164 is worse than CR-NeRF's 0.152. These exceptions are retained according to the printed numbers.

The efficiency column reports 0.9 GPU hours / 58 FPS for WilLaGS, 7.8 hours / 73 FPS for WildGaussians, and 5.6 hours / 47 FPS for AsymGS. WilLaGS therefore trains faster but does not render faster than WildGaussians. The prose describes standardized configurations, but SWAG is starred in Table 1, indicating results taken directly from its original paper; the entire table should not be treated as identically rerun experiments.

Ablation Study

Original Table 3 reports scene averages for each dataset. Removing the VAE replaces it with per-image discrete embeddings; removing the 3D appearance field directly concatenates the global code with intrinsic features; removing masking trains on all pixels.

Config	PT: PSNR / SSIM / LPIPS	NeRF-OSR: PSNR / SSIM / LPIPS
Without VAE latent space	22.15 / 0.835 / 0.197	21.54 / 0.802 / 0.233
Without 3D appearance field	25.05 / 0.853 / 0.191	23.02 / 0.817 / 0.235
Without teacher-student mask	26.19 / 0.861 / 0.182	23.94 / 0.825 / 0.226
Full model	26.77 / 0.873 / 0.170	24.18 / 0.832 / 0.218

Key Findings

• Replacing the VAE causes the largest drop: PT PSNR falls from 26.77 to 22.15, a difference of 4.62 dB; NeRF-OSR falls from 24.18 to 21.54, a difference of 2.64 dB. This supports the representation in this configuration, not a universal claim that all discrete embeddings must overfit.
• Removing the 3D appearance field lowers PT PSNR by 1.72 dB; original Figure 6 also describes flatter local shadows. Its contribution is position-dependent appearance conditioning, not validated physical inverse rendering.
• Removing masking lowers PT PSNR by 0.58 dB. A small average drop can coexist with local ghosting, motivating inspection of occluded regions rather than relying only on full-image averages.
• Figures 7 and 8 demonstrate fixed-code multi-view rendering, latent interpolation, and prior sampling, but provide no separate quantitative evaluation of generative diversity, cross-view consistency error, or physical lighting accuracy.

Highlights & Insights

• One latent space supports adaptation and sampling. It conditions observed photos and supplies new conditions for an already reconstructed scene without a separately trained appearance generator; the VAE and appearance field themselves still incur memory and training costs.
• Local appearance is tied to 3D position. Shared spatial queries across views support consistent novel-view appearance more naturally than independent 2D recoloring. Local colors can change while geometry remains represented by a shared Gaussian scene.
• Masks depend on teacher predictions rather than category lists. Transient categories need not be specified in advance, accommodating dataset-specific occlusions; the trade-off is dependence on the teacher reaching a reliable scene consensus.

Limitations & Future Work

• The authors explicitly acknowledge dense occlusions and the lack of physical interpretability, and propose semantic control of the latent space and larger-scale scenes as future work.
• Persistent occluders in similar positions may be absorbed into the teacher's static content; early reconstruction errors may also mask out genuinely static regions. These are potential failure modes identified by this note, not systematically tested in the paper.
• No external segmenter does not mean no pretrained features. Frozen VGG-16 introduces perceptual priors and additional memory, and the method does not establish reliable separation of illumination and transients under every severe appearance change.
• Appearance interpolation and sampling are supported mainly by visual examples. These do not guarantee coverage of real weather distributions, strict physical consistency, or direct transfer to untrained scenes.

Related Work & Insights

• vs NeRF-W: per-image embeddings are replaced by shared posterior modeling with a standard Gaussian prior. Distributional regularization and the sampling interface are the central differences, not an inability of earlier embeddings to interpolate.
• vs GS-W / Wild-GS: these methods already investigate local or hierarchical appearance. WilLaGS instead generates spatial feature planes dynamically from a generative latent code and queries them by Gaussian position; prior work should not collectively be reduced to global tone adjustment.
• vs segmenters and uncertainty networks: teacher consensus avoids training a separate transient predictor but still needs a perceptual network. Adapting the masking threshold to teacher stability and distinguishing temporary occlusions from unconverged geometry are promising directions, not capabilities implemented by the paper.

Rating

• Novelty: 4/5. Generative appearance priors, spatial 3D modulation, and teacher perceptual masking address a clear bottleneck in combination, although their underlying components have established precedents.
• Experimental Thoroughness: 4/5. Two datasets, component ablations, and efficiency comparisons offer substantial evidence; generative applications, failure modes, and mask accuracy still lack quantitative support.
• Writing Quality: 3/5. Component roles are clear, but blanket superiority in a table caption conflicts with its numbers, and claims about external priors require qualification given VGG dependence; damaged cached equations are not attributed to original writing errors.
• Value: 4/5. The method offers a useful quality-efficiency trade-off for unconstrained photo reconstruction and an actionable interface for variable-appearance rendering within one scene.