Complex-Valued 2D Gaussian Representation for Computer-Generated Holography

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/complight/Complex-Valued_2D_Gaussian_Representation
Area: 3D Vision
Keywords: Computer-Generated Holography, Complex-Valued 2D Gaussians, Diffraction Light Propagation, Hologram Optimization, Space-Frequency Uncertainty

TL;DR

This paper introduces a compact hologram representation based on complex-valued 2D Gaussian primitives and a customized CUDA rasterization-propagation pipeline, reducing the parameter search space by 5:1 via Gabor's minimum space-frequency uncertainty while maintaining high-fidelity diffraction reconstructions across both smooth and random phase-only holograms.

Background & Motivation

Computer-Generated Holography (CGH) modulates the wavefront of light via spatial light modulators (SLMs) to simultaneously reconstruct the intensity, interference, and diffraction phenomena of 3D scenes, serving as a foundational technology for next-generation holographic displays and near-eye virtual and augmented reality. Unlike natural images that display smooth and continuous intensity transitions, a hologram encodes a complex wave-optical field whose microscopic structure exhibits dense high-frequency oscillations and pseudo-random phase patterns. Conventional hologram generation approaches either rely on dense, per-pixel complex amplitude or phase optimization—which entails an immense parameter search space and slow numerical convergence—or train deep learning models such as implicit neural representations (INRs) or autoencoders. However, continuous coordinate networks and natural-image priors inherently favor low-frequency content, struggling to resolve the sharp interference fringes characteristic of diffraction optics.

While 2D and 3D Gaussian Splatting have recently demonstrated remarkable fitting capacity and rendering efficiency in neural rendering and image compression, existing natural image Gaussian primitives operate exclusively on real-valued color distributions, ignoring the physical wave optics governing interference and free-space diffraction propagation. More critically, a naive adaptation that pairs two independent real-valued Gaussians to model the real and imaginary components incurs a bloated parameter budget (18 parameters per pair) and introduces spatial misalignment between amplitude and phase during gradient descent, leading to local field distortions and severe defocus blur upon propagation.

To address these limitations, this paper migrates the formulation of complex-valued Gaussians from multi-view holographic radiance fields to the single-hologram optimization regime, parameterizing the hologram plane with structured complex-valued 2D Gaussian primitives. Grounded in Gabor's uncertainty principle, Gaussian functions uniquely attain the theoretical minimum bound of joint space and spatial frequency uncertainty, rendering them the mathematically optimal basis to compress parameter dimensions while preserving high-frequency diffraction details. Core idea: parameterize the hologram as a compact set of complex-valued 2D Gaussian primitives optimized end-to-end via a differentiable CUDA rasterizer and a GPU-optimized band-limited angular spectrum propagation kernel, reducing the parameter search space by 5:1 while serving as an effective structural prior for both smooth and random phase-only holograms.

Method

Overall Architecture

Given a target RGB image \(I_{\text{target}} \in \mathbb{R}^{C \times H \times W}\) and an optional depth map \(D \in \mathbb{R}^{H \times W}\), the goal is to synthesize a complex hologram \(H \in \mathbb{C}^{C \times H \times W}\) whose optical propagation reproduces \(I_{\text{target}}\) with accurate focal and defocus cues. The overall pipeline proceeds through four distinct stages: a set of learnable complex-valued 2D Gaussian primitives is rasterized onto the hologram plane to construct the complex wavefield; the complex field is propagated along the optical axis to multiple focal depths using a GPU-optimized Band-Limited Angular Spectrum Method (BLASM) kernel; the simulated multi-plane intensities are supervised against focal-stack targets using a depth-masked defocus reconstruction loss combined with structural similarity; finally, the converged complex Gaussian representation serves as an intermediate structural prior to generate hardware-compatible smooth and random phase-only holograms.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input RGB target and depth map<br/>I_target, D"] --> B["Unified complex-valued 2D Gaussian primitives<br/>12-parameter compact representation"]
    B --> C["CUDA rasterizer and optimized BLASM kernel<br/>Tiled radix sorting + frequency-domain BLASM propagation"]
    C --> D["Differentiable light propagation and focal-stack loss<br/>Multi-plane masked defocus loss L_recon + L_SSIM"]
    D --> E["Decoupled POH conversion procedure<br/>DPAC smooth POH / Structurally guided random POH"]
    E --> F["Hardware-compatible holographic outputs<br/>Smooth POH / Random POH"]

Key Designs

1. Unified complex-valued 2D Gaussian primitives: slashing parameter search space via minimum uncertainty

A naive extension of Gaussians to the complex domain assigns independent real Gaussians to the real and imaginary parts, demanding 18 parameters per pair and risking decoupled spatial drift that destroys phase coherence. Instead, this paper devises a unified complex-valued 2D Gaussian primitive \(\mathcal{G}_n = \{\tilde{\mathbf{x}}_n, \tilde{\mathbf{s}}_n, \theta_n, \mathbf{c}_n, \tilde{\alpha}_n, \boldsymbol{\varphi}_n\}\) requiring only 12 learnable parameters. Here, \(\mathbf{x}_n \in \mathbb{R}^2\) denotes the central coordinate, \(\mathbf{s}_n \in \mathbb{R}^2\) and in-plane rotation \(\theta_n \in \mathbb{R}\) define the spatial covariance matrix \(\boldsymbol{\Sigma}_n = R(\theta_n) \text{diag}(\mathbf{s}_n)^2 R(\theta_n)^\top\), while \(\mathbf{c}_n \in \mathbb{R}^C\), \(\alpha_n \in \mathbb{R}\), and \(\boldsymbol{\varphi}_n \in \mathbb{R}^C\) represent color amplitude, opacity, and per-channel phase, respectively. At pixel position \(\mathbf{p}\), the primitive's complex contribution is analytically expressed as: $\(H_n(\mathbf{p}) = \alpha_n \mathbf{c}_n \exp\left(-\frac{1}{2}(\mathbf{p}-\mathbf{x}_n)^\top \boldsymbol{\Sigma}_n^{-1} (\mathbf{p}-\mathbf{x}_n)\right) \exp(j \boldsymbol{\varphi}_n)\)$ According to Gabor's theory of communication, the Gaussian envelope is the unique analytical function that reaches the fundamental lower bound of space and spatial-frequency uncertainty: \(\Delta x \Delta f_x = \frac{1}{2}\). Because CGH requires simultaneously controlling where light originates on the SLM (spatial domain) and the angles at which it diffracts (frequency domain), complex Gaussians minimize joint uncertainty in both dimensions, enabling a 5:1 parameter reduction relative to dense per-pixel representations without sacrificing diffraction details.

2. Differentiable light propagation and focal-stack supervision: depth-aware multi-plane constraints

Supervising the hologram field directly on the SLM plane without optical propagation fails to capture volumetric diffraction effects. The method incorporates free-space scalar diffraction propagation over \(L\) uniformly spaced parallel planes \(\Pi_l\) along the optical axis, centered at propagation distance \(d_0\) with spacing \(\Delta z\). Propagating the synthesized complex field \(H\) to distance \(d_l\) via BLASM: $\(U(d_l) = \mathcal{F}^{-1} \left\{ H_{\text{BLASM}}^{d_l}(f_x, f_y) \cdot \mathcal{F}\{H\} \right\}\)$ yields reconstructed intensity \(I_l = |U(d_l)|^2\). To accurately reproduce defocus blur, quantized depth maps provide binary focal masks \(M_l\), formulating a multi-plane masked reconstruction objective: $\(\mathcal{L}_{\text{recon}} = \frac{1}{L} \sum_{l=1}^L \left( \|I_l - \hat{I}_l\|_2^2 + \|I_l \odot M_l - \hat{I}_l \odot M_l\|_2^2 + \|I_l \odot \hat{I}_l - \hat{I}_l \odot \hat{I}_l\|_2^2 \right)\)$ This objective drives Gaussian positions, covariances, and complex phases to align synergistically, maintaining crisp in-focus edges while accurately modeling realistic out-of-focus defocus blur.

3. Decoupled POH conversion procedure: structurally guided smooth and random phase modulation

Because commercial phase-only SLMs cannot modulate amplitude and phase simultaneously, the proposed complex Gaussian field acts as a versatile intermediate representation decoupled from display-specific physical constraints. For Smooth Phase-Only Holograms (Smooth POH), Double Phase-Amplitude Coding (DPAC) multiplexes amplitude and phase onto an alternating checkerboard grid, mapping amplitude into phase differences on even pixels and phase values on odd pixels. For Random Phase-Only Holograms (Random POH), optimizing per-pixel phase directly leads to chaotic local minima and severe speckle noise; the method propagates both the complex Gaussian field \(H\) and the learnable random phase field \(H_{\text{rand}} = \exp(j \boldsymbol{\varphi}_{\text{rand}})\) simultaneously to focal planes, incorporating a complex-field discrepancy penalty \(\lambda_{\text{field}} \|U(d_l) - U_{\text{rand}}(d_l)\|_{1, \mathbb{C}}\) alongside intensity matching. The compact spatial regularization embedded within the Gaussian field effectively guides the optimization of random phase values, suppressing granular speckle artifacts.

4. CUDA rasterizer and optimized BLASM kernel: constant VRAM footprint and coalesced cache access

Standard autodiff implementations in general deep learning frameworks encounter excessive memory footprints and kernel launch overheads when performing dense 2D Gaussian rendering and FFT-based diffraction propagation. This work implements dedicated CUDA operations. The tile-based (\(16 \times 16\)) rasterizer sorts Gaussian primitives using radix sort and executes single-pass trigonometric evaluations to accumulate real and imaginary parts concurrently; only accumulated opacity is stored during forward rendering, while intermediate parameters are recovered via division in the backward pass, bounding peak VRAM to a constant overhead. The companion BLASM kernel computes transfer function evaluations with coalesced read-only cache accesses and handles backward propagation with band-limited conjugate kernels, cutting VRAM consumption by ~30% and accelerating per-step training by 40%–50%.

Loss & Training

The overall training objective combines multi-plane focal-stack reconstruction with structural similarity: $\(\mathcal{L} = \mathcal{L}_{\text{recon}} + \lambda_1 \mathcal{L}_{\text{SSIM}}\)$ with weighting factor \(\lambda_1 = 0.005\). Optimization uses the Adan optimizer for 2000 steps, with convergence typically achieved within 1000+ steps. Primitives are initialized uniformly across the image canvas and projected to unbounded domain via \(\text{atanh}\), scales are initialized to \([1.5, 5.0]\) pixels, phases set to zero, and opacity pre-activations set to \(-0.5\) (\(\alpha_n \approx 0.38\)). Parameter-specific learning rates are set to \(10^{-2}\) for positions (cosine-annealed to \(10^{-3}\)), \(5 \times 10^{-3}\) for scales, \(2.5 \times 10^{-3}\) for amplitudes and phases, \(2.5 \times 10^{-2}\) for opacities, and \(10^{-3}\) for rotations. Optical parameters follow standard setups: RGB wavelengths of 639 nm, 532 nm, and 473 nm, pixel pitch of \(3.74\ \mu\text{m}\), and base propagation distance of 3 mm.

Key Experimental Results

Main Results

Evaluations are conducted on the first 50 images of DIV2K at \(3 \times 1024 \times 640\) resolution across \(L = 2\) focal planes, reporting average PSNR, SSIM, LPIPS, peak VRAM, and training duration.

Table 1: Quantitative comparison against learned and Gaussian-based image representations

Method	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	VRAM	Params	Time (min)
TAESD (Pretrained autoencoder)	11.6	0.09	0.79	2.7 G	2.5 M	-
MLP (Coordinate INR)	7.5	0.04	0.85	9.9 G	1.0 M	6.9
SIREN (Periodic activation INR)	7.6	0.05	0.84	13.1 G	1.0 M	7.8
Image-GS (Natural image 2DGS)	17.2	0.29	0.70	1.3 G	2.4 M	1.6
GI (GaussianImage)	22.6	0.49	0.59	1.1 G	2.4 M	0.8
Instant-GI	23.5	0.56	0.56	3.4 G	2.8 M	0.9
Ours (5:1 reduction)	30.7	0.86	0.33	2.2 G	0.8 M	1.4

Table 2: Quantitative comparison across CGH methods for Smooth and Random POH

Category / Method	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	VRAM	Params	Time (min)	Render (ms)
Smooth POH
NH3D (Neural 3D Holography)	28.3	0.92	0.31	7.1 G	3.9 M	90	31.0
TensorV2	27.1	0.94	0.29	8.7 G	0.1 M	80	48.0
GWS (Gaussian Wave Splatting)	28.2	0.76	0.41	2.5 G	1.1 M	5.1	6840.0
U-Net	27.2	0.91	0.35	6.3 G	2.2 M	100	18.0
Multi-color	27.9	0.74	0.40	3.2 G	4.0 M	5.3	-
Ours	29.0	0.81	0.38	2.4 G	0.8 M	1.4	2.13
Random POH
Naive Opt (Direct Phase Opt)	19.8	0.33	0.60	2.9 G	2.0 M	2.9	-
Wirtinger Holography	25.3	0.47	0.48	3.5 G	2.0 M	2.8	-
Multi-color	20.3	0.35	0.64	3.1 G	4.0 M	3.0	-
Ours	29.4	0.81	0.34	3.4 G	2.0 M	3.8	-

Ablation Study

Table 3: Impact of parameter reduction ratio on reconstruction quality and rendering latency

Parameter Reduction Ratio	Params	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	Render (ms)
Dense Per-pixel	4.0 M	32.3	0.893	0.29	4.02
2:1	2.0 M	31.9	0.891	0.30	2.58
3:1	1.3 M	31.5	0.885	0.31	2.33
5:1 (Default)	0.8 M	30.7	0.863	0.33	2.13
7:1	0.6 M	30.3	0.856	0.34	1.90
10:1 (Upper Bound)	0.4 M	29.4	0.835	0.37	1.72

Table 4: Ablation study on structural guidance and primitive parameterization

Type	Configuration	PSNR ↑ (dB)	SSIM ↑	LPIPS ↓	Params	Render (ms)
Random POH	w/o guidance	19.1	0.37	0.52	2.0 M	-
Random POH	with guidance (Ours)	30.6	0.88	0.22	2.0 M	-
Complex Formulation	Naive Paired Gaussians	25.5	0.74	0.47	1.2 M	20.1
Complex Formulation	Unified Complex 2D Gaussians (Ours)	31.8	0.89	0.31	0.8 M	2.13

Key Findings

• Decisive role of structural guidance for random POH: Optimizing Random POH without guidance yields only 19.1 dB PSNR; introducing the complex-valued Gaussian prior boosts reconstruction quality to 30.6 dB (+11.5 dB improvement), confirming that the compact representation provides vital regularization to constrain the ill-posed phase retrieval problem.
• Superiority of unified complex primitives over paired real Gaussians: Modeling amplitude and phase via two independent sets of real Gaussians increases parameter count by 33% and causes spatial misalignment during optimization, degrading PSNR by 6.3 dB while increasing rendering latency tenfold (20.1 ms vs 2.13 ms) due to duplicated rasterization passes.
• Graceful quality degradation under aggressive compression: Aligned with Gabor's uncertainty theory, scaling the parameter reduction ratio from 2:1 up to 10:1 only causes PSNR to drop moderately from 31.9 dB to 29.4 dB; at the 5:1 operating point, the model achieves 30.7 dB with only 0.8M parameters, offering an optimal balance of quality and compactness.
• Substantial memory and runtime savings from custom CUDA kernels: Custom CUDA rasterization and BLASM operations consistently deliver 29%–36% VRAM reduction and 40%–50% optimization speedup across diverse resolutions, allowing single-GPU scaling to \(3200 \times 1800\) (5.8M pixels) with \(L=3\) depth planes without encountering Out-Of-Memory errors.

Highlights & Insights

• Bridging classical communication theory with wave optics: The work ties Gabor's 1946 proof of minimum joint space-frequency uncertainty to modern computer-generated holography, providing mathematical justification for why Gaussian primitives retain holographic fidelity far better than pixel grids under reduced parameter budgets.
• Decoupled hardware-agnostic intermediate representation: By decoupling continuous wavefield representation from display-specific quantization, a single complex Gaussian representation can be converted into both smooth (DPAC) and random phase-only holograms, circumventing brittle end-to-end retraining for differing hardware setups.
• Constant-overhead backward pass design: Storing only accumulated opacity in the forward rasterizer and recovering intermediate terms analytically in the backward pass eliminates memory bottlenecks in multi-plane wave propagation pipelines.

Limitations & Future Work

• Hardware gap under physical capture: Optical experiments on a bench prototype (LASOS laser, Jasper SLM, Point Grey sensor) show smaller quality gains compared to numerical simulations. This is caused by laser speckle noise, 8-bit SLM phase quantization, and optical relay aberrations that degrade captured wavefronts.
• Two-stage pipeline for random POH: The current workflow requires first optimizing the complex Gaussian field before performing guided phase retrieval; formulating direct, end-to-end discrete phase rasterization remains an open problem.
• Single-view focus without dynamic eyebox expansion: The representation targets high-fidelity focal-stack reconstruction from a central pupil viewpoint and does not natively integrate wide-angle pupil steering or dynamic eyebox expansion, pointing toward future integration with time-multiplexing and camera-in-the-loop adaptation.

Related Work & Insights

• vs GWS (Choi et al., SIGGRAPH 2025): GWS synthesizes holographic scenes from multi-view inputs and suffers from high rendering latency (6840 ms); this work focuses on compact single-view hologram optimization and rendering, accelerating inference to 2.13 ms (~3200× faster) with lower memory consumption.
• vs Complex-Valued Holographic Radiance Fields (Zhan et al., TOG 2026): While that method builds 3D complex Gaussians for volumetric view synthesis, this work formulates 2D complex Gaussians directly on the hologram plane, formalizes the parameter search space reduction via Gabor theory, and introduces conversion techniques for smooth and random POH.
• vs Wirtinger Holography (Chakravarthula et al., SIGGRAPH 2019): Wirtinger Holography optimizes per-pixel phase directly in the hologram domain, often getting trapped in high-frequency speckle noise; this method provides structured spatial priors that boost Random POH PSNR by +4.1 dB.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Introduces complex-valued 2D Gaussian primitives grounded in Gabor's minimum uncertainty theorem for compact hologram optimization.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluations covering baseline comparisons, aggressive compression scaling, custom CUDA profiling, and real-world optical bench captures.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical derivation, cohesive motivation, and meticulous documentation of physical optical setups.
• Value: ⭐⭐⭐⭐⭐ Establishes an efficient, scalable representation paradigm for next-generation holographic displays and spatial computing devices.