NeLU3D: Neural Inverse Structured Light without Modeling the Projector

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/geometryprocessing/neural-lookup
Area: 3D Vision
Keywords: Structured Light, Neural Inverse Rendering, Neural Implicit Surfaces, Projector Calibration-Free, Differentiable Volume Rendering

TL;DR

NeLU3D presents a neural inverse structured light reconstruction framework that operates without explicit geometric or radiometric modeling of the projector, baking a one-time 3D-to-color mapping into a multiresolution hash-encoded MLP and optimizing a signed distance function with physically bounded ambient decoupling to deliver sub-millimeter surface reconstructions from as few as three suboptimal patterns and a white flash.

Background & Motivation

Structured Light (SL) scanning remains a cornerstone 3D capture technology across cultural heritage preservation, industrial metrology, and robotics due to its non-contact nature, high repeatability, and fine accuracy across varied object scales and materials. Traditional SL systems rely on a calibrated camera-projector stereo pair, where the projector sequentially casts coded illumination patterns onto the target and the camera records the modulated surface appearances. Triangulating matching light rays via epipolar geometry yields dense point clouds with sub-millimeter precision. However, this classical triangulation is notoriously brittle: it mandates exhaustive calibration of both extrinsic alignment and intrinsic optical traits, including non-linear lens distortion, radiometric response curves, inter-pixel sensor gaps, focal blur, and optical vignetting. Any imperfection in projector modeling propagates directly into structural artifacts and systematic geometric distortion.

To break free from fragile optical calibration, two prominent research directions have recently emerged. The first direction is neural inverse structured light, exemplified by TurboSL, which fits implicit surface representations using differentiable rendering; yet TurboSL still rigidly relies on pinhole projector models, discrete pose estimation, and learned 2D blur kernels, while requiring specialized patterns tailored to specific baseline arrangements. The second direction is data-driven look-up scanning, as introduced in LookUp3D, which bypasses optical parameter estimation entirely by building an empirical look-up table that directly maps 3D coordinates to projected color intensities. Nevertheless, LookUp3D performs brute-force pixelwise nearest-neighbor searches, consuming tens to hundreds of gigabytes of memory and suffering from extreme vulnerability to image noise, leaving reconstructed point clouds heavily contaminated by spatial outliers.

This paper tackles this bottleneck by asking: can we marry the uncalibrated optical flexibility of data-driven lookup tables with the continuous geometric inductive bias of neural implicit inverse rendering? NeLU3D achieves this by distilling discrete empirical calibrations into a compact, differentiable neural color mapping field, which is subsequently frozen and embedded into an implicit neural volume rendering loop. Core idea: compress the 3D-position-to-projected-color mapping into a frozen multiresolution hash-encoded neural field through a one-time sweep calibration, and optimize a scene signed distance function alongside a bounded ambient light field via differentiable volume rendering, completely eliminating explicit projector modeling while delivering sub-millimeter 3D reconstruction under suboptimal patterns and ultra-low-cost or analog projectors.

Method

Overall Architecture

The NeLU3D pipeline operates in two distinct phases: an offline one-time neural calibration stage and an online implicit surface inverse rendering stage. During offline calibration, a planar white target equipped with fiducial markers is stepped through the working volume on a linear translation stage under camera observation. Capturing responses across \(k\) projected patterns, ambient illumination, and a white flash provides empirical coordinates and color measurements used to train a compact color MLP \(c(x)\). During online scanning of an unknown target, \(c(x)\) is frozen. The scene geometry is modeled by an MLP \(f(x)\) representing a continuous Signed Distance Function (SDF), while scene appearance residuals are handled by an ambient MLP \(a(i)\). Differentiable volume rendering synthesizes predicted camera views, and the network parameters are updated by minimizing image photometric discrepancies against the camera captures.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["One-Time Calibration Sweep<br/>Target 3D poses + k pattern responses"] --> B["Neural LookUp Learning<br/>MHE+MLP mapping 3D points to projected colors"]
    B -->|Freeze weights c| C["Decoupled Differentiable Volume Rendering<br/>Ray marching integrating SDF and frozen color field"]
    D["Unknown Scene SL Capture<br/>k structured patterns + 1 white flash"] --> C
    C --> E["Bounded Ambient & Hybrid Loss Optimization<br/>Predict physical ambient and backpropagate to SDF"]
    E --> F["Extract Precise 3D Surface & Normals"]

Key Designs

1. Neural LookUp Learning: Differentiable Compression and Projector Decoupling

Classical triangulation and neural SL formulations demand explicit modeling of projector intrinsics, extrinsics, and point spread functions. Inspired by LookUp3D, any 3D point \(x \in \mathbb{R}^3\) illuminated by a fixed projector exhibits a unique normalized intensity signature across \(k\) projected patterns. However, discretizing this space into dense volumetric grids requires prohibitive memory (\(15\text{ GB}\) at \(512^3\) resolution and GPU out-of-memory at \(1024^3\)). NeLU3D bypasses this limitation by training a lightweight Multiresolution Hash Encoding (MHE) coupled with a shallow MLP to form a continuous representation \(c: \mathbb{R}^3 \to [0, 1]^k\). This network trains in approximately three minutes on an RTX A6000 and compresses calibration data from $50\text}121\text{ GB}$ down to just \(49\text{ MB}\). Crucially, the learned color network inherently absorbs optical aberrations, vignetting, lens defocus, and inter-pixel gaps without parametric assumptions, acting as a differentiable, frozen forward-projection prior during reconstruction.

2. Decoupled Differentiable Volume Rendering: Implicit Smoothness and Outlier Suppression

Discrete lookup decoding (LookUp3D) performs independent pixelwise minimization, which is acutely sensitive to sensor noise, high-frequency camera gain fluctuations, and subtle shadowing, resulting in severe surface outlier contamination. NeLU3D represents scene geometry as a continuous SDF \(f: \mathbb{R}^3 \to \mathbb{R}\) and converts distances into volume densities \(\sigma(t)\) following NeuS. For each camera pixel ray \(x(t) = o + t v_i\), the direct pattern illumination is integrated via volumetric compositing:

\[\tilde{I}_i = \int_{t_{\min}}^{t_{\max}} T(t) \sigma(t) c(x(t)) dt\]

where \(T(t) = \exp(-\int_{t_{\min}}^t \sigma(s)ds)\) denotes accumulated transmittance and \(c(x(t))\) is queried directly from the frozen color network. Because the underlying SDF network enforces a global inductive bias toward continuous, manifold geometry, inverse rendering naturally filters high-frequency noise and discards isolated outliers without ad-hoc post-processing filters.

3. Bounded Ambient & Hybrid Loss Optimization: Physical Regularization and Dynamic Range Balancing

Physical scanning environments exhibit significant indirect illumination, ambient lighting, and subsurface scattering that corrupt geometric convergence if ignored. NeLU3D employs an ambient network \(a: \mathbb{R}^2 \to [0, 1]\) to estimate residual irradiance \(I_{\text{ambient}}\) per pixel. To avert catastrophic gradient explosions during backpropagation, the authors enforce an upper-bound physical constraint: because the white flash capture \(I_{\text{white}}\) illuminates the scene at full projector intensity, ambient residuals cannot exceed this ceiling (\(I_{\text{ambient}} \le I_{\text{white}}\)). This constraint is implemented by passing ambient predictions through a sigmoid activation and scaling pixelwise by \(I_{\text{white}}\). The total photometric objective blends raw intensity differences and normalized lookup objectives via parameter \(\alpha = 0.1\):

\[\mathcal{L}_{\text{image}} = (1 - \alpha) \mathcal{L}_{\text{pattern}} + \alpha \mathcal{L}_{\text{LookUp}}\]

where \(\mathcal{L}_{\text{pattern}}\) compares \(\tilde{I}_{\text{pattern}} = \tilde{I} + I_{\text{ambient}}\) with the captured pattern images to stabilize well-exposed regions, while \(\mathcal{L}_{\text{LookUp}}\) operates on flash-normalized ratios to preserve sensitivity over dark, low-albedo surfaces.

Loss & Training

Optimization runs for 10,000 iterations per scene on an NVIDIA RTX A6000 GPU. The combined loss function incorporates geometric and volumetric regularizers:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{image}} + \lambda_{\text{mask}} \mathcal{L}_{\text{mask}} + \lambda_{\text{eik}} \mathcal{L}_{\text{eik}} + \lambda_{\text{sp}} \mathcal{L}_{\text{sp}}\]

where: - \(\mathcal{L}_{\text{mask}}\) applies Binary Cross-Entropy (BCE) to enforce accumulated ray opacity against a silhouette foreground mask \(\mathcal{M}\), with weight \(\lambda_{\text{mask}} = 0.1\); - \(\mathcal{L}_{\text{eik}} = \sum_{i, d} (\|\nabla f(x(i, d))\|_2 - 1)^2\) imposes the Eikonal constraint to preserve valid SDF gradient magnitudes, with weight \(\lambda_{\text{eik}} = 0.1\); - \(\mathcal{L}_{\text{sp}} = \sum_{i, d} \exp(-|f(x(i, d))|)\) acts as a sparsity penalty to remove floaters in free space, with weight \(\lambda_{\text{sp}} = 0.01\); - \(\mathcal{L}_{\text{LookUp}}\) and \(\mathcal{L}_{\text{pattern}}\) employ mixed \(L_1\) and \(L_2\) penalties with weighting factors 1.0 and 10.0, respectively.

Key Experimental Results

Main Results

NeLU3D was comprehensively evaluated across four distinct hardware rigs and over 25 real physical objects, including an industrial DLP projector (TI DLP4710EVM-LC), an ultra-budget 50 USD LCD projector (VOPLLS Mini Beaner N3), and a custom analog projector fitted with a 35mm photographic film pattern and kHz pulsed LEDs. Comparisons are made against dense SL ground truth reconstructed with 44 Gray code patterns using ZNCC.

Table 1: Quantitative depth error (MAE, in mm) and disparity error (in px) against dense SL across diverse patterns

Object	Pattern Configuration	ZNCC (mm / px)	LookUp3D (mm / px)	TurboSL (mm / px)	NeLU3D (Ours) (mm / px)
wipes	3 Hilbert patterns	196.70 / 832.95	8.81 / 153.23	29.91 / 181.68	3.87 / 8.13
first aid	3 Random patterns	298.44 / 514.57	13.27 / 38.93	122.00 / 415.71	0.53 / 1.39
monkey	3 Random patterns	265.24 / 525.16	41.11 / 298.44	40.04 / 409.94	0.96 / 2.90
teapot	3 A La Carte patterns	92.44 / 333.71	5.70 / 16.82	2.90 / 6.41	9.17 / 25.43
pawn	4 Micro Phase Shifting (MPS)	44.98 / 232.86	3.17 / 9.70	2.76 / 7.54	1.02 / 2.78
keyboard	11 Gray codes	0.39 / 1.14	33.09 / 137.16	0.97 / 2.90	0.33 / 0.84

Table 2: Computational overhead and VRAM usage on an RTX A6000 GPU across 3, 6, and 11 pattern settings

Method	VRAM (GB)	Reconstruction Time (pawn, s)	Reconstruction Time (monkey, s)	Requires Explicit Projector Model
ZNCC	0.1 ~ 0.3	0.002 ~ 0.004	0.100 ~ 0.101	Yes
LookUp3D	10.2 ~ 29.2	0.0016 ~ 0.0022	0.0018 ~ 0.0025	No
TurboSL	4.3 ~ 5.2	1020 ~ 2364	1160 ~ 2468	Yes (Intrinsics, Extrinsics, Blur Kernel)
NeLU3D (Ours)	3.7 ~ 4.4	420 ~ 431	534 ~ 562	No (Calibration-Free)

Ablation Study

The ablation study was performed on a CNC-milled pawn with glossy paint (approx. 10cm × 16cm × 10cm, CAD ground truth available) captured using the DLP projector with \(k=3\) suboptimal patterns (1 sinusoidal, 1 triangular, 1 random) plus the white flash.

Table 3: Ablation of loss formulations and ambient illumination constraints

Configuration	MAE against CAD (mm)	Surface Fidelity & Observation
LookUp loss only (w/o \(I_{\text{ambient}}\), only \(\mathcal{L}_{\text{LookUp}}\))	1.63	Surface heavily degraded by noise and dark-region intensity instability
Pattern loss only (w/o \(I_{\text{ambient}}\), only \(\mathcal{L}_{\text{pattern}}\))	0.99	Smoother surface, but residual global illumination warps geometry
Unconstrained ambient (\(\mathcal{L}_{\text{image}}\), unconstrained \(I_{\text{ambient}} \le I_{\text{white}}\))	FAILED	Gradients explode due to negative \((I_{\text{white}} - I_{\text{ambient}})\) terms
Full Model (\(\alpha=0.1\) and constrained \(I_{\text{ambient}} \le I_{\text{white}}\))	0.65	Clean geometry aligning closely with CAD model, sub-millimeter precision

Key Findings

• Unrivaled Resilience to Suboptimal Patterns: On arbitrary or pseudo-random patterns (Hilbert or Random), baseline methods like TurboSL and ZNCC fail catastrophically (errors exceeding $30\text{120\text{ mm}\(), whereas NeLU3D reliably maintains sub-millimeter to low-millimeter accuracy (\)0.53\text}3.87\text{ mm}$).
• Enabling Low-Cost and Defocused Optics: When tested with an uncalibrated 50 USD consumer LCD projector suffering from focal curvature and severe optical distortion, TurboSL achieves MAEs of \(74.33\text{ mm}\) (3 patterns) and \(41.63\text{ mm}\) (11 patterns), whereas NeLU3D achieves \(0.89\text{ mm}\) and \(0.94\text{ mm}\), demonstrating true democratization of high-precision active scanning.
• Dynamic Microsecond Reconstruction: Paired with an analog 35mm film projector and a kHz strobe LED, NeLU3D achieved dynamic 3D captures at 200 fps (e.g., cloth impacts and plastic beam bending) using only one RGB pattern and one white flash frame.

Highlights & Insights

• Replacing Parametric Optics with an Empirical Continuous Neural Field: By encoding complex projector physics, vignetting, lens aberrations, and pixel gaps into a fast-training multiresolution hash encoding MLP, NeLU3D sidesteps cumbersome hardware modeling without precision loss.
• Physics-Informed Bound on Residual Irradiance: Capping estimated ambient illumination by the white flash intensity (\(I_{\text{ambient}} \le I_{\text{white}}\)) stabilizes neural inverse rendering, circumventing gradient explosions while eliminating the need for dedicated albedo or BRDF estimation networks.
• Inherent Regularization of Disordered Point Sets: Moving from discrete point-by-point table lookup to continuous implicit surface optimization leverages the inductive bias of neural SDFs, turning noisy, outlier-ridden active scans into watertight manifolds with high-fidelity normals.

Limitations & Future Work

• Bounded Reconstruction Volume: Reconstruction is fundamentally bounded by the volume swept during the one-time planar stage calibration. Any object geometry extending outside this calibrated bounding box (e.g., the skull held by the monkey figurine) cannot be reconstructed.
• Residual Artifacts under Mirror Specularity: The smooth ambient field cannot fully compensate for sharp specular highlights or sensor saturation, occasionally inducing local surface pitting on highly reflective objects.
• Non-Real-Time Optimization: Iterative inverse rendering requires roughly 7 to 10 minutes per frame (10,000 optimization steps), precluding instant online visualization.
• Camera Calibration Prerequisite: While projector modeling is completely removed, camera intrinsics still require standard calibration; future extensions could target joint camera-projector calibration-free architectures.

Related Work & Insights

• vs LookUp3D (Pereira et al., 2025): LookUp3D introduced uncalibrated projector-free active scanning via discrete color lookup tables, but suffered from prohibitive memory footprints ($10\text{30\text{ GB}$) and noisy point cloud outputs. NeLU3D compresses the representation by over 99.9% into a differentiable neural field and utilizes volumetric rendering to deliver smooth, watertight surfaces.
• vs TurboSL (Mirdehghan et al., 2024): TurboSL pioneered single-view neural inverse structured light, but requires rigid parametric calibration of the projector's pinhole geometry and blur kernels, struggling under cheap optics or generic non-custom patterns. NeLU3D decouples all projector parameters, running faster and handling arbitrary hardware rigs robustly.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant synthesis of empirical color lookup tables and neural implicit surface rendering, establishing a practical projector-unmodeled active scanning paradigm.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 4 distinct optical set-ups, 25+ diverse real-world targets, static and dynamic regimes, paired with CAD and 44-pattern dense SL baselines.
• Writing Quality: ⭐⭐⭐⭐⭐ Well-structured narrative, mathematically disciplined methodology, and clear, convincing illustrations.
• Value: ⭐⭐⭐⭐⭐ Dramatically lowers the cost and calibration complexity of industrial-grade structured light scanning, offering tremendous practical value to 3D vision and computational photography.