Closing the Capacity–Convergence Gap: Globally Optimal Configuration of Implicit Neural Representations

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/Sippengg/OptiINR
Area: 3D Vision
Keywords: Implicit Neural Representations, Bayesian Optimization, Activation Functions, Architecture Search, Capacity–Convergence Gap

TL;DR

OptiINR reformulates layer-wise activation selection, initialization scaling, and learning rate configuration as a mixed-variable black-box global optimization problem, using product-kernel Gaussian processes and Matheron's rule-accelerated Empirical Expected Improvement to bridge the capacity–convergence gap in coordinate-MLPs.

Background & Motivation

Implicit Neural Representations (INRs, or coordinate-MLPs) parameterize continuous signals—spanning 2D images, 3D geometry, audio waveforms, and neural radiance fields—as deep networks mapping coordinates to signal values, offering inherent resolution independence and storage efficiency over discrete grids. To combat spectral bias, where standard MLPs preferentially fit low frequencies and fail to capture high-frequency details, prior literature introduced high-capacity non-linearities such as sinusoidal (SIREN), Gabor wavelet (WIRE), Gaussian, and variable-periodic (FINER/FINER++) activations. However, these expressive activations are acutely sensitive to initialization scales and per-layer hyperparameters.

This interdependence induces a severe "capacity–convergence gap": architectures with superior theoretical representation capacity frequently suffer optimization instability or collapse into suboptimal local minima, where minor hyperparameter shifts can swing reconstruction performance by more than 10 dB PSNR. Prevailing workflows rely on manual trial-and-error or coarse grid search, while recent automated efforts like MIRE adopt greedy layer-by-layer selection. Because the spectral response of earlier layers fundamentally alters the input distribution and conditioning of downstream layers, greedy search is provably suboptimal.

Bridging this gap requires simultaneously optimizing activation choices, initialization parameters, and layer-wise training dynamics within a unified global landscape. Core idea: recast INR configuration as a mixed-variable black-box global optimization problem over a structured layer-wise space via OptiINR, employing a decoupled product-kernel Gaussian process surrogate alongside Matheron's rule-accelerated Monte Carlo Empirical Expected Improvement (EEI) to achieve sample-efficient, globally optimal INR configurations.

Method

Overall Architecture

OptiINR formulates the entire configuration of an \(L\)-layer coordinate-MLP as a global vector \(\Lambda_{\text{network}} = (\lambda_1, \dots, \lambda_L) \in \mathcal{L}\). Each layer tuple specifies a categorical activation family, a binary flag for scale-preserving initialization, and continuous hyperparameters including frequency/scale parameters and independent layer-wise learning rates. Fully training an INR to convergence and evaluating its validation performance (such as PSNR or IoU) acts as the expensive black-box objective \(f(\Lambda)\). OptiINR iteratively fits a mixed-variable Gaussian Process (GP) surrogate and utilizes Monte Carlo Empirical Expected Improvement (EEI) to navigate exploration versus exploitation across mixed discrete-continuous domains:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Target Signal & Task Definition<br/>Audio / Image / 3D Occupancy / PDE / NeRF"] --> B["Latin Hypercube Sampling Initialization<br/>Space-filling initial observation set"]
    B --> C["Mixed-Variable Product Kernel GP Fitting<br/>Continuous Matérn kernel × One-hot ARD categorical kernel"]
    C --> D["Matheron's Rule Fast Posterior Sampling<br/>Single kernel matrix inversion amortized"]
    D --> E["Empirical Expected Improvement EEI Acquisition<br/>Balances high-uncertainty exploration & high-mean exploitation"]
    E --> F["End-to-End Training of Selected INR Config<br/>Obtain ground-truth evaluation metric"]
    F -->|Iterative dataset update loop| C
    F -->|Budget exhausted| G["Optimal Configuration & Reconstructed Signal"]

Key Designs

1. Structured Mixed-Variable Global Configuration Space: Overcoming Greedy Layer-Wise Suboptimality

Prior automated methods rely on bottom-up greedy construction, failing to account for cross-layer coupling where intermediate feature representations depend heavily on both preceding spectral filtering and subsequent transformations. OptiINR establishes a joint configuration vector across all layers simultaneously, incorporating optional positional encoding (PE) and scale at the input level. For each hidden layer \(l\), a categorical variable \(\sigma_l\) selects from candidate activation families (e.g., SIREN, WIRE, FINER, FINER++, Gauss), a binary variable \(I_l \in \{0, 1\}\) chooses whether to enforce scale-preserving initialization, and continuous vector \(p_l\) tunes family-specific spectral parameters alongside layer-wise learning rates \(\alpha_l\). This structure enables shallow and deep layers to adapt distinct dynamics tailored to signal frequency characteristics.

2. Decoupled Product-Kernel Surrogate Model: Principled Mixed-Type Function Approximation

Because candidate architectures contain both discrete activation choices and conditional continuous hyperparameters, standard Euclidean kernels cannot accurately measure covariance. OptiINR designs a structured product covariance kernel:

\[k(\Lambda, \Lambda') = k_{\text{cont}}(\Lambda_c, \Lambda'_c) \times k_{\text{cat}}(\Lambda_{\text{cat}}, \Lambda'_{\text{cat}})\]

The continuous component employs a Matérn kernel with smoothness parameter \(\nu\) to accommodate non-smooth performance landscapes, fitted via GP marginal likelihood maximization. For categorical dimensions, one-hot encodings are paired with a Squared Exponential kernel featuring Automatic Relevance Determination (ARD), assigning dedicated length-scales \(\ell_j\) to each activation type. This guarantees that the posterior predictive variance shrinks toward the true function as sampling density increases.

3. Matheron's Rule and Empirical Expected Improvement: Efficient Exact Posterior Sampling

To handle batch evaluations and remain robust against model misspecification, OptiINR applies Monte Carlo Empirical Expected Improvement \(\widehat{\text{EI}}(\lambda) = \frac{1}{S} \sum_{s=1}^S \max(0, f^{(s)}(\lambda) - f_{\text{best}})\). Naive posterior path sampling incurs an impractical \(\mathcal{O}(Sn^3)\) cost due to repeated \(n \times n\) covariance factorizations. OptiINR leverages Matheron's rule for conditional simulation:

\[f_{\text{post}}(\cdot) \stackrel{d}{=} f_{\text{prior}}(\cdot) + k(\cdot, X)[K + \sigma_n^2 I]^{-1}(y - f_{\text{prior}}(X))\]

By precomputing the matrix inversion \([K + \sigma_n^2 I]^{-1}\) only once at cost \(\mathcal{O}(n^3)\), exact posterior evaluations across \(S\) sample paths scale at \(\mathcal{O}(Sn^2)\). This eliminates approximation errors from inducing-point or random-feature surrogates and allows vectorized GPU execution, reducing Bayesian optimization overhead to a negligible fraction of INR training time.

Loss & Training

During black-box evaluation, each sampled configuration is trained for 10,000 iterations using AdamW, optimized under its layer-specific searched learning rates \(\alpha_l\). The Bayesian optimization procedure begins with 30 initial configurations sampled via Latin Hypercube Sampling (LHS) for uniform coverage, followed by 100 BO iterations balancing exploration of uncertain regions with exploitation of high-performing modes to discover globally optimal configurations.

Key Experimental Results

Main Results

OptiINR was evaluated across four distinct modalities—2D images, 1D audio, 3D occupancy grids, and NeRF novel view synthesis—using standard 4-layer, 256-width MLP coordinate networks (8 layers for NeRF) to isolate configuration effects.

Task / Dataset	Metric	OptiINR (Ours)	Best Baseline	Gain	Baseline Breakdown
Image: Kodak (24 imgs avg)	PSNR (dB) ↑	41.38 ± 3.05	40.24 ± 3.23	+1.14 dB	FINER (40.24), SIREN (38.47), WIRE (38.69)
Image: DIV2K (16 patches avg)	PSNR (dB) ↑	46.24 ± 3.49	45.56 ± 3.84	+0.68 dB	FINER (45.56), SIREN (42.75), IGA (41.77)
Audio: Bach	PSNR (dB) ↑	60.84	54.94	+5.90 dB	FR (54.94), SIREN (52.59), IGA (52.35)
Audio: Counting	PSNR (dB) ↑	49.60	39.35	+10.25 dB	FINER (39.35), FR (36.93), SIREN (34.39)
Audio: Two Speakers	PSNR (dB) ↑	68.39	56.36	+12.03 dB	FR (56.36), IGA (42.39), FINER (42.27)
3D Occupancy: Dragon (\(512^3\))	IoU ↑	0.9936	0.9934	+0.0002	Gauss (0.9934), WIRE (0.9924), FR (0.9919)
3D Occupancy: Thai Statue (\(512^3\))	IoU ↑	0.9884	0.9871	+0.0013	Gauss (0.9871), IGA (0.9834), WIRE (0.9861)
Novel View Synthesis: NeRF Lego	PSNR (dB) ↑	25.63	25.05	+0.58 dB	ReLU+PE (25.05), WIRE (24.63), FINER (24.25)

Ablation Study

To isolate the contributions of the proposed structured search space versus the Bayesian optimization algorithm, ablation comparisons were conducted on Kodak under identical evaluation budgets:

Search Strategy / Method	Kodak Mean PSNR (dB)	Relative Gap	Note
OptiINR (Full Framework: Space + BO)	41.38	—	Product kernel GP + Matheron's rule EEI
Random Search (over OptiINR space)	41.23	-0.15 dB	Confirms the expressive search space delivers primary gains
Grid Search (over OptiINR space)	41.02	-0.36 dB	Coarse discretizations miss continuous hyperparameter optima
FreSh (Frequency Shifting baseline)	37.41	-3.97 dB	Fixed paradigm without joint layer-wise tuning
SPDER (Semi-periodic damping baseline)	35.03	-6.35 dB	Suffers from unadapted activation-initialization coupling

Key Findings

• Emergence of Heterogeneous Layer-Wise Patterns: For smooth images, OptiINR selects Gaussian or FINER++ in shallow layers for smooth interpolation and periodic SIREN in deeper layers for residual details. For high-frequency textures, WIRE wavelets dominate throughout the architecture.
• Search Space Architecture as the Core Driver: Operating random search over OptiINR's configuration space yields 41.23 dB, vastly outperforming existing automated baselines like FreSh (37.41 dB), showing that defining a joint activation-initialization space provides the foundational performance leap, while BO extracts the final 0.15–0.36 dB margin.
• NeRF Layer-Wise Dynamics: In volumetric radiance fields, OptiINR consistently pairs smooth activations in early layers to stabilize coarse geometry with oscillatory activations in deep layers for view-dependent specularities, alongside larger initial learning rates in shallow layers to expedite global convergence.

Highlights & Insights

• Reconceptualizing INR Architecture Search: Dispels the long-standing heuristic assumption that coordinate networks require a single activation family across all layers, proving that heterogeneous layer-wise pairings yield superior spectral control.
• Computationally Feasible Exact Posterior Sampling: By leveraging Matheron's rule, OptiINR avoids costly repeated Cholesky factorizations, executing exact posterior sampling without the variance distortions of sparse approximations.
• Transferable Training Insights: The discovered pattern of pairing higher learning rates and smoother activations in shallow layers with lower rates and high-frequency non-linearities in deeper layers offers actionable design guidelines for neural field architectures generally.

Limitations & Future Work

• Per-Signal Search Overhead: Discovering optimal configurations requires 130 full training runs per signal, which is computationally expensive for real-time streaming or low-latency deployment.
• Fixed Layer Depth and Channel Width: Network depth and layer widths remain fixed, leaving architectural depth and width exploration outside the active search space.
• Promising Direction: Integrating meta-learning (such as MetaSDF) to learn informative GP prior distributions across signal datasets, enabling the framework to pinpoint optimal configurations within a handful of Bayesian optimization iterations.

Related Work & Insights

• vs SIREN / WIRE / FINER: Prior works propose individual activation families with uniform network-wide application; OptiINR shows that combining complementary activations across depths yields superior performance over any single family.
• vs MIRE (CVPR 2025): MIRE performs greedy layer-by-layer activation selection; OptiINR proves greedy search is suboptimal due to non-convex layer coupling and solves the problem globally.
• vs MetaSDF / Meta-SparseINR: Meta-learning optimizes weight initializations across signal populations for fast adaptation; OptiINR optimizes architectural hyperparameters for individual signals, representing orthogonal and complementary directions.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates INR layer-wise configuration as a global mixed-variable optimization problem, discarding greedy layer-wise heuristics.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation spanning 1D, 2D, 3D, PDE, and NeRF modalities with thorough ablations separating space and search contributions.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured mathematical derivation connecting Matheron's rule with empirical acquisition optimization.
• Value: ⭐⭐⭐⭐⭐ Provides an open-source, principled methodology that systematically closes the capacity–convergence gap in coordinate-based neural representations.