Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

Conference: NeurIPS2026
arXiv: 2609.31107
Area: Optimization & Theory
Keywords: Bayesian optimization, Fisher information geometry, acquisition-gradient bounds, trust regions, high-dimensional optimization

TL;DR

The paper separates surrogate geometry from utility sensitivity through the pullback Fisher tensor of the posterior map, then replaces TuRBO's kernel-lengthscale weights with regularized local Fisher diagonal weights, achieving competitive SE-kernel benchmark performance without a regret or global convergence guarantee for outer Bayesian optimization.

Background & Motivation

Bayesian optimization (BO) seeks good solutions with few expensive black-box evaluations, fitting a probabilistic surrogate and optimizing an acquisition function to select each new query. In high dimensions, difficulty does not come only from the objective: even after fitting the surrogate, inner acquisition optimization can encounter large regions with almost no gradient. Larger lengthscales, Random Axis Aligned Subspace Perturbations (RAASP) near observed data, and trust regions can help, but they are usually explained separately through kernels, initialization, or local search rather than a shared sensitivity perspective.

An acquisition depends on the input through predictive-distribution parameters. A scalar Gaussian prediction, for example, maps an input to only its mean and variance; if substantial input movement barely changes these parameters, a sophisticated acquisition utility may still have little useful input gradient. This distinguishes sensitivity of the acquisition to the predictive distribution from sensitivity of that distribution to input movement. Although TuRBO adjusts its search-box shape using lengthscales, explicit kernel lengthscales are neither the local posterior sensitivity at the current position nor available for every probabilistic surrogate.

The paper treats the predictive family as a statistical manifold equipped with the Fisher–Rao metric, then pulls that metric back through the surrogate posterior map. This explains gradient collapse and constructs local search shapes without explicit lengthscales. Core Idea: shape the trust region using Fisher sensitivity of the predictive distribution to input changes, while leaving overall search scale and restart exploration to TuRBO's outer control.

Method

Overall Architecture

The theory considers a fixed surrogate within one BO iteration: the posterior map converts the input into predictive parameters, and the acquisition computes expected utility from those parameters. The pullback Fisher tensor describes local sensitivity of the first step, acquisition-side sensitivity describes the second, and together they bound the input gradient.

The algorithm is called Fisher-Information Trust Region (FITR). Each iteration fits the surrogate, estimates local Fisher diagonal entries at the current trust-region center by averaging squared input gradients of predictive log densities, adds a curvature floor, converts them to inverse-square-root weights, and applies volume normalization. The resulting axis-aligned search box bounds LogEI optimization; black-box evaluation then triggers the original TuRBO expansion, contraction, or restart rules.

The change concerns relative box side lengths, not the acquisition function, surrogate training objective, or success/failure counters. Experimental FITR-REI and TuRBO-REI share GP modeling and Regional Expected Improvement (REI) for restart-center selection; default FITR is a diagonal box, not a full Fisher ellipsoid. The mechanism is local matrix geometry, explained directly with formulas below rather than presenting a theoretical derivation as a network diagram.

Key Designs

1. Pullback Fisher tensor: measure local movement by predictive change rather than input distance

Write the posterior map as \(\varphi_t(\mathbf{x})=\bm{\theta}_{t,\mathbf{x}}\), with Jacobian \(J_t(\mathbf{x})\). The Fisher matrix \(\mathcal{I}_{\mathcal{S}}\) measures how predictive-parameter changes affect the distribution; the chain rule yields an input-space sensitivity tensor:

\[ \mathbf{g}_t(\mathbf{x})=J_t(\mathbf{x})^{\top}\mathcal{I}_{\mathcal{S}}\!\left(\varphi_t(\mathbf{x})\right)J_t(\mathbf{x}). \]

In any input direction, its quadratic form measures predictive change under a small displacement. Appendix D.7 shows that the second-order term of KL divergence between center and nearby predictions is half this displacement quadratic form. This provides a local probabilistic interpretation, not an exact KL constraint throughout a finite region.

For scalar Gaussian predictions, \(\Sigma\) denotes variance rather than standard deviation. The tensor consists of outer products of the mean and variance gradients:

\[ \mathbf{g}_t(\mathbf{x})=\frac{\nabla\mu_{t,\mathbf{x}}\nabla\mu_{t,\mathbf{x}}^{\top}}{\Sigma_{t,\mathbf{x}}}+\frac{\nabla\Sigma_{t,\mathbf{x}}\nabla\Sigma_{t,\mathbf{x}}^{\top}}{2\Sigma_{t,\mathbf{x}}^{2}}. \]

The first term captures mean changes relative to uncertainty, while the second captures changes in uncertainty itself; using only the mean gradient therefore misses part of the predictive geometry. A positive-definite Fisher matrix on the statistical manifold does not imply a positive-definite pullback tensor. Scalar Gaussian predictions have only two parameters, so the tensor has rank at most \(2\) and is necessarily rank-deficient when input dimension exceeds \(2\). A null direction means only first-order invariance of predictive parameters, not invariance of the true black-box function.

2. Acquisition-gradient factorization: explain when changing utility still cannot restore gradients

Proposition 4.2 requires a \(C^1\) posterior map, a positive-definite predictive-parameter Fisher matrix, acquisition reparameterization with input-independent base noise, valid interchange of gradient and expectation, and the required square-integrability of gradients. Let \(\bar\alpha\) denote the reparameterized integrand. Acquisition-side sensitivity and the gradient bound are:

\[ C_{\alpha}(\bm{\theta})=\mathbb{E}_{\bm{\zeta}}\!\left[\nabla_{\bm{\theta}}\bar\alpha(\bm{\theta},\bm{\zeta})^{\top}\mathcal{I}_{\mathcal{S}}(\bm{\theta})^{-1}\nabla_{\bm{\theta}}\bar\alpha(\bm{\theta},\bm{\zeta})\right],\qquad \|\nabla_{\mathbf{x}}\alpha(\mathbf{x})\|_2\leq\sqrt{C_{\alpha}(\bm{\theta}_{t,\mathbf{x}})\,\mathrm{tr}(\mathbf{g}_t(\mathbf{x}))}. \]

The proof first expresses the input gradient as the Jacobian transpose times the expected predictive-parameter gradient, then inserts the Fisher square root and inverse square root. Cauchy–Schwarz introduces the squared Frobenius norm of the geometric factor, equal to the pullback trace; Jensen's inequality bounds the acquisition factor by the expected quadratic form above. This does not require invertibility of the input-space pullback tensor, only of the predictive-parameter Fisher matrix.

When predictions revert to a nearly constant prior over a region, the pullback trace approaches zero; bounded acquisition-side sensitivity then forces small input gradients. The converse does not hold: a large trace does not guarantee a large gradient, the bound provides no lower bound, and it does not guarantee access to an optimum. Different surrogates with the same Gaussian predictive state share the acquisition-side formula, but may still have different input Jacobians.

The appendix derives sensitivity bounds for analytic EI, UCB, and PI. Analytic PI has a global bound, whereas EI and UCB additionally require control of predictive variance. The experiments use stabilized LogEI, for which the paper discusses local smoothness only on compact subsets with strictly positive variance; the explicit analytic-EI bound must not be treated as a global uniform LogEI bound.

The high-dimensional interpretation also has conditions. Distances between independent uniform points in the unit cube grow with the square root of dimension, so isotropic SE and Matérn kernels need corresponding lengthscale growth to avoid degenerate correlations. Under the asymptotic analysis in Appendix D.5, the critical pullback trace is still \(\Theta(D_x^{-1})\), not constant-order sensitivity. Lengthscale scaling mitigates faster degeneration rather than proving that gradients do not vanish. RAASP is interpreted as placing initialization near existing data instead of starting in regions with nearly constant posteriors.

3. Diagonal Fisher estimation and damping: turn rank-deficient geometry into finite coordinate weights

An ideal full Fisher region extends along low-sensitivity directions and contracts along high-sensitivity directions, but rank deficiency leaves null-direction radii uncontrolled. A full tensor also requires whitening and yields clipped regions that are difficult to sample efficiently at input boundaries. Default FITR therefore estimates only coordinate-wise Fisher diagonal entries, preserving usable directional stretching in an axis-aligned box without retaining all rotations and cross terms.

At the center prediction, draw \(S\) samples, back-propagate each predictive log density to the input, and average squared coordinate scores:

\[ \widehat g_{t,j}=\frac{1}{S}\sum_{s=1}^{S}\left(\left.\frac{\partial}{\partial x_j}\log p(\tilde y_s;\varphi_t(\mathbf{x}))\right|_{\mathbf{x}=\mathbf{x}_{t,c}}\right)^2,\qquad \tilde y_s\sim p(y;\varphi_t(\mathbf{x}_{t,c})). \]

Sample values are held fixed during differentiation: this is the input-direction score, not a total derivative through a sampling path that treats the sample itself as an input-dependent function. Appendix D.8 uses the score outer product and chain rule to show that its expectation equals the corresponding unregularized pullback diagonal entry. Unbiasedness applies only to this estimation step, not to weights after damping, inverse square roots, or normalization.

Damping uses the mean estimated diagonal curvature at the center plus small numerical jitter. More sensitive coordinates receive smaller raw weights and less sensitive coordinates receive larger ones:

\[ \varepsilon_t=\frac{1}{D_x}\sum_{j=1}^{D_x}\widehat g_{t,j}+\varepsilon_{\mathrm{jit}},\qquad r_{t,j}=(\widehat g_{t,j}+\varepsilon_t)^{-1/2},\qquad \varepsilon_{\mathrm{jit}}=10^{-12}. \]

Here \(r_{t,j}\) distinguishes raw weights from normalized weights, for which the paper reuses its weight notation. Mean curvature supplies an isotropic floor that prevents unlimited stretching of nearly zero coordinates. If every diagonal estimate is zero, jitter makes raw weights equal and normalization recovers an isotropic box. This is a conservative adjustment to search shape, not recovery of true black-box information through regularization.

4. Volume normalization and outer control: separate local shape from overall exploration

Using raw inverse-square-root weights as side lengths would let overall Fisher magnitude change search volume, conflating region shape and size. Both FITR and TuRBO divide weights by their geometric mean, making their product one, and let the common length parameter control the unclipped box volume:

\[ w_{t,j}=\frac{r_{t,j}}{(\prod_{k=1}^{D_x}r_{t,k})^{1/D_x}},\qquad \mathcal{X}^{\mathcal{TR}}_t=\left\{\mathbf{x}\in\mathcal{X}:\|\mathrm{diag}(\mathbf{w}_t)^{-1}(\mathbf{x}-\mathbf{x}_{t,c})\|_{\infty}\leq\frac{\kappa_t}{2}\right\}. \]

Each box coordinate is intersected with the input-domain bounds. Product normalization therefore fixes pre-clipping volume, while actual feasible volume may be smaller near boundaries. LogEI optimization within the box, true-objective evaluation, observation updates, and counters are unchanged from TuRBO-REI. The paper does not introduce a Fisher natural-gradient acquisition optimizer or replace this outer loop with exact KL constraints.

Retaining outer control matters because the authors tried an augmented-Lagrangian treatment of direct KL constraints and observed excessive local exploitation and inadequate exploration. FITR assigns relative side lengths to geometry, while expansion after successes, contraction after failures, and REI center selection after collapse preserve the existing exploration channel. Only FullFITR uses whitening with the full regularized tensor, as an additional ablation rather than the main implementation.

Loss & Training

This work does not train a new neural network. SE experiments fit a GP with dimension-scaled LogNormal lengthscale priors, normalize inputs to the unit cube, and standardize outputs. LogEI is optimized using L-BFGS with \(5\) restarts and \(256\) raw samples; Fisher diagonal estimation also uses \(S=256\), a separate sampling purpose.

Each run starts with \(30\) random observations and uses sequential evaluations, \(q=1\). The trust-region length doubles after \(10\) consecutive improvements, halves when failures reach \(\min\{\lceil\max(4/q,D_x/q)\rceil,20\}\), and triggers a restart below \(0.5^7\). The text does not provide directly verifiable complete evaluation budgets for each task; absent plot-axis information is not replaced with an invented common budget.

Key Experimental Results

Main Results

Results are primarily reported as curves. The cache contains captions and discussion but no verifiable numerical endpoints. The table records experimental scope and the authors' qualitative conclusions, not a numerical leaderboard or percentage gains.

Experiment	Explicit conditions	Comparison scope	Reported conclusion and evidence boundary
GP-SE main comparison (Figure 4)	9 continuous benchmarks, 11 independent seeds, best observed objective, lower is better	FITR-REI, TuRBO-REI, DSP, BOUNCE	FITR improves over TuRBO-REI on several tasks and is competitive with strong baselines elsewhere; no verifiable per-task endpoint values
GP-IBNN transfer check (Figure 5)	Depth 3, 5 independent seeds, also reported with lower being better	FITR versus a non-trust-region baseline	Improvements on some tasks, comparable performance elsewhere; not a full comparison against other non-lengthscale trust-region methods
Controlled main-comparison settings	30 initial observations, sequential evaluations q=1	FITR-REI and TuRBO-REI share GP modeling and REI center selection	The main difference is local weights, not a new restart strategy

The SE suite includes control and robotics tasks, HPA102-1 human-powered aircraft design, MOPTA08 automotive design, and LassoDNA and SVM tasks. The cached prose does not enumerate all nine tasks and their dimensions, so dimensions are not inferred from names. Figures follow benchmark minimization conventions while the algorithm is written in maximization form; these are not contradictory experimental outcomes.

Captions for Figures 4, 5, and 6 call the shaded bands standard error, but checklist item 7 uses “min-max shaded region” with an unclear explanation. This note records means and standard error following the captions while preserving the conflict; it does not relabel the bands as confidence intervals or significance tests.

Ablation Study

The following table summarizes ablations and diagnostics. Apart from fixed experimental settings, performance and timing entries are qualitative reports rather than numerical readings from plots.

Config / analysis	Explicit reported observation	What it supports	What it does not support
Default diagonal FITR	Regularization keeps anisotropy bounded while adapting to local predictions	Damping prevents uncontrolled stretching	No quantitative undamped ablation; no specific contribution percentage can be claimed
FullFITR	Evaluated only on a subset of tasks; did not outperform diagonal FITR	Full rotational geometry is not automatically better	Does not show the full tensor is worse on every task
Length trajectories (Figure 6)	FITR and TuRBO-REI lengths are qualitatively similar	Differences are more plausibly related to shape than different length control	Does not imply identical lengths or restart locations at every iteration
Candidate-generation time (Figure 6)	Same order as TuRBO, with modest overhead for local weights	Practical feasibility of the diagonal implementation	No verifiable seconds or fixed overhead ratio
KL constraint + ALM variant	Authors report excessive exploitation, reduced exploration, and degraded performance	Empirical motivation for retaining outer exploration	No complete quantitative table or degradation magnitude

Appendix diagnostics define anisotropy as the ratio of the largest to smallest normalized coordinate weight. A value near \(1\) means a nearly isotropic region, not optimal search efficiency. Rapid geometry variation near the center, boundary clipping, and numerical overhead are proposed explanations for FullFITR limitations, not individually established causal ablations.

Key Findings

• The cleanest comparison is FITR-REI versus TuRBO-REI with SE kernels: modeling and outer control are shared, while local side-length construction differs. Curve-level conclusions must not be elevated to uniform wins across all tasks.
• Independence from explicit lengthscales is a construction-level generalization. The IBNN comparison verifies that the same pipeline can be used, but does not isolate its advantage over other trust-region shapes.
• More complete geometry need not improve finite-region search: local accuracy at the center, geometry variation across the region, and feasible-boundary handling all affect gains.

Highlights & Insights

• Separating acquisition-side sensitivity from surrogate geometry helps locate the source of gradient failure. Replacing acquisition utility alone cannot guarantee recovery when the posterior map is nearly constant.
• Fisher estimation back-propagates predictive log densities to inputs without requiring exposed ARD lengthscales. This supplies a common interface for smooth probabilistic surrogates, provided densities, input derivatives, and regularity conditions remain available.
• Geometry controls shape while TuRBO controls volume and restarts, a more conservative transfer than treating KL constraints as a complete optimization strategy. Its reusable aspect is a local-weight replacement without rewriting the BO controller.

Limitations & Future Work

• The theory bounds gradients of inner acquisition optimization; it is not a BO regret bound and does not guarantee global convergence, sample efficiency, or a matching lower bound on actual gradients.
• Scalar Gaussian pullback tensors have rank at most \(2\); diagonal approximation removes rotations and damping changes the original geometry. The default box is neither an exact Fisher–Rao ball nor an exact KL trust region.
• Non-lengthscale, anisotropic surrogates may not exhibit the same high-dimensional trace collapse, and FITR gains are task-dependent. The IBNN experiment has only a non-trust-region comparator and cannot establish comprehensive superiority in that setting.
• Results are mostly curves and FullFITR covers only a subset of tasks; the standard-error versus checklist shading conflict needs clarification. The code checklist states that GitHub code exists, but the cache gives no concrete repository URL, so Code metadata is not guessed.
• Future work may study Fisher geometry variation across regions, tighter boundary handling for rotated regions, and batch or multi-objective extensions. These are research directions, not capabilities already validated by the current sequential scalar experiments.

Related Work & Insights

• vs TuRBO-REI: TuRBO scales boxes by ARD lengthscales; FITR uses regularized Fisher diagonal weights of the center posterior, with unchanged volume normalization and outer control. Appendix D.6 establishes only qualitative agreement that longer lengthscales imply wider search directions, not exact equality of weights.
• vs DSP and RAASP: DSP maintains kernel correlations through dimension-scaled priors; RAASP avoids flat regions through initialization. This paper explains both through pullback trace and then uses that sensitivity to shape local regions.
• vs GIT-BO: GIT-BO identifies a global active subspace using predictive-mean gradients; this paper considers predictive-distribution changes and locally shapes trust regions. No direct comparison is reported, so surrogate differences cannot be ignored to claim superiority.
• vs geometry-aware BO and natural gradients: Geometry-aware BO may exploit an intrinsic input manifold, whereas this paper obtains geometry from the surrogate posterior. Natural gradients change optimization-step directions; default FITR changes the permissible axis-aligned search region, which is a different operation.

Rating

• Novelty: 4/5 — Connects acquisition-gradient factorization with local posterior Fisher trust regions through an interface more general than kernel lengthscales.
• Experimental Thoroughness: 3/5 — Includes 9 SE benchmarks and repeated runs, but non-lengthscale comparisons and full-tensor coverage remain limited.
• Writing Quality: 4/5 — Claims and guarantee boundaries are largely explicit; shading statistics and some notation need clarification.
• Value: 4/5 — Provides an interpretable, replaceable trust-region shape construction whose gains require surrogate- and task-specific validation.