Be Tangential to Manifold: Discovering Riemannian Metric for Diffusion Models

Conference: ECCV 2026
Paper: Official paper page · PDF
Authors: Shinnosuke Saito, Takashi Matsubara
Area: Image Generation / Diffusion Models
Keywords: Riemannian metric, score Jacobian, geodesic interpolation, tangent and normal directions, classifier-free guidance

TL;DR

The paper turns a pretrained diffusion model's score Jacobian into a training-free Riemannian metric that favors motion along the data manifold during interpolation and guidance, reducing Animal Faces-HQ interpolation FID from GeoDiff's 25.80 to 21.01 and generation FID from 17.28 to 16.04 at CFG scale 12.5.

Background & Motivation

VAEs and GANs usually provide explicit low-dimensional latent spaces in which a decoder can pull image-space distances back to define paths respecting the generator's geometry. Diffusion models lack comparable low-dimensional semantic coordinates: even though Stable Diffusion uses compressed latent variables, its noise space has the same dimensionality as its denoising representation, rather than serving as coordinates specifically parameterizing a semantic manifold. Consequently, linear interpolation, LERP, can pass through unnatural regions; spherical interpolation, SLERP, preserves norms but does not identify the manifold's local orientation.

Existing geometric approaches address this in different ways. GeodesicDiffusion, abbreviated GeoDiff below, constructs distances from density to favor high-density regions; a FIM-inspired approach constructs a metric from the score direction. Yet a more probable sample does not necessarily contain better perceptual detail, and attraction toward density peaks can over-smooth intermediate images. More fundamentally, a score vector supplies only one direction, whereas the normal space of a manifold embedded in a high-dimensional representation is typically multidimensional. One direction cannot explain all ways of leaving that manifold.

The paper builds on earlier observations about the score Jacobian's spectrum: small singular values correspond to approximate tangent directions, while large ones correspond to approximate normal directions. Its contribution is not discovering this separation for the first time, but turning it into a metric usable for path optimization and sampling correction. Core idea: charge for motion according to the resulting change in the score, making normal displacement expensive and tangential motion cheap, so operations respect manifold orientation rather than merely seek high probability.

Method

Overall Architecture

The inputs are a pretrained score network and a noisy sample; the central construction uses local network derivatives to define the cost of moving in different directions. This metric supports two independent applications: global interpolation fixes two endpoints and optimizes the intermediate path, while local guidance correction suppresses off-manifold CFG components at each denoising step. These are not modules that must execute sequentially.

The interpolation branch first maps clean endpoints to a fixed noise time using DDIM inversion, optimizes a path there, and deterministically denoises its points. The guidance branch starts from the current conditional score and CFG increment, finding a correction that retains textual guidance while reducing geometrically expensive motion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pretrained score network<br/>Noisy sample"] --> B["Jacobian Pullback Metric"]
    B -->|Fixed-endpoint path| C["Score-Difference Geodesics"]
    B -->|Per-step CFG increment| D["Metric-Based Guidance Correction"]
    E["Image endpoints<br/>DDIM inversion"] --> C
    C --> F["Deterministic denoising<br/>Interpolated images"]
    D --> G["Corrected denoising step<br/>Generated image"]
    G -->|Next timestep| B

Key Designs

1. Jacobian Pullback Metric: measure motion costs using all local directions

Let \(s_\theta(x_t,t)\) denote the score function and \(J_{x_t}=\nabla_{x_t}s_\theta(x_t,t)\) its input Jacobian. Here, the score is the model's estimate of a log-density gradient, not an image-quality rating. Equation (1) defines:

\[ g_{x_t}(v,w)=\langle J_{x_t}v,J_{x_t}w\rangle =v^\top G_{x_t}w,\qquad G_{x_t}=J_{x_t}^\top J_{x_t}. \]

Intuitively, a small step is expensive if it changes the score substantially, even when its Euclidean length matches that of another step. In the Jacobian's singular directions, each cost is controlled by the corresponding squared singular value. Small-singular-value directions are inexpensive and approximately tangential; large-singular-value directions are expensive and approximately normal. This is richer than the FIM-inspired metric \(I+\lambda s_\theta s_\theta^\top\), which emphasizes only one direction beyond its isotropic background and cannot distinguish multiple normal directions. The practical algorithms do not require materializing the full Jacobian or computing a full SVD.

There is an important condition: \(J^\top J\) is always positive semidefinite, but is positive definite only when the Jacobian is full-rank. The authors argue that realistic noisy data at intermediate times \(t>0\) give full-rank Jacobians while retaining a spectral gap; an ideal clean low-dimensional manifold at \(t=0\) can instead be degenerate. The association between this gap and tangent/normal directions is the geometric approximation supporting the method, not a strict guarantee for every network and noise level.

2. Score-Difference Geodesics: fix endpoints and smooth score changes along the path

At a fixed time \(\tau\), path energy integrates the squared metric norm of its velocity. Applying the chain rule gives the following form of Equation (2). Its cached typesetting is damaged; this expression uses standard notation reconstructed from Equation (1) and the surrounding explanation:

\[ E[\gamma]=\frac12\int_0^1\|J_{\gamma(u)}\gamma'(u)\|_2^2\,du =\frac12\int_0^1\left\|\frac{d}{du}s_\theta(\gamma(u),\tau)\right\|_2^2\,du. \]

Discretizing the path into \(N+1\) points with spacing \(\Delta u=1/N\) replaces the directional derivative along the path with differences between neighboring scores. The cached rendering of Equation (3) is also damaged. The following finite-difference expression follows from the energy above; its coefficient and notation should be checked against the original paper:

\[ E_{\mathrm{disc}}=\frac{1}{2\Delta u}\sum_{i=0}^{N-1} \left\|s_\theta(x_\tau^{(u_{i+1})},\tau)-s_\theta(x_\tau^{(u_i)},\tau)\right\|_2^2. \]

This avoids explicitly constructing a high-dimensional metric matrix: evaluate network outputs along the discrete path and optimize the intermediate points. DDIM inversion supplies the fixed endpoints, SLERP initializes the interior, and the DDIM reverse process finally maps all path points back to images. The optimization changes the input path, not the score network's parameters. Compared with closed-form LERP/SLERP, it adds numerical optimization, but this is the same broad class of overhead incurred by other geodesic methods.

Minimizing score variation differs from attracting samples toward a density peak. A noisy sample near an endpoint can move parallel to the manifold without first being pulled into its densest region. The paper interprets this as preserving endpoint probability levels, but this note does not elevate that interpretation to a strict constant-density theorem for arbitrary distributions. The high-dimensional experiments assess reconstruction, perceptual transitions, and distributional distance, rather than directly verifying constant density along the entire path.

3. Metric-Based Guidance Correction: preserve guidance intent while softly suppressing expensive directions

During sampling, Algorithm 2 uses the conditional score \(s=s_\theta(x_t,t,c)\) as its base and defines \(\Delta s\) as the full CFG score minus that base. This base should not be mistaken for the unconditional score. The objective keeps the corrected update close to the original CFG update in Euclidean distance while keeping it close, under the proposed metric, to the update without the extra guidance increment. Approximating adjacent-time manifold changes as smooth and the local path with a single interval yields a linear system:

\[ (I+\lambda G_{x_t})\Delta\hat s=\Delta s. \]

For an ideal exact solution, a direction with singular value \(\sigma\) is scaled by \(1/(1+\lambda\sigma^2)\): large-singular-value normal components are attenuated more than small-singular-value tangent components. This interpretation follows from the system; it is not a hard projection onto a known tangent space, nor does it mean the implementation scales every component exactly this way. Instead of inverting a full matrix, the paper initializes at \(\Delta s\) and takes just one conjugate-gradient update:

\[ \epsilon=\Delta s-(I+\lambda G_{x_t})\Delta s,\qquad \Delta\hat s\approx\Delta s+ \frac{\epsilon^\top\epsilon}{\epsilon^\top(I+\lambda G_{x_t})\epsilon}\epsilon. \]

Computing \(Gv\) ordinarily requires a Jacobian-vector product followed by a transpose-Jacobian-vector product. The authors approximate the former with finite differences and, under the empirical assumption that the Jacobian is approximately symmetric, use the same type of finite difference for the latter. This adds 4 score-network evaluations per step. Baseline CFG uses 2, giving approximately 3 times the baseline cost overall. Training-free does not mean computation-free. Applying correction to a distillation teacher instead allows the student to inherit the benefit without additional student inference cost.

A Worked Example

Consider a semantically similar pair of Animal Faces-HQ images. DDIM inversion first maps both images to \(\tau=0.6T\), with \(T=50\) in the experiments. SLERP initializes noisy representations for 9 intermediate images, giving 11 total path points with both endpoints fixed. During 500 path-optimization iterations, a segment crossing directions of rapid score change incurs larger neighboring-score differences. Optimization can then adjust its interior points toward a less expensive route instead of explicitly increasing sample density.

Deterministic denoising produces the reconstructed endpoints and intermediate sequence. Perceptual and distributional metrics indirectly assess texture preservation, while RE separately checks endpoint damage. This example follows the experimental setup; it is not an additional measured single-pair result. CFG correction is a separate application branch and is not implicitly added to this interpolation procedure.

Loss & Training

Image experiments use Stable Diffusion v2.1-base and Adam for path optimization, with a learning rate decayed from \(10^{-3}\) to \(10^{-4}\) by cosine annealing. DDIM inversion and denoising use neither CFG nor negative prompts. However, geodesic computation uses an update incorporating negative prompts as its score function and adopts GeoDiff's prompt-embedding adjustment. Thus, training-free means no retraining of the diffusion backbone, not the absence of input-path or prompt-embedding optimization.

Guidance experiments use finite-difference step size \(h=10^{-4}\) and weight \(\lambda=0.1\), applying local correction across 50 sampling steps. Distillation follows the LCM protocol and default diffusers settings with AdamW, and the student uses 4 inference steps. Appendix D.3 is absent from the cache, so training iterations, batch size, and other unreported details cannot be filled in from this source.

Key Experimental Results

Main Results

Image interpolation is evaluated on MorphBench animation/metamorphosis, MB(A)/MB(M), CelebA-HQ, CA, and Animal Faces-HQ, AF. CA and AF each use 50 endpoint pairs selected to have LPIPS below 0.6. PPL sums adjacent-image LPIPS, PDV is the standard deviation of adjacent LPIPS, RE is endpoint reconstruction mean squared error, and FID compares reference and interpolated image-feature distributions; lower is better for all four. CA/AF FID uses 900 interpolated images and 200 reference images, a small-sample protocol not directly comparable to standard large-sample generation FID.

The table selects three representative methods from paper Table 2. RE retains the source's \(10^{-3}\) units rather than reporting unscaled raw values.

Dataset	Method	PPL	PDV	FID	RE (\(10^{-3}\))
MB(A)	SLERP	0.644	0.030	62.81	0.401
MB(A)	GeoDiff	0.402	0.024	28.70	0.188
MB(A)	Ours	0.380	0.021	27.44	0.177
MB(M)	SLERP	1.065	0.055	48.99	0.397
MB(M)	GeoDiff	1.021	0.073	38.12	0.272
MB(M)	Ours	0.977	0.073	36.00	0.201
CA	SLERP	0.707	0.033	37.84	1.010
CA	GeoDiff	0.669	0.044	35.98	0.891
CA	Ours	0.633	0.036	32.54	0.888
AF	SLERP	0.871	0.022	26.07	2.049
AF	GeoDiff	0.842	0.027	25.80	1.969
AF	Ours	0.767	0.023	21.01	1.962

To address the gap between better indirect metrics and greater manifold faithfulness, video experiments take the first and last of three frames as inputs and evaluate against the actual middle frame. DAVIS, Human, and RE10K contain 21, 56, and 26 clips, respectively, resized to \(512\times512\). The following values are selected from paper Table 3. This tests geometric plausibility, not superiority over dedicated video interpolation systems.

Dataset	GeoDiff MSE (\(10^{-3}\))	Ours MSE (\(10^{-3}\))	GeoDiff LPIPS	Ours LPIPS
DAVIS	13.253	8.777	0.334	0.318
Human	3.363	2.018	0.184	0.170
RE10K	5.941	2.771	0.229	0.178

Ablation Study

Evidence boundary: the cache contains the main paper and references, but not the component ablations in Appendix E. The following table therefore reports guidance-scale sensitivity and method comparisons from Table 4, not component-removal ablations. It cannot establish which approximation contributes most. Generation experiments use MS-COCO 2014 validation prompts to generate 30,000 images; lower FID and higher CLIP Score are better.

CFG Scale \(w\)	CFG FID	CFG++ FID	Ours FID	CLIP Score for All Three Methods
5.0	11.69	11.87	11.53	0.313
7.5	14.29	13.98	13.81	0.314
12.5	17.28	17.76	16.04	0.315

At \(w=12.5\), the absolute FID reduction over CFG is 1.24. CLIP Score is identical at the reported three-decimal precision, which does not prove perfectly unchanged semantics. Table 5 reports 4-step distilled FID of 17.91 versus 16.98, with CLIP Score of 0.306 for both. These are means over 5 runs; the authors report improvement in every run and \(p<0.05\) under a one-sided exact binomial test.

Key Findings

• In the image table, the method achieves the best PPL, FID, and RE on every dataset, but the best PDV only on MB(A); SLERP has lower PDV elsewhere. Shorter, clearer paths and more uniform perceptual steps are different objectives.
• On synthetic C-shaped data, 50 endpoint pairs give path-density standard deviations of 0.1606 for LERP, 0.0833 for SLERP, 0.1073 for the density-based method, and 0.0701 for the proposed method. This supports probability-level preservation on that distribution, not strict constant density for high-dimensional real images.
• Correction provides a larger benefit relative to CFG at higher guidance scales, but this is not an ablation of \(\lambda\), finite-difference step size, or CG iteration count. The available cache provides no verifiable numerical sensitivity results for those choices.

Highlights & Insights

• The method separates where to move from where probability is high. Derivatives of the score identify directional structure, avoiding the assumption that high density is a suitable proxy for perceptual quality.
• One geometric definition supports both global paths and local updates. Score differences optimize the former, while a linear system attenuates expensive motion for the latter, without learning another semantic latent space.
• Expensive geometric correction can be placed on the teacher side. Distillation transfers corrected sampling behavior to a student, providing evidence for reducing deployment overhead, although teacher training and sampling costs remain.

Limitations & Future Work

• The geometric assumptions have a limited scope. Full rank, a spectral gap, and approximate Jacobian symmetry are not automatically guaranteed for every pretrained network. Transfer to flow matching and newer architectures remains future work identified by the authors.
• Computation is substantial. Interpolation takes 500 path-optimization iterations, and direct CFG correction uses approximately 3 times as many network evaluations. Removing additional student inference cost through distillation does not make the overall method free.
• The exact system and implementation differ. One-step CG, finite differences, and replacing transpose operations through approximate symmetry may jointly affect correction accuracy. Comparing exact automatic-differentiation products, more CG iterations, and the current implementation would help, but the cache contains no numerical ablations of these choices.
• Evaluation centers on an older backbone and restricted pairs. CA/AF select semantically similar endpoints, and video experiments use consecutive three-frame sequences. These do not establish large cross-semantic transformations or long-video temporal stability; small-sample FID adds interpretive uncertainty.
• The available evidence is incomplete. The main text refers to SD v2.0-base, spectral-gap visualizations, and ablations in an appendix absent from the local cache. This note supplies neither their missing numbers nor code or arXiv links unconfirmed by the cache.

Related Work & Insights

• Versus GeoDiff: GeoDiff defines noise-space paths through density, whereas this paper defines directional costs through a Jacobian pullback metric. Both require path optimization; the main distinction is avoiding attraction toward high density as a geometric proxy.
• Versus the FIM-inspired metric: The anisotropic part of \(I+\lambda s_\theta s_\theta^\top\) represents one direction, while this method uses the full local spectral structure. The paper's FIM baseline is an author reimplementation, so results should be interpreted together with its implementation and hyperparameters.
• Versus CFG++ and TCFG: These methods also seek to reduce manifold deviation. This paper differs by explicitly defining a metric before deriving a soft correction. The quantitative main-paper table compares CFG++, but contains no same-protocol TCFG result, so it cannot establish comprehensive superiority over TCFG.
• Further directions: The metric could support editing or noise-space clustering, but this requires checking spectral structure in the chosen backbone before testing downstream gains. The demonstrated applications are interpolation and guidance, not completed evaluations of those extensions.

Rating

• Novelty: 4/5. Turning prior tangent/normal spectral observations into an operational pullback metric unifies two applications, but does not introduce manifold spectral structure for the first time.
• Experimental Thoroughness: 3/5. Synthetic data, images, actual video frames, generation, and distillation provide varied evidence, but the local source lacks the ablation appendix and the backbone and evaluation scale remain limited.
• Writing Quality: 4/5. The theory-to-algorithm chain is clear, with care needed to distinguish approximate solutions, geometric interpretations, and strict guarantees; damaged cached equation typesetting is not counted against the paper's writing.
• Value: 4/5. The method supplies reusable manifold-aware operations, with practical deployment requiring a trade-off against additional network evaluations and path optimization.