FlowFace: Rectifying Identity Conditioning with Riemannian Geometry for Face Generation¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/SunFly0/Flowface
Area: Human Understanding
Keywords: Face Personalization, Riemannian Geometry, Diffusion Model, BCH Expansion, Conditioning Geometry Rectification

TL;DR¶

FlowFace tackles identity drift and attribute order sensitivity in tuning-free face customization by proposing a fiber-bundle dual-stream manifold encoder, Lie-algebraic attribute composition via truncated BCH expansion, and training-time local SPD metric regularization, boosting identity preservation and composition stability without modifying the inference sampler.

Background & Motivation¶

Diffusion-based face personalization has rapidly evolved from test-time per-subject optimization to tuning-free adapters on large-scale models such as SDXL. Dominant paradigms like IP-Adapter, InstantID, and PhotoMaker extract reference identity features and inject them into text-conditioning spaces or UNet intermediate features via token substitution, concatenation, addition, or residual cross-attention branches. While the underlying denoising UNet is highly nonlinear, the conditioning interface remains fundamentally flat and Euclidean, treating facial identity and various edit attributes as mutually independent additive offsets. However, valid facial data distributions reside on curved, tightly coupled low-dimensional manifolds; repeatedly applying naive Euclidean linear additions across iterative denoising steps routinely forces conditioning trajectories into low-density, semantically invalid regions, triggering visible identity drift, severe texture leakage under pose variations, and background artifacts.

This geometric-algebraic mismatch becomes especially severe in compositional multi-attribute editing scenarios. From a user's semantic perspective, facial attribute composition should be commutative: applying "smile" followed by "age" ought to yield the exact same subject displaying both attributes. However, while operations at the interface appear linear, the nonlinear cross-attention mechanisms combined with iterative accumulation over reverse diffusion steps break this commutativity in practice: altering the prompt or injection order of attributes produces noticeable discrepancies in facial geometry and identity attributes. Furthermore, prior efforts that extend diffusion models to Riemannian manifolds typically necessitate solving geodesic differential equations or executing manifold-aware SDE sampling at every single denoising step, which introduces prohibitive computational overhead and numerical instability on high-resolution foundation models like SDXL.

The core angle of attack in FlowFace is to decouple training-time geometric rectification from inference-time sampling efficiency: instead of running expensive numerical integration along curved manifolds during inference, the framework reshapes the conditioning geometry during training so that standard Euclidean operations naturally track geodesic-consistent paths. Core idea: disentangle geometric pose variation from semantic identity using a fiber-bundle dual-stream encoder, introduce an explicit Lie-algebraic interaction term via truncated BCH expansion to absorb coupled edit deviations, and enforce geodesic consistency with a local SPD metric tensor during training, thereby achieving robust face personalization while preserving standard DDIM inference sampling.

Method¶

Overall Architecture¶

FlowFace is built upon a frozen SDXL-base-1.0 backbone, leaving the VAE, text encoders, and denoising UNet entirely untouched while training only lightweight geometric conditioning modules. Given $N$ reference face images and a text prompt specifying attribute edits, the system first extracts reference face embeddings and inputs them into a dual-stream manifold encoder, which decouples them into a geometry base stream and a semantic identity fiber stream to yield token-level manifold positions. For multi-attribute compositional editing, smooth vector fields parameterized by SIREN are constructed, and a truncated Baker-Campbell-Hausdorff (BCH) expansion explicitly evaluates a Lie bracket interaction term to update conditioning coordinates. Next, updated coordinates are resampled into a fixed number of identity tokens via a Perceiver resampler and injected into UNet cross-attention layers using spatial face region gating. During training, a local symmetric positive definite (SPD) metric network supervises geodesic consistency and directional covariant smoothness, effectively rectifying the intrinsic geometry of the conditioning space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Reference Faces + Edit Text Prompt"] --> B["Dual-Stream Manifold Encoder<br/>Base geometry & fiber semantics disentanglement"]
    B --> C["Lie-Algebraic Attribute Composition<br/>SIREN vector fields & truncated BCH expansion"]
    C --> D["Training-Time Metric Rectification<br/>Local SPD metric & geodesic consistency"]
    D --> E["Region-Gated Cross-Attention Injection<br/>Standard DDIM sampling to target image"]

Key Designs¶

1. Dual-Stream Manifold Encoder: Disentangling Geometry from Identity Semantics Conventional identity adapters feed raw facial embeddings directly into projection layers, causing transient geometric variations such as head pose and facial expression to entangle with essential identity cues, precipitating severe feature leakage under extreme angles. Inspired by fiber bundles in differential geometry, FlowFace decomposes facial representations into a base space and a fiber space: the base stream $B = f_{\text{base}}(E)$ captures geometry-sensitive variations, while the fiber stream $F = f_{\text{fiber}}(E)$ preserves identity-sensitive traits. During encoding, the fiber stream aggregates multi-reference cues through self-attention and is geometrically modulated by a transport signal $\phi(B)$ predicted from the base stream: $$F' = F + \text{SelfAttn}(F) + \phi(B)$$ The final token-level manifold positions are formed via layer-normalized fusion: $U = \text{LN}(W_b B + W_f F') \in \mathbb{R}^{B \times N \times d_m}$, with a pooled coordinate $u = \text{Pool}(U)$ extracted for metric computation. This asymmetric modulation ensures identity stability across varied poses before being resampled into $T=16$ compact identity tokens $T_{\text{id}}$ via a Perceiver-style resampler.

2. Lie-Algebraic Attribute Composition: Explicit Interaction Modeling via Truncated BCH When composing multiple facial edits (e.g., simultaneously applying "smile" and "age"), standard vector addition implicitly assumes attribute orthogonality, whereas physical facial attributes are intricately coupled in muscle deformation and skin texture, causing reverse diffusion trajectories to diverge under different execution orders. FlowFace models each attribute edit as a smooth vector field $V(\cdot; a): \mathbb{R}^{d_m} \to \mathbb{R}^{d_m}$ over conditioning coordinates, parameterized by a SIREN network with periodic sine activations for stable higher-order derivatives. Given two edit fields $X(\cdot) = V(\cdot; a_x)$ and $Y(\cdot) = V(\cdot; a_y)$, their noncommutative geometric interaction is captured explicitly by the Lie bracket: $$[X, Y](u) = J_Y(u)X(u) - J_X(u)Y(u)$$ evaluated efficiently via Jacobian-vector products (JVPs). FlowFace adopts a second-order truncated Baker-Campbell-Hausdorff (BCH) expansion to synthesize the composite edit field: $$V_{\text{fused}}(U) = X(U) + Y(U) + \frac{1}{2}[X, Y](U)$$ Because the Lie bracket is antisymmetric, reversing the edit order merely flips the sign of the interaction term, replacing erratic compositional divergence with a controlled, geometrically grounded first-order correction. The updated coordinate $U^+ = U + \eta V_{\text{fused}}(U)$ is passed through the resampler to regenerate $T_{\text{id}}$ without altering the diffusion backbone.

3. Training-Time Metric Rectification: Geodesic Consistency and Local Metric Reshaping To avoid costly Riemannian numerical integration during inference, FlowFace implements a relaxed strategy: rectifying the geometry during training while maintaining Euclidean paths at test time. A metric network maps the pooled coordinate $u$ to a lower-triangular Cholesky factor $L(u)$ with strictly positive diagonal entries enforced by softplus offsets, constructing a strictly symmetric positive definite (SPD) local Riemannian metric tensor $G(u) = L(u)L(u)^\top \succ 0$. During training, for any reference pair $(I_1, I_2)$, the straight line connecting their coordinates is partitioned into $S=20$ segments to compute a discrete geodesic length surrogate: $$d_g(u_1, u_2) = \sum_{s=1}^{S} \sqrt{\Delta u^{(s)\top} G(m^{(s)}) \Delta u^{(s)}}$$ which is aligned with the perceptual identity distance $d_{\text{perc}}(I_1, I_2) = 1 - \cos(\bar{e}_1, \bar{e}_2)$ extracted from frozen face recognition embeddings via $\mathcal{L}_{\text{geo}}$. To suppress high-frequency oscillations in the learned vector fields, a directional covariant acceleration regularizer $\mathcal{L}_{\text{cov}} = \mathbb{E}[\|J_V(u)V(u; a)\|_{G(u)}^2]$ penalizes directional derivatives along flow trajectories. This dual constraint embeds manifold curvature directly into the learned coordinates, ensuring that simple Euclidean operations stay close to the data manifold.

4. Region-Gated Cross-Attention Injection: Spatial Decoupling of Facial Identity Injecting reference facial features globally across all attention maps often causes identity textures to bleed into the background or ambient lighting to degrade facial boundaries. FlowFace establishes a dedicated cross-attention branch for identity injection parallel to the text condition. Using a face segmentation mask $M \in [0, 1]^{H_m \times W_m}$, the mask is spatially resized and flattened at UNet layer $\ell$ into an attention gate $g_\ell$. The layer output is computed as: $$H_\ell^{\text{out}} = H_\ell^{\text{txt}} + \alpha \cdot \Big(g_\ell \odot \text{Attn}(Q_\ell, K_\ell^{\text{id}}, V_\ell^{\text{id}})\Big)$$ where $(K_\ell^{\text{id}}, V_\ell^{\text{id}})$ are linear projections of identity tokens $T_{\text{id}}$, and $\alpha$ balances identity injection strength, guaranteeing precise spatial localization of facial identity.

Loss & Training¶

The overall training objective combines denoising loss with geometric regularizers: $$\mathcal{L} = \mathcal{L}_{\text{denoise}} + \lambda_{\text{geo}}\mathcal{L}_{\text{geo}} + \lambda_{\text{cov}}\mathcal{L}_{\text{cov}}$$ where $\mathcal{L}_{\text{denoise}}$ is the standard SDXL latent noise prediction loss, with hyperparameters set to $\lambda_{\text{geo}} = 0.1$ and $\lambda_{\text{cov}} = 0.1$. The SDXL VAE, text encoders, and UNet backbone remain completely frozen, optimizing only the dual-stream encoder, resampler, SIREN vector field, metric network, and attention projections. The model is trained using Adam with a constant learning rate of $1 \times 10^{-4}$ for 100k steps across 4 NVIDIA RTX 4090 GPUs with a batch size of 4. During inference, standard DDIM sampling with classifier-free guidance scale 7.5 is employed, matching the inference speed of lightweight adapter baselines.

Key Experimental Results¶

Main Results¶

Evaluation is conducted on an identity-cluster benchmark constructed from FFHQ and CelebA-HQ (comprising 5,000 held-out test identities) with zero train-eval leakage. Metrics include SimRef (maximum ArcFace cosine similarity to $N$ reference images), SimTarget (cosine similarity to a held-out same-identity target image), Paste Score ($\text{SimRef} - \text{SimTarget}$, where lower values indicate reduced reference copying and superior generalization), CLIP-T text alignment, and image quality measured by FID.

Method	SimRef ↑	SimTarget ↑	CLIP-T ↑	Paste ↓	FID ↓
IP-Adapter	64.5	55.8	73.6	8.7	89.2
FastComposer	56.2	48.6	75.4	7.6	92.5
InstantID	68.7	58.3	71.8	10.4	82.4
PhotoMaker	62.8	54.2	74.5	8.6	86.7
StoryMaker	65.4	56.8	74.8	8.6	85.3
HiFi-Portrait	71.8	60.4	72.5	11.4	79.2
FlowFace (Ours)	71.3	64.8	77.2	6.5	76.8

Ablation Study¶

Ablation experiments systematically isolate and replace individual components of FlowFace: replacing the dual-stream manifold encoder with a single-stream MLP, replacing Lie-bracket fusion with naive vector addition $X + Y$, removing geodesic regularization $\mathcal{L}_{\text{geo}}$, and replacing covariant regularization $\mathcal{L}_{\text{cov}}$ with finite differences.

Config	SimTarget ↑	CLIP-T ↑	Expr.Acc ↑	FID ↓	Note
FlowFace (full model)	64.8	77.2	52.4	76.8	Complete proposed framework
w/o Manifold Encoder	48.3	68.5	38.2	95.4	Replaced by single-stream MLP; severe identity entanglement
w/o Lie-Bracket Fusion	52.1	71.2	42.6	88.7	Replaced by linear addition; noticeable composition interference
w/o Geodesic regularization	55.7	74.8	45.3	82.1	Metric unconstrained by identity distance; off-manifold drift
w/o Covariant regularization	58.4	75.1	47.8	79.5	Vector field oscillates; compromised transition smoothness

Key Findings¶

Dominant Impact of Manifold Encoder and Lie-Bracket Fusion: Removing the dual-stream manifold encoder drops SimTarget dramatically by 16.5 points (from 64.8 to 48.3), proving that separating pose geometry from identity semantics is essential to prevent collapse under pose changes. Disabling the Lie-bracket fusion degrades SimTarget by 12.7 points and reduces expression accuracy, underscoring the necessity of second-order interaction modeling in coupled attribute edits.
Substantial Reduction in Order Discrepancy: In two-attribute composition tests, replacing BCH with linear addition causes the order gap metric OrderGap-ID to jump from 0.052 to 0.146. For three-attribute permutations, FlowFace reduces the worst-case permutation discrepancy Worst PermGap-ID from 0.291 to 0.118 while lifting all-attribute editing success from 58.4% to 81.2%.
Efficiency and Reference Scaling Trade-offs: Identity fidelity plateaus at $N=4$ reference images (SimTarget=64.8 vs. 66.2 at $N=8$). In terms of inference speed, FlowFace with $r=2$ truncation requires 12.4s and 10.5GB VRAM on an RTX 4090, remaining competitive with InstantID (11.7s, 9.6GB) while offering vastly superior identity consistency and image quality.

Highlights & Insights¶

Training-Time Geometry Rectification without Inference Overhead: Instead of running computationally prohibitive manifold SDE solvers during denoising, FlowFace reshapes the conditioning metric at training time, allowing standard Euclidean linear updates at inference to reliably follow manifold-consistent paths.
Lie Bracket as an Elegant Solution to Commutativity Failure: By recognizing that iterative cross-attention and denoising steps break semantic commutativity, the authors adopt the Lie bracket to provide an explicit first-order interaction correction, offering an interpretable and mathematically principled tool for multi-attribute composition.
Fiber-Bundle Inductive Bias for Face Modeling: Formulating facial features as a base geometry space modulating an identity fiber space provides an effective inductive bias that outperforms conventional unconstrained disentanglement losses under large pose and expression variations.

Limitations & Future Work¶

Degradation under Extreme Head Poses and Severe Occlusions: When reference images exhibit extreme yaw angles (>60°) or face regions are heavily occluded by sunglasses or masks, the upstream InsightFace encoder fails to produce reliable identity cues, limiting the fiber stream's ability to recover fine facial details.
Higher-Order Truncation Approximations in Complex Edits: The framework adopts an $r=2$ truncation for the BCH expansion to maintain single-JVP tractability. When simultaneously composing four or more heavily conflicting attributes, higher-order commutators are neglected, leaving room for adaptive truncation schemes.
Generalization to Non-Human Subject Customization: While demonstrated on human faces with facial landmark and recognition priors, adapting this Riemannian conditioning framework to open-domain objects and multi-concept personalization remains an exciting future direction.

vs IP-Adapter / PhotoMaker: IP-Adapter decouples cross-attention for image prompts, while PhotoMaker projects stacked image tokens into text embedding space. Both operate within flat Euclidean interfaces, suffering from identity drift and attribute interference when editing coupled attributes. FlowFace integrates fiber bundle disentanglement and Lie-bracket interaction to enforce intrinsic manifold constraints.
vs InstantID: InstantID enforces identity alignment via landmark-dense ControlNet conditioning, which frequently over-constrains facial geometry, impeding natural artistic stylization and expressive edits. FlowFace treats geometric variations as modulations rather than rigid constraints, maintaining high expression accuracy (52.37% vs. 33.67%) alongside superior contextual flexibility.
vs Riemannian Diffusion Models: Conventional Riemannian diffusion approaches require computing Riemannian log/exp maps and Christoffel symbols at every denoising step, making them too slow for high-resolution SDXL synthesis. FlowFace proves that training-time metric surrogate regularization delivers Riemannian benefits while retaining standard DDIM sampling speeds.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Seamlessly integrates fiber bundle disentanglement, Lie-algebraic BCH composition, and training-time Riemannian metric regularization into tuning-free face personalization.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluation across 5,000 identities, 3D facial action unit and expression accuracy metrics, permutation sensitivity tests, and representation probing.
Writing Quality: ⭐⭐⭐⭐⭐ Mathematically rigorous, well-motivated, and structured with clear logic and progressive empirical validation.
Value: ⭐⭐⭐⭐⭐ Provides a principled geometric perspective to resolve attribute interference in diffusion conditioning; code is open-sourced with immediate practical utility.