Skip to content

Omni-ID: Holistic Identity Representation Designed for Generative Tasks

Conference: CVPR 2025
arXiv: 2412.09694
Code: https://snap-research.github.io/Omni-ID/
Area: Human Understanding
Keywords: Face Representation, Identity-Preserving Generation, Multi-Decoder Training, Few-to-Many Reconstruction, Face Personalization

TL;DR

Omni-ID proposes a holistic facial identity representation designed specifically for generative tasks. Through a few-to-many identity reconstruction training paradigm and multi-decoder objectives (Masked Transformer + Flow Matching), it encodes an arbitrary number of input images into a fixed-size structured representation, significantly outperforming ArcFace and CLIP in controllable face generation and personalized T2I tasks.

Background & Motivation

Background: Mainstream approaches in face generation rely on feature representations trained on discriminative tasks (e.g., ArcFace for face recognition, CLIP for image-text alignment). These representations are widely used in personalized text-to-image generation (e.g., IP-Adapter) and controllable face synthesis.

Limitations of Prior Work: Discriminative/contrastive representations suffer from two fundamental issues. First, they are single-image encodings—a front-facing photograph with neutral expression has almost no information about appearance from profile views, during smiles, or frowns. Second, according to the information bottleneck principle, subtle variations that are irrelevant to classification but vital for generation (such as nose shape, beard details) are discarded during discriminative training.

Key Challenge: There is a fundamental conflict between discriminative training objectives and the requirements of generative tasks—the former seeks intra-class compactness and inter-class separability, while the latter requires preserving rich individual details. With multi-image inputs, simple feature averaging or concatenation cannot effectively integrate complementary information from different viewpoints.

Goal: To design a facial identity representation directly optimized for generative tasks, which can extract a fixed-size structured encoding from an arbitrary number of input images, and whose representation quality scales with the number of input images.

Key Insight: If the encoder is trained to reconstruct more unseen poses and expression images of the same identity from a few images (few-to-many), the encoder is forced to learn identity features that generalize to unseen poses. Meanwhile, utilizing multiple decoders can leverage the complementary advantages of different decoding frameworks.

Core Idea: Train the face encoder using generative objectives (instead of discriminative ones) through a training strategy of few-to-many reconstruction + multi-decoders (Masked Transformer + Flow Matching) to learn a holistic identity representation.

Method

Overall Architecture

The training consists of two stages. The first stage trains the Masked Transformer Decoder (MTD), where the encoder extracts identity representations from a few input images, and the MTD uses this representation along with heavily masked (95%) target images to reconstruct multiple different images of the same identity. The second stage trains the Flow Matching Decoder, performing denoising reconstruction based on the FLUX model by injecting identity representations via IP-Adapter. Both stages share the same encoder, allowing flexible integration into various downstream generators during inference.

Key Designs

  1. Omni-ID Encoder (Transformer Encoder):

    • Function: Encodes an arbitrary number of input images into a fixed-size structured identity representation (256 tokens x 1280 dims).
    • Mechanism: Uses CLIP-H as the image feature extractor, concatenates the patch tokens of each image as Key and Value, aggregates information through cross-attention with learnable query tokens, and refines it via self-attention layers. Because the queries are fixed learnable parameters, the output dimension remains consistent.
    • Design Motivation: Fixed-size structured coding allows downstream tasks to rely on specific positions in the encoding to correspond to specific semantic attributes. Attention visualization confirms that different queries indeed attend to different facial semantic regions (eyes, mouth, outlines, etc.).

  2. Few-to-Many Identity Reconstruction Training Paradigm:

    • Function: Forces the encoder to learn identity features that generalize to unseen poses and expressions.
    • Mechanism: During training, a small number of inputs and a larger number of reconstruction targets are sampled from the complete image set of the same identity. The model must reconstruct multiple target images of unseen poses from few inputs. Ablation studies show that a 3-input-to-8-target configuration is optimal, outperforming 8-to-8 (which can degenerate into autoencoding).
    • Design Motivation: If only 1-to-1 reconstruction is performed, the encoder might overfit to specific attributes of the input image; few-to-many forces the encoder to extract core identity information that is invariant across poses.

  3. Multi-Decoder Objectives (MTD + Flow Matching):

    • Function: Combines the complementary advantages of two decoders to train the encoder.
    • Mechanism: MTD uses a high mask ratio of 95% to ensure that identity information comes entirely from the encoder representation, which is beneficial for learning widely covered representations but outputs blurry results. The Flow Matching decoder encourages the encoder to capture details of different granularities through denoising tasks at distinct noise levels.
    • Design Motivation: MTD excels at representation learning but has limited output quality; Flow Matching can recover fine-grained details but is not ideal for representation learning when used alone. The two complement each other.

Loss & Training

The MTD stage is trained for 200K steps with a 95% mask ratio and L1 reconstruction loss. The Flow Matching stage is trained for 10K steps, using FLUX dev as the base model and a flow matching loss. Both stages are trained on the self-built MFHQ dataset (134K identities, 8 images of \(448+\) resolution per identity).

Key Experimental Results

Main Results

Controllable face generation (IP-Adapter + ControlNet on FLUX):

Method MFHQ Test ID Sim (1/3/5/7 inputs) Webface Test ID Sim (3/5/8/16 inputs)
ArcFace 0.515/0.523/0.529/0.535 0.379/0.373/0.370/0.371
CLIP 0.648/0.670/0.680/0.682 0.695/0.696/0.696/0.695
ArcFace+CLIP 0.638/0.655/0.663/0.664 0.652/0.654/0.656/0.658
Omni-ID 0.708/0.728/0.737/0.742 0.774/0.779/0.781/0.784

Ablation Study

Configuration ID Similarity (1/3 img) Description
Full model 0.708 / 0.728 Full model
w/o MTD pretraining 0.468 / 0.473 Dropped by 34%, the most critical component
w/o Flow-Matching 0.672 / 0.685 Dropped by 5%, detail capture impaired
w/o Few-to-many 0.616 / 0.633 Dropped by 13%, generalization capability decreased
w/o MFHQ dataset 0.678 / 0.693 Dropped by 4%, affected by data quality

Key Findings

  • MTD pre-training is the most critical component (removing it drops ID similarity from 0.708 to 0.468), showing that the mask-and-reconstruct paradigm is vital for representation learning.
  • Omni-ID's performance steadily scales with the number of input images (while ArcFace barely increases), demonstrating that the encoder indeed integrates complementary information from multiple images.
  • Attention visualization shows that different queries attend to distinct facial semantic areas and can adaptively handle occlusions.
  • The MFHQ dataset is more effective than WebFace because the latter introduces noise due to excessive intra-class ID variation.

Highlights & Insights

  • Generative training objectives replacing discriminative training: This is a paradigm shift in the field. The information bottleneck principle provides theoretical backing—discriminative training discards details that are unimportant for classification but necessary for generation.
  • Interpretability of structured representations: Fixed queries automatically learn semantic division of labor without explicit supervision. This emergent structure indicates that cross-attention + learnable queries constitute a powerful information organization mechanism.
  • Extremely high masking rate design for MTD (95%): Far exceeding MAE's 75%, it ensures that identity information comes entirely from the encoder rather than the target image, effectively preventing identity leakage.

Limitations & Future Work

  • Omni-ID does not encode attributes outside of the face (such as hairstyle), causing these attributes to be "hallucinated" during downstream generation.
  • Only verified on FLUX for two downstream tasks, without direct comparison to newer methods like PhotoMaker or InstantID.
  • The MFHQ dataset is derived from videos, meaning variations in lighting and background are limited.
  • vs ArcFace: Overly invariant to age and skin tone; single-image encoding cannot represent multi-view information. Omni-ID outperforms it by 37%+ in ID similarity.
  • vs CLIP: Retains more visual features but lacks face fine-tuning; multi-image inputs barely provide any improvement.
  • vs IP-Adapter/FaceIDPlus: Focuses on injecting facial features into generative models, whereas Omni-ID focuses on obtaining better facial representations themselves. The two are orthogonal.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Generative facial representation learning is a paradigm innovation; few-to-many + multi-decoder is novel.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Ablations are detailed, but direct comparisons with more SOTA personalization methods are missing.
  • Writing Quality: ⭐⭐⭐⭐ Motivation is clearly explained, though some details require referring to the appendix.
  • Value: ⭐⭐⭐⭐⭐ A foundational representation that can be widely applied to various downstream tasks.