Skip to content

FaceMoE: Mixture of Experts for Low-Resolution Face Recognition

Conference: ECCV 2026
arXiv: 2606.32040
Code: https://github.com/Kartik-3004/FaceMoE
Area: Human Understanding / Face Recognition
Keywords: Low-Resolution Face Recognition, Mixture of Experts, Transformer, Catastrophic Forgetting, Resolution-Aware Feature Extraction

TL;DR

FaceMoE replaces the single FFN in Transformers with multiple sparsely activated MoE experts and a Top-k router, allowing different experts to automatically specialize in distinct semantic regions of the face (high-frequency texture, low-frequency smooth, landmarks). This achieves resolution-aware feature extraction. It comprehensively outperforms the state-of-the-art (SOTA) on three low-resolution benchmarks (BRIAR, IJB-S, TinyFace) while suffering almost no degradation in high-resolution (HR) pre-trained performance.

Background & Motivation

Background: Low-resolution face recognition (LR-FR) is urgently needed in scenarios such as surveillance and security. Existing approaches mainly fall into four categories: pre-processing methods based on super-resolution reconstruction, training methods based on knowledge distillation, post-processing methods based on frame selection or feature fusion (e.g., CAFace, CoNAN, ProxyFusion), and architecture-modification methods (e.g., PETALface introducing quality-adaptive LoRA). The mainstream paradigm remains pre-training on HR data followed by fine-tuning on LR data.

Limitations of Prior Work: Three core challenges co-exist. First, LR probe images suffer severe degradation (blur, occlusion, low contrast), and only a few frames contain valid identity information, making feature extraction and aggregation difficult. Second, the gallery images are HR while the probe images are LR, meaning they originate from different domains. In the HR domain, face models focus on skin texture and landmark details, whereas in the LR domain, they must rely on coarse-grained contours and shapes. A single feature encoder cannot perform optimally on both domains simultaneously. Third, catastrophic forgetting occurs during HR\(\rightarrow\)LR fine-tuning: gradients are unstable in the early stages of fine-tuning, causing the model to lose its pre-trained HR discriminative ability while failing to effectively adapt to LR data.

Key Challenge: The single-FFN Transformer encoder has a fundamental bottleneck in representation capacity. It processes all face patch tokens with a shared feed-forward network, failing to handle them differentially based on the input resolution and semantic regions. This leads to "HR gallery and LR probe sharing the exact same feature extraction logic," causing the feature quality of both domains to compromise each other.

Goal: (1) Improve the feature extraction quality of LR probes to fully capture residual identity clues in degraded frames; (2) Narrow the domain gap between HR galleries and LR probes; (3) Retain pre-trained knowledge during LR fine-tuning to prevent catastrophic forgetting.

Key Insight: The authors observe that models focus on different semantic regions depending on the resolution (HR focuses on texture, LR focuses on contours), and only a subset of parameters needs to adapt to the new domain during fine-tuning. This implies that if different sub-networks can perform distinct roles—some specializing in HR patterns and others adapting to LR degradation—both the domain gap and catastrophic forgetting can be solved simultaneously. The sparse activation property of MoE naturally supports this "selective drift."

Core Idea: Replace the single FFN in the Transformer with MoE (multi-expert FFN + Top-k router) so that different experts dynamically specialize in different semantic regions of the face. The router then dynamically allocates tokens to the most relevant experts based on the input token's resolution and content.

Method

Overall Architecture

FaceMoE utilizes Swin-B (or ViT-B) as the backbone. The core modification is replacing the single FFN after Self-Attention in each Transformer block with an MoE-MLP layer. The input face image is split into \(T\) tokens via Patch Embedding and sequentially passes through multiple Transformer blocks. Within each block, the MoE-MLP performs routing independently for each token: the Router computes scores for each token across \(N\) experts, selects the Top-k experts, each expert performs a non-linear transformation on the token, and the outputs are finally weighted and summed. The entire network is appended with a CosFace loss for identity classification. During training, a router z-loss (to suppress excessively large logits) and a load balancing loss (to prevent unbalanced expert loads) are introduced to ensure stable routing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Face Image<br/>LR Probe / HR Gallery"] --> B["Patch Embedding"]
    B --> C["Transformer Block<br/>Self-Attention"]
    C --> D["MoE-MLP Layer<br/>Replacing Single FFN"]
    D --> E["Router Computes Routing Logits"]
    E --> F["Top-k Selection<br/>k=2, N=3"]
    F --> G["Expert 0<br/>High-frequency Texture"]
    F --> H["Expert 1<br/>Low-frequency Smooth Area"]
    F --> I["Expert 2<br/>Landmark Area"]
    G --> J["Weighted Combination Output"]
    H --> J
    I --> J
    J --> K["Feature Embedding"]
    K --> L["CosFace + Auxiliary Loss"]

Key Designs

1. MoE-MLP Layer: Replacing a single FFN with multiple FFN experts to achieve semantic region specialization

In traditional Transformers for face recognition, Self-Attention is followed by a shared FFN (two fully connected layers + GELU activation) to process all tokens. This design implicitly assumes that all tokens require the same set of non-linear transformations. However, in LR-FR scenarios, different tokens corresponding to different facial regions (eyes/cheeks/hair) show highly varying degrees of degradation and information content. A single FFN can only learn a "compromised" transformation.

FaceMoE replaces the single FFN with \(N\) independent expert MLPs. Each expert \(f_i\) is a standard two-layer fully connected network: \(f_i(x_t) = W_{i,2} \cdot \sigma(W_{i,1} x_t + b_{i,1}) + b_{i,2}\), where \(W_{i,1} \in \mathbb{R}^{d \times h}\), \(W_{i,2} \in \mathbb{R}^{h \times d}\), \(\sigma\) is GELU, and \(h\) is the hidden dimension. All experts exist in parallel, but only those selected by routing participate in the computation. The experiments set \(N=3\) and \(k=2\). The activation maps (Figure 2/6) demonstrate that three experts naturally specialize: Expert 0 focuses on high-frequency regions (edges, contours, hair textures), Expert 1 focuses on low-frequency smooth regions (cheeks, forehead), and Expert 2 focuses on landmark regions (eyes, nose, mouth). This specialization emerges naturally during training without any explicit supervision—the router automatically learns to assign tokens of different semantic regions to the experts best suited for them through the optimization of the CosFace loss.

2. Top-k Router: Token-level dynamic allocation for resolution-aware feature extraction

The router is the core scheduling module of FaceMoE. For each token \(x_t \in \mathbb{R}^d\), the router computes scores for \(N\) experts using a learnable linear projection \(W_r \in \mathbb{R}^{d \times N}\): \(z_t = x_t W_r\). It then selects the Top-k highest-scoring experts and applies softmax to their logits to obtain routing weights: \(w_{i_j}(x_t) = \frac{\exp(z_{t, i_j})}{\sum_{j=1}^k \exp(z_{t, i_j})}\). The final MoE output for the token is \(y_t = \sum_{j=1}^k w_{i_j}(x_t) f_{i_j}(x_t)\).

The elegance of this design lies in sparse activation + token-level granularity. Setting \(k=2 \ll N=3\) means each token only activates 2/3 of the experts, which keeps the computational overhead significantly lower than that of dense MoE. More importantly, routing is performed token-wise rather than sample-wise—different patches of the same face can be assigned to different expert combinations. When the input is an LR image where high-frequency textures are severely degraded, the router minimizes allocation to Expert 0 (high-frequency expert) and increases allocation to Expert 1 (low-frequency expert), naturally achieving "resolution awareness." The paper formalizes this phenomenon using conditional routing probability: \(\mathbb{P}(i_j \mid R_t = r) > \mathbb{P}(i_j \mid R_t \neq r)\), meaning that given the semantic/frequency region \(R_t\) to which the token belongs, the probability of routing to a specific expert is significantly higher than that of routing to other regions.

3. Auxiliary Loss Design: router z-loss + load balancing loss to ensure training stability and expert balance

Directly training MoE with CosFace loss faces two typical issues: (1) routing logits can easily increase dramatically in the early stages of training, causing softmax to collapse into a one-hot distribution, which leads to vanishing gradients and expert collapse; (2) the router might become "lazy"—allocating all tokens to a single expert and idle-listing the others.

Router z-loss penalizes the L2 norm of the routing logits:

\[\mathcal{L}_z = \lambda_z \cdot \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \|z_{b,t}\|_2^2\]

This quadratic penalty term encourages the router to produce smooth, low-variance logits, preventing any single expert's score from becoming excessively high, thereby guaranteeing stable gradient backpropagation.

The load balancing loss simultaneously considers the "importance" (sum of softmax probabilities) and "load" (number of selected tokens) of each expert:

\[\mathcal{L}_{\text{balance}} = \lambda_b \cdot N \cdot \frac{1}{(BT)^2} \sum_{i=1}^N \left(\sum_{b,t} p_{b,t,i}\right) \cdot \left(\sum_{b,t} \mathbb{1}[i \in \text{TopK}(z_{b,t})]\right)\]

This term increases when an expert has high importance but low load (or vice versa), pushing the router toward balanced allocation. The final total loss is

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{face}} + \lambda_1 \mathcal{L}_z + \lambda_2 \mathcal{L}_{\text{balance}}\]

where \(\lambda_1 = \lambda_2 = 10\) (hyperparameter search scaled the auxiliary losses to the same order of magnitude as the CosFace loss).

Loss & Training

FaceMoE is trained in two stages. Pre-training stage: Trained on WebFace4M (approx. 4M images, 205,990 identities) using AdamW (weight decay \(5 \times 10^{-2}\)), Polynomial LR scheduler (1 epoch warmup, initial LR \(10^{-3}\)), and a batch size of 128/GPU for 26 epochs.

The fine-tuning stage consists of two steps—linear probing (training only the classification head with LR \(10^{-3}\)) followed by full fine-tuning (LR \(10^{-4}\) or \(5 \times 10^{-6}\)). For TinyFace: 10 epochs of linear probing + 40 epochs of full fine-tuning. For BRIAR: 20 epochs for each of the two stages. Auxiliary loss weights are set to \(\lambda_1 = \lambda_2 = 10\), with router z-loss coefficient \(\lambda_z = 1\) and load balancing coefficient \(\lambda_b = 1\). All experiments are completed on 8 NVIDIA A6000 (48GB) GPUs.

Compared to the standard Swin-B (15.88 GFLOPs), FaceMoE (N=3, k=2) increases computational cost only to 26.29 GFLOPs (1.66x), while its model capacity expands by 2.17x, demonstrating the efficiency advantage of sparse activation.

Key Experimental Results

Main Results

BRIAR Protocol 3.1 Results (Protocol 1): On the BRIAR dataset, FaceMoE substantially outperforms previous SOTA methods under all FAR thresholds. Compared to the runner-up ProxyFusion, TAR@FAR=1% increases from 68.90% to 81.27% (an absolute gain of 12.37%); compared to PETALface, TAR@FAR=0.01% improves from 35.12% to 42.36% (an absolute gain of 7.24%). Notably, CosFace (ViT-B) with only pre-training and no fine-tuning achieves 34.29% at 0.01% FAR, whereas the fine-tuned CosFace plunges to 11.62%—direct evidence of catastrophic forgetting, while FaceMoE avoids this drop and achieves significant improvements instead.

Method TAR@FAR=0.01% TAR@FAR=0.1% TAR@FAR=1%
CosFace (ViT-B, Pre-trained Only) 34.29 47.41 62.81
CosFace (Fine-tuned) 11.62 29.68 58.66
CAFace (NeurIPS 2022) 33.41 41.95 51.31
CoNAN (IJCB 2023) 36.52 46.14 56.32
ProxyFusion (NeurIPS 2024) 40.10 53.90 68.90
PETALface (WaCV 2025) 35.12 55.35 75.43
FaceMoE 42.36 61.47 81.27

IJB-S Cross-Domain Generalization: Under the Surveillance-to-Surveillance protocol, FaceMoE achieves 14.85% TPIR@FPIR=1% and 44.81% Rank-1 retrieval, significantly outperforming the previous best PETALface (12.25% / 38.32%), illustrating that the resolution-aware capability of MoE experts remains effective on out-of-distribution surveillance data.

TinyFace (Protocol 2): After fine-tuning, the Rank-1/5/10 on TinyFace reaches 76.18/79.69/81.75, surpassing PETALface (75.45/79.05/81.19). Critically, FaceMoE's performance on HR datasets (LFW, CFP-FP, etc.) and mixed-quality datasets (IJB-B, IJB-C) drops only marginally compared to the pre-trained baseline, whereas fine-tuned CosFace/Swin-B suffers severe performance drops on these datasets—proving that the sparse activation of MoE effectively mitigates catastrophic forgetting.

Ablation Study

Configuration MoE Top-k Router Aux Loss TinyFace Rank-1 TinyFace Rank-5 TinyFace Rank-10
Baseline (Swin-B Only) x x x 75.10 78.16 80.20
MoE Only, No Routing v x x 75.40 78.46 80.63
MoE + Routing, No Aux Loss v v x 74.94 77.92 79.90
Full FaceMoE v v v 76.18 79.69 81.75

Three findings: (1) Adding MoE without routing (meaning all tokens share the average of all expert outputs) only improves Rank-1 from 75.10 to 75.40, showing that MoE without routing is almost ineffective; (2) Adding Top-k routing but omitting the auxiliary loss drops performance further to 74.94, which is below the Swin-B baseline—since training is unstable without routing regularization, causing expert collapse; (3) The complete system containing all three modules reaches the best performance at 76.18%, proving each component is indispensable.

Ablation on number of experts N and k: \(N=3, k=2\) is the optimal configuration (yielding 73.1% Rank-1 on BRIAR). \(N=1\) degenerates to a standard FFN (70.2%); \(N=4\) degrades performance due to unstable routing and model fragmentation. \(k=2\) achieves the best balance between effectiveness and efficiency.

Key Findings

  • Top-k Router is the most critical component: In the ablation study, removing routing causes a performance drop nearly back to the standard Swin-B level. Stripping away the auxiliary loss (having MoE and routing but no regularization) yields a Rank-1 score even lower than standard Swin-B, indicating that unregularized MoE routing is harmful.
  • Expert specialization is naturally emergent: Activation maps show that the three experts automatically divide into high-frequency, low-frequency, and landmark patterns purely driven by the CosFace loss, without any explicit semantic supervision. After fine-tuning, the activation pattern of Expert 2 (low-frequency contours) remains almost unchanged (retaining pre-trained knowledge), while token allocation to Expert 1 increases significantly (adapting to the LR domain)—representing direct visual evidence of "selective drift."
  • Catastrophic forgetting is effectively suppressed: Quantitative evidence includes CKA similarity analysis—after fine-tuning, the CKA similarity of most layers is \(>0.99\), with significant drift occurring only in the deepest two layers (0.8086 and 0.8677). The L2 displacement of parameters for each expert is only about 8%. In contrast, fine-tuned CosFace experiences a catastrophic drop in HR performance.
  • Robustness to resolution variations: Tested across resolutions from 8x8 to 96x96, performance on LFW smoothly transitions from 80.75% to 99.73%, avoiding any sudden "cliff-like" drop at any particular resolution.
  • Insensitivity to hyperparameter \(\lambda\): Tuning the auxiliary loss weights across two orders of magnitude (from 1 to 100) results in TinyFace Rank-1 fluctuations only between 76.09% and 76.48% (\(<0.4\) percentage points), and BRIAR [email protected]% fluctuations between 42.27% and 42.56%, showcasing the robustness of the method to hyperparameters.

Highlights & Insights

  • "Selective Drift" Mechanism: The reason MoE resists catastrophic forgetting is not due to a larger parameter count, but because sparse activation restricts the parameter updating range during each iteration. During fine-tuning, only the activated subset of experts updates weights, while others naturally preserve their pre-trained states. This "modular isolation" is more elegant than traditional methods like L2 regularization or EWC, requiring no additional design of forgetting-suppression mechanisms.
  • Emergent Expert Specialization Without Explicit Supervision: The automatic division into high-frequency, low-frequency, and landmark specializations is completely driven by the CosFace loss. This demonstrates that as long as the model is given sufficient "representational redundancy" (multiple experts) and a "competitive mechanism" (Top-k routing), specialization emerges naturally. This insight can be extended to other tasks requiring multi-modal/multi-resolution feature extraction.
  • Token-level Granularity of Top-k Routing is Key: If routing were performed at the sample level (assigning the entire image to the same set of experts), differential processing across various regions of the same face would be impossible. Token-level granularity ensures that even as input resolutions change (e.g., in LR images, the eye region still contains edge information but cheeks become smooth blocks), the router can make distinct decisions for different patches of the same image, which serves as the core mechanism for "resolution awareness."
  • Dual Role of Auxiliary Losses: Router z-loss not only stabilizes training but also indirectly encourages expert specialization. Compressing logits produces a smoother softmax, prompting the router to "jointly consider" multiple experts instead of always selecting the same one. The load balancing loss ensures all experts have opportunities to be trained, avoiding the "rich-get-richer" Matthew effect.

Limitations & Future Work

  • Inability to Handle Extreme Degradation: The authors acknowledge that at resolutions below 8x8, extreme head poses (\(>60^\circ\) yaw), severe occlusions (masks/hats/scarves), and atmospheric turbulence, the expert activation maps become cluttered and scattered, causing a sharp decline in performance. In these scenarios, identity information is intrinsically lost, which is beyond what architectural improvements can solve.
  • Training Data Bias: WebFace4M primarily consists of Western, young, and light-skinned populations, without balanced sampling or de-biasing losses. Although the authors' Bias Analysis shows that FaceMoE's bias is smaller than the Swin-B baseline (using SeR/DoB metrics), this is only a relative improvement rather than absolute fairness.
  • Challenges in Scaling up the Number of Experts: Increasing \(N\) beyond 4 leads to unstable routing and training collapse (at \(N=8\), Rank-1 drops to 2.31%), indicating that current routing regularization mechanisms are insufficient to support larger-scale MoE. Future work should investigate more robust routing strategies or progressive expert growth methods.
  • Unexplored Directions for Improvement: (1) Extending MoE to the Q/K/V projection layers in Self-Attention may further improve resolution-aware capabilities; (2) Explicitly injecting resolution information into the router (such as input scale embeddings) to link routing decisions directly with resolution; (3) Exploring adaptive expert growth—dynamically adding experts during fine-tuning based on data complexity to avoid wasting capacity during the pre-training phase.
  • vs CAFace / CoNAN / ProxyFusion: These methods work "after feature extraction"—focusing on which frames to select and how to weight them. They are limited by the quality of the frontend feature encoder. FaceMoE takes the opposite path, directly improving the feature encoder itself. If the features of each token are improved, the backend fusion will naturally benefit. The two directions are complementary and can theoretically be combined.
  • vs PETALface: PETALface modifies the encoder with quality-adaptive LoRA modules, which essentially adds "conditional branches" to the encoder. FaceMoE uses multiple FFN experts + routing, which adds a "competitive division of labor" to the encoder. Although both target improved feature extraction, FaceMoE's sparse activation mechanism inherently provides resistance to catastrophic forgetting (LoRA updates all parameters, which can still lead to forgetting).
  • vs MoE-FFD / AMEL (MoE in Face Analysis): These works apply MoE to face forgery detection and anti-spoofing, but routing is sample-level and experts are parameter-efficient (LoRA/Adapter). FaceMoE distinguishes itself by employing token-level routing and full-scale FFN experts. Token-level granularity allows for different handling of "different regions of the same face," which is vital for resolution awareness in LR-FR; full-scale experts also offer stronger representational capacity than LoRA.

Rating

  • Novelty: ⭐⭐⭐⭐ This is the first work to introduce MoE to low-resolution face recognition, using token-level routing for resolution awareness. The idea is straightforward but has not been explored before. Although MoE is not a new technology, its adaptation and validation in the LR-FR context represent a substantial contribution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 11 datasets (complete coverage of HR/mixed/LR), two protocols, extensive ablation studies (components, number of experts, backbones, data scale, resolution, degradation robustness, routing stability, bias analysis, CKA selective drift, and inference efficiency). The experimental design is comprehensive and highly convincing.
  • Writing Quality: ⭐⭐⭐⭐ The formulation of the three challenges is clear and compelling, the method description includes complete mathematical equations, and the activation map visualization is intuitive. However, some paragraphs are slightly repetitive (the analysis of data scale in the main text is almost duplicated in the appendix).
  • Value: ⭐⭐⭐⭐ Achieves significant improvements in the LR-FR task (12%+ gain on BRIAR 1% FAR) while providing a reusable technical paradigm: the concept of sparse MoE as an "anti-forgetting adapter" can be extended to other vision tasks requiring domain adaptation.