Skip to content

FaceMoE: Mixture of Experts for Low-Resolution Face Recognition

Conference: ECCV 2026
arXiv: 2606.32040
Code: https://github.com/Kartik-3004/FaceMoE
Area: Human Understanding / Face Recognition
Keywords: Low-Resolution Face Recognition, Mixture of Experts, Transformer, Catastrophic Forgetting, Resolution-Aware Feature Extraction

TL;DR

FaceMoE replaces the single FFN in Transformers with multiple sparsely activated MoE experts and a Top-k router, allowing different experts to automatically specialize in different semantic regions of the face (high-frequency texture, low-frequency smooth, and landmark regions) to achieve resolution-aware feature extraction. It comprehensively outperforms the previous SOTA on three low-resolution benchmarks (BRIAR, IJB-S, and TinyFace) while suffering almost zero degradation in HR pre-training performance.

Background & Motivation

Background: Low-Resolution Face Recognition (LR-FR) is highly demanded in scenarios such as surveillance and security. Existing methods primarily fall into four categories: pre-processing methods based on super-resolution reconstruction, training methods based on knowledge distillation, post-processing methods based on frame selection/feature fusion (e.g., CAFace, CoNAN, ProxyFusion), and architecture-modification methods (e.g., PETALface introducing quality-adaptive LoRA). The mainstream approach remains pre-training on HR data and then fine-tuning on LR data.

Limitations of Prior Work: Three core challenges overlap. First, LR probe images are severely degraded (blur, occlusion, low contrast), and only a few frames contain valid identity information, making feature extraction and aggregation difficult. Second, the gallery images are HR while the probe images are LR, meaning they come from different domains—under HR, face models focus on skin texture and landmark details, whereas under LR, they must rely on coarse-grained contours and shapes. A single feature encoder cannot perform well on both domains simultaneously. Third, catastrophic forgetting occurs during HR\(\rightarrow\)LR fine-tuning: gradients are unstable in the early stages of fine-tuning, causing the model to lose its pre-trained HR discriminative ability without effectively adapting to LR data.

Key Challenge: Transformer encoders with a single FFN suffer from a fundamental bottleneck in representation capacity—they process all face patch tokens with a shared feed-forward network, failing to handle inputs differently based on resolution and semantic regions. This leads to sharing the same feature extraction logic between the HR gallery and LR probe, with the feature quality of both domains dragging each other down.

Goal: (1) To improve the feature extraction quality of LR probes so that residual identity clues in degraded frames can be fully captured; (2) To narrow the domain gap between HR galleries and LR probes; (3) To retain pre-trained knowledge during LR fine-tuning and avoid catastrophic forgetting.

Key Insight: The authors observe that the model focuses on different semantic regions at different resolutions (textures for HR, contours for LR), and only a subset of parameters needs to adapt to the new domain during fine-tuning. This implies that if different sub-networks could fulfill separate roles—some specializing in HR patterns and others adapting to LR degradation—both the domain gap and catastrophic forgetting could be solved simultaneously. The sparse activation property of MoE naturally supports this "selective drift."

Core Idea: Replace the single FFN in the Transformer with MoE (multiple expert FFNs + Top-k routing), enabling different experts to automatically specialize in different semantic regions of the face, with the router dynamically assigning tokens to the most relevant experts based on the input token's resolution and content.

Method

Overall Architecture

FaceMoE uses Swin-B (or ViT-B) as the backbone. The core modification is replacing the single FFN after Self-Attention in each Transformer block with a MoE-MLP layer. The input face image is partitioned into T tokens by Patch Embedding and sequentially passes through multiple Transformer blocks. Inside each block, the MoE-MLP routes each token independently: the Router calculates routing scores for each token to N experts, selects the Top-k experts, each expert performs non-linear transformation on the token, and the outputs are finally weighted and summed. The entire network is ended with a CosFace loss for identity classification. During training, router z-loss (to suppress excessively large logits) and load balancing loss (to prevent unbalanced expert loads) are additionally incorporated to ensure routing stability.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Face Image<br/>LR Probe / HR Gallery"] --> B["Patch Embedding"]
    B --> C["Transformer Block<br/>Self-Attention"]
    C --> D["MoE-MLP Layer<br/>Replaces Single FFN"]
    D --> E["Router Computes Routing Logits"]
    E --> F["Top-k Selection<br/>k=2, N=3"]
    F --> G["Expert 0<br/>High-frequency texture region"]
    F --> H["Expert 1<br/>Low-frequency smooth region"]
    F --> I["Expert 2<br/>Landmark region"]
    G --> J["Weighted Combination Output"]
    H --> J
    I --> J
    J --> K["Feature Embedding"]
    K --> L["CosFace + Auxiliary Losses"]

Key Designs

1. MoE-MLP Layer: Replacing Single FFN with Multiple FFN Experts to Achieve Semantic Region Specialization

In traditional Transformers for face recognition, a shared FFN (two fully connected layers + GELU activation) follows the Self-Attention layer to process all tokens. This design implicitly assumes that all tokens require the same set of non-linear transformations. However, in the LR-FR context, the degree of degradation and information content of the face regions (eyes/cheeks/hair) corresponding to different tokens vary drastically, meaning a single FFN can only learn a "compromised" transformation.

FaceMoE replaces the single FFN with N independent expert MLPs. Each expert \(f_i\) is a standard two-layer fully connected network: \(f_i(x_t) = W_{i,2} \cdot \sigma(W_{i,1} x_t + b_{i,1}) + b_{i,2}\), where \(W_{i,1} \in \mathbb{R}^{d \times h}\), \(W_{i,2} \in \mathbb{R}^{h \times d}\), \(\sigma\) is GELU, and \(h\) is the hidden dimension. All experts coexist in parallel, but only those selected by routing participate in the computation. The experiments set N=3, k=2. Activation maps (Figure 2/6) show that the three experts differentiate naturally: Expert 0 focuses on high-frequency regions (edges, contours, hair texture), Expert 1 focuses on low-frequency smooth regions (cheeks, forehead), and Expert 2 focuses on landmark regions (eyes, nose, mouth). This specialization emerges naturally during training without any explicit supervision—the router automatically learns to assign tokens of different semantic regions to the experts best suited for them through optimizing the CosFace loss.

2. Top-k Router: Token-Level Dynamic Allocation for Resolution-Aware Feature Extraction

The router is the core scheduling module of FaceMoE. For each token \(x_t \in \mathbb{R}^d\), the router calculates the scores of N experts through a learnable linear projection \(W_r \in \mathbb{R}^{d \times N}\): \(z_t = x_t W_r\). It then selects the Top-k experts with the highest scores, applying softmax to their logits to obtain routing weights: \(w_{i_j}(x_t) = \frac{\exp(z_{t, i_j})}{\sum_{j=1}^k \exp(z_{t, i_j})}\). The final MoE output for this token is \(y_t = \sum_{j=1}^k w_{i_j}(x_t) f_{i_j}(x_t)\).

The brilliance of this design lies in its sparse activation + token-level granularity. Setting k=2 << N=3 means each token only uses 2/3 of the experts, which makes the computational overhead much smaller than dense MoEs. More importantly, routing is performed per token rather than per sample—different patches of the same face can be assigned to different combinations of experts. When the input is an LR image, where high-frequency textures are heavily degraded, the router automatically decreases allocation to Expert 0 (high-frequency expert) and increases allocation to Expert 1 (low-frequency expert), naturally achieving "resolution awareness." The paper formalizes this phenomenon using conditional routing probability: \(\mathbb{P}(i_j \mid R_t = r) > \mathbb{P}(i_j \mid R_t \neq r)\), meaning that once the semantic/frequency region \(R_t\) of a given token is known, the probability of routing to a specific expert is significantly higher than to other regions.

3. Auxiliary Loss Design: Router z-loss + Load Balancing Loss to Ensure Training Stability and Expert Balance

Directly training MoE with CosFace loss encounters two typical issues: (1) routing logits tend to increase drastically in the early stages of training, causing softmax to degenerate into one-hot vectors, which leads to gradient vanishing and expert collapse; (2) the router may get "lazy" and assign all tokens to a single expert, rendering the remaining experts obsolete.

Router z-loss penalizes the L2-norm of routing logits: \(\mathcal{L}_z = \lambda_z \cdot \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \|z_{b,t}\|_2^2\). This quadratic penalty encourages the router to generate smooth, low-variance logits, preventing any single expert's score from becoming excessively high and thus ensuring stable gradient backpropagation.

Load balancing loss considers both the "importance" of experts (sum of softmax probabilities) and their "load" (number of selected tokens): \(\mathcal{L}_{\text{balance}} = \lambda_b \cdot N \cdot \frac{1}{(BT)^2} \sum_{i=1}^N \left(\sum_{b,t} p_{b,t,i}\right) \cdot \left(\sum_{b,t} \mathbb{1}[i \in \text{TopK}(z_{b,t})]\right)\). When an expert's importance is high but its load is low (or vice versa), this term increases, driving the router towards a balanced allocation. The final total loss is \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{face}} + \lambda_1 \mathcal{L}_z + \lambda_2 \mathcal{L}_{\text{balance}}\), where \(\lambda_1 = \lambda_2 = 10\) (scaled to match the magnitude of the CosFace loss through hyperparameter search).

Loss & Training

FaceMoE is trained in two stages. Pre-training stage: trained on WebFace4M (approx. 4M images, 205,990 identities) using AdamW (weight decay \(5 \times 10^{-2}\)), Polynomial LR scheduler (1 epoch warmup, initial LR \(10^{-3}\)), and a batch size of 128/GPU for 26 epochs.

The fine-tuning stage split into two steps: linear probing (training the classification head only, LR \(10^{-3}\)) and then full fine-tuning (LR \(10^{-4}\) or \(5 \times 10^{-6}\)). For TinyFace: linear probing for 10 epochs + full fine-tuning for 40 epochs; for BRIAR: 20 epochs for each of the two phases. The auxiliary loss weights are \(\lambda_1 = \lambda_2 = 10\), with the router z-loss coefficient \(\lambda_z = 1\) and the load balancing coefficient \(\lambda_b = 1\). All experiments were conducted on 8 NVIDIA A6000 (48GB) GPUs.

Compared to the standard Swin-B (15.88 GFLOPs), FaceMoE (N=3, k=2) only increases the computational cost to 26.29 GFLOPs (1.66x), while expanding model capacity by 2.17x, illustrating the efficiency advantage of sparse activation.

Key Experimental Results

Main Results

BRIAR Protocol 3.1 Results (Protocol 1): On the BRIAR dataset, FaceMoE substantially outperforms previous SOTA methods across all FAR thresholds. Compared to the runner-up ProxyFusion, the TAR at 1% FAR is improved from 68.90% to 81.27% (an absolute improvement of 12.37%); compared to PETALface, TAR at 0.01% FAR increases from 35.12% to 42.36% (an absolute improvement of 7.24%). Notably, CosFace (ViT-B) with only pre-training and no fine-tuning achieves 34.29% at 0.01% FAR, whereas the fine-tuned CosFace plunges to 11.62%—direct evidence of catastrophic forgetting, which FaceMoE not only avoids but dramatically improves upon.

Method TAR@FAR=0.01% TAR@FAR=0.1% TAR@FAR=1%
CosFace (ViT-B, Pretrain only) 34.29 47.41 62.81
CosFace (Fine-tuned) 11.62 29.68 58.66
CAFace (NeurIPS 2022) 33.41 41.95 51.31
CoNAN (IJCB 2023) 36.52 46.14 56.32
ProxyFusion (NeurIPS 2024) 40.10 53.90 68.90
PETALface (WaCV 2025) 35.12 55.35 75.43
FaceMoE 42.36 61.47 81.27

IJB-S Cross-Domain Generalization: Under the Surveillance-to-Surveillance protocol, FaceMoE achieves a TPIR@FPIR=1% of 14.85% and Rank-1 search of 44.81%, both significantly outperforming the previous best PETALface (12.25% / 38.32%), indicating that the resolution-aware capability of MoE experts is also effective on out-of-distribution surveillance data.

TinyFace (Protocol 2): After fine-tuning, the Rank-1/5/10 accuracy on TinyFace reaches 76.18/79.69/81.75, surpassing PETALface (75.45/79.05/81.19). Crucially, FaceMoE's performance on HR datasets (LFW, CFP-FP, etc.) and mixed-quality datasets (IJB-B, IJB-C) only drops minimally compared to the pre-trained baseline, whereas CosFace/Swin-B undergoes significant degradation on those datasets after fine-tuning—proving that the sparse activation of MoE effectively mitigates catastrophic forgetting.

Ablation Study

Configuration MoE Top-k Router Aux Loss TinyFace Rank-1 TinyFace Rank-5 TinyFace Rank-10
Baseline (Pure Swin-B) x x x 75.10 78.16 80.20
MoE only, no routing v x x 75.40 78.46 80.63
MoE + routing, no auxiliary loss v v x 74.94 77.92 79.90
FaceMoE Full v v v 76.18 79.69 81.75

Three findings: (1) Adding MoE without routing (meaning all tokens share the average of all expert outputs) only increases Rank-1 from 75.10% to 75.40%, indicating that MoE without routing is virtually ineffective; (2) Adding Top-k routing but removing auxiliary losses actually drops the performance to 74.94%, which is lower than the pure Swin-B baseline—since training is unstable without routing regularization, experts collapse easily; (3) When all three components are present, it reaches the optimal 76.18%, proving that each component is indispensable.

Ablation on the number of experts N and k: N=3, k=2 is the optimal configuration (BRIAR Rank-1 73.1%). When N=1, it degenerates into a normal FFN (70.2%); when N=4, performance drops instead due to routing instability and model fragmentation. k=2 achieves the best balance between performance and efficiency.

Key Findings

  • The Top-k router is the most critical component: Removing routing in the ablation study causes performance to plummet back to a level close to pure Swin-B. Moreover, the configuration that removes auxiliary losses (with MoE and routing but no regularization) yields a Rank-1 score even lower than pure Swin-B, showing that unregularized MoE routing is harmful.
  • Expert specialization emerges naturally: Activation maps show that the three experts automatically differentiate into high-frequency, low-frequency, and landmark modes without any explicit semantic supervision, driven solely by the CosFace loss. In fine-tuning, the activation patterns of Expert 2 (low-frequency contours) remain almost unchanged (preserving pre-trained knowledge), while the token allocation to Expert 1 significantly increases (adapting to the LR domain)—representing direct visual proof of "selective drift."
  • Catastrophic forgetting is effectively suppressed: Quantitative evidence includes CKA similarity analysis—after fine-tuning, the CKA of most layers is > 0.99, and only the deepest two layers show significant drift (0.8086 and 0.8677); the L2 displacement of each expert's parameters is only about 8%. In contrast, the HR performance of CosFace drops precipitously after fine-tuning.
  • Robust to resolution changes: Tested across a resolution range of 8x8 to 96x96, the performance on LFW transitions smoothly from 80.75% to 99.73%, with no "cliff-like" sudden collapse at any specific resolution.
  • Insensitive to hyperparameter \(\lambda\): Adjusting the auxiliary loss weights from 1 to 100 (across two orders of magnitude), the TinyFace Rank-1 accuracy fluctuates only between 76.09% and 76.48% (<0.4 percentage points), and the BRIAR [email protected]% varies between 42.27% and 42.56%, showing the robustness of the method to hyperparameters.

Highlights & Insights

  • The "selective drift" mechanism: The reason MoE resists catastrophic forgetting is not its parameter scale, but because sparse activation limits the scope of parameters updated in each step. During fine-tuning, only the activated subset of experts updates weights, while the remaining experts naturally retain their pre-trained states. This "modular isolation" is more elegant than traditional L2 regularization or EWC, requiring no additional forgetting suppression mechanisms.
  • Expert specialization requires no explicit supervision: The automatic division into high-frequency, low-frequency, and landmark specializations among the three experts is entirely driven by the CosFace loss. This demonstrates that as long as the model is provided with sufficient "representational redundancy" (multiple experts) and a "competitive mechanism" (Top-k routing), specialization emerges naturally. This insight can be generalized to other tasks requiring multi-modal or multi-resolution feature extraction.
  • Token-level granularity of Top-k routing is key: If routing were sample-level (where the entire image is assigned to the same set of experts), it would be impossible to realize differentiated processing for different areas on the same face. Token-level granularity ensures that even if input resolution changes (e.g., in LR, the eye region still contains edge information while the cheeks become a smooth block), the router can make different decisions on different patches of the same image, which is the underlying mechanism of "resolution awareness."
  • Dual roles of auxiliary losses: The router z-loss not only stabilizes training but also indirectly encourages expert division of labor—since logits are penalised and kept low, the softmax becomes smoother, prompting the router to "synthesize" multiple experts rather than picking the same one every time. The load balancing loss ensures all experts have the chance to be trained, avoiding the "rich-get-richer" Matthew effect.

Limitations & Future Work

  • Powerless against extreme degradation: The authors acknowledge that at resolutions below 8x8, extreme head poses (>60-degree yaw), severe occlusions (masks/hats/scarves), and atmospheric turbulence, the expert activation maps become chaotic and scattered, causing a sharp drop in model performance. In these scenarios, identity information is inherently lost and cannot be resolved solely by architectural improvements.
  • Training data bias: WebFace4M is dominated by Western, young, and light-skinned populations, without incorporating balanced sampling or de-biasing losses. Although the authors show in the Bias Analysis that FaceMoE exhibits less bias than the Swin-B baseline (measured by SeR/DoB metrics), this is only a relative improvement, not absolute fairness.
  • Challenges in scaling the number of experts: Increasing N beyond 4 leads to routing instability and training collapse (at N=8, Rank-1 drops to 2.31%), indicating that current routing regularization is insufficient to support larger-scale MoEs. Future studies need to explore more robust routing strategies or progressive expert growth methods.
  • Unexplored directions for improvement: (1) Extending MoE to the Q/K/V projection layers of Self-Attention, which could further boost resolution-aware capabilities; (2) Explicitly injecting resolution information into the router (such as input scale coding) to tie routing decisions directly to resolution; (3) Exploring adaptive expert growth—dynamically adding experts during the fine-tuning phase based on data complexity to avoid wasting capacity during pre-training.
  • vs CAFace / CoNAN / ProxyFusion: These methods focus on processing after feature extraction—deciding which frames to fuse and how to weight them. They are limited by the quality of the front-end feature encoder. FaceMoE approaches the problem from the opposite direction by directly improving the feature encoder itself; if features for each token are better, downstream fusion naturally benefits. The two paradigms are complementary and potentially combinable.
  • vs PETALface: PETALface modifies the encoder with quality-adaptive LoRA modules, essentially adding a "conditional branch" to the encoder. FaceMoE uses multiple FFN experts + routing, essentially adding a "competitive division of labor" to the encoder. Although both share the same goal (improving feature extraction), the sparse activation mechanism of FaceMoE naturally provides resistance against catastrophic forgetting (updating all LoRA parameters can still lead to forgetting).
  • vs MoE-FFD / AMEL (MoE in Face Analysis): These works apply MoE to face forgery detection and anti-spoofing, but their routing is sample-level, and experts are parameter-efficient (e.g., LoRA/Adapter). In contrast, FaceMoE features token-level routing and full FFN experts. Token-level granularity allows differentiated processing of different regions on the same face, which is vital for resolution-aware modeling in LR-FR, and full FFN experts provide stronger representational capacity than LoRAs.

Rating

  • Novelty: ⭐⭐⭐⭐ First to introduce MoE to low-resolution face recognition, using token-level routing to achieve resolution awareness. Although MoE itself is not a new technology, its adaptation and validation in this specific LR-FR context represent a substantial contribution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ 11 datasets (full coverage of HR/mixed/LR), two protocols, extensive ablation studies (components, number of experts, backbone, data scale, resolution, degradation robustness, routing stability, bias analysis, CKA selective drift, and inference efficiency). The experimental design is comprehensive and highly convincing.
  • Writing Quality: ⭐⭐⭐⭐ The formulation of the three challenges is clear and compelling. The method is thoroughly described with complete mathematical formulations, and the activation maps are intuitively visualized. However, there is slight repetition in some sections (the analysis of data scale in the main body is almost identical to the appendix).
  • Value: ⭐⭐⭐⭐ Achieves significant improvements on LR-FR tasks (12%+ increase in BRIAR at 1% FAR), while providing a reusable technical path: using sparse MoD/MoE as an "anti-forgetting adapter" is highly generalizable to other vision tasks requiring domain adaptation.