ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

Conference: ECCV 2026
arXiv: 2605.01971
Code: None
Area: AI Safety / Fairness / Self-Supervised Learning
Keywords: Self-Supervised Learning, Contrastive Learning, Fair Representations, Pseudo-Counterfactual Pairs, Clustering Prototypes

TL;DR

ProtoFair proposes a plug-and-play fairness regularization term. Without modifying existing self-supervised contrastive learning objectives, it identifies "same semantic content but different sensitive groups" pseudo-counterfactual pairs through momentum-updated unsupervised clustering prototypes, pulling these cross-group samples closer in the embedding space to force the encoder to learn representations invariant to sensitive attributes. Combined with SimCLR / SupCon / BarlowTwins / BYOL on CelebA, UTKFace, and NIH Chest X-rays, it significantly reduces Equalized Odds while maintaining competitive accuracy.

Background & Motivation

Self-supervised contrastive learning (such as SimCLR, SupCon, BYOL) has become a primary paradigm for learning high-quality visual representations without relying on annotations, matching or even outperforming supervised methods on multiple downstream tasks. However, recent studies reveal a concerning fact: even if the training process does not access explicit labels, these self-supervised representations still encode demographic biases present in the training data—for instance, in facial attribute classification, models systematically utilize sensitive information such as gender and race to make predictions, leading to significant disparities in error rates across different demographic groups.

To address this issue, the two existing mainstream approaches both suffer from fundamental limitations. Adversarial de-biasing methods (such as GRL, LNL) train auxiliary discriminators to remove sensitive information from representations, but the introduced minimax optimization is unstable, sensitive to hyperparameters, and inherently modifies the training process itself rather than supplementing the original objective. Methods modifying the contrastive objective (such as FSCL) redesign positive and negative sample sampling strategies in the loss function to reduce bias, but these methods are tightly coupled with specific SSL frameworks—every time a new SOTA self-supervised method emerges, the fairness scheme must be redesigned from scratch. The common pain point of both approaches is: they require replacing or fundamentally altering the base training target, which has extremely poor portability in the currently rapidly iterating SSL landscape.

This paper raises a different question: Is it possible to introduce a lightweight auxiliary regularization term instead of redesigning the SSL objective, allowing existing SSL methods to naturally become fairer? The core idea stems from the intuition of counterfactual fairness—if an individual's sensitive attribute changes but their representation remains unchanged, the representation is fair. ProtoFair uses unsupervised clustering as a proxy for semantic content to construct pseudo-counterfactual pairs (samples in the same cluster but from different sensitive groups), pulling them closer in the embedding space to actively encourage the encoder to learn features invariant to sensitive attributes.

Method

Overall Architecture

The core mechanism of ProtoFair is to frame fairness regularization as a plug-and-play auxiliary loss \(\mathcal{L}_{\text{CF}}\), added to any existing SSL loss \(\mathcal{L}_{\text{SSL}}\), rendering the total loss as \(\mathcal{L} = \mathcal{L}_{\text{SSL}} + \lambda \mathcal{L}_{\text{CF}}\), where \(\lambda\) controls the strength of the fairness regularization. The base SSL objective remains completely unchanged, and ProtoFair influences the encoder solely through the gradient of \(\mathcal{L}_{\text{CF}}\).

The overall pipeline is split into three parallel branches: after receiving input samples, the shared encoder \(f_\theta\) maps them through the contrastive projection head \(g_\phi\) (generating L2-normalized features \(z_i\), shared by both the base SSL loss and the ProtoFair loss) and the clustering projection head \(h_\psi\) (generating clustering space features \(\bar{h}_i\), used exclusively for prototype assignment). The clustering side maintains \(K\) momentum-updated prototype vectors \(\{c_k\}_{k=1}^K\), which assign hard clustering labels \(\hat{k}_i\) to each sample through K-Means initialization and EMA tracking. These clustering labels and the sensitive attributes \(s_i\) jointly define the pseudo-counterfactual positive sample sets \(\mathcal{P}_i = \{j \mid \hat{k}_j = \hat{k}_i \text{ 且 } s_j \neq s_i\}\). The positive pairs are retrieved within the current batch (\(\mathcal{L}_{\text{within}}\)) and across a FIFO queue of previous batches (\(\mathcal{L}_{\text{cross}}\)), which sum up to form \(\mathcal{L}_{\text{CF}}\), which is then weighted and added to \(\mathcal{L}_{\text{SSL}}\). The clustering assignment is detached from the computational graph when passed into the ProtoFair loss, establishing an EM-style alternating optimization: the E-step fixes the clustering pseudo-labels, and the M-step optimizes the encoder and the contrastive head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input batch x<br/>with sensitive attributes s"] --> B["Shared Encoder f_θ"]
    B --> C["Contrastive Projection Head g_φ<br/>→ Normalized feature z"]
    B --> D["Clustering Projection Head h_ψ<br/>→ Clustering feature h̄"]
    D --> E["Momentum-Updated Clustering Prototypes<br/>K-Means Initialization + EMA Tracking"]
    E --> F["Hard Cluster Assignment k̂ = argmax(h̄ᵀc)"]
    F --> G["Pseudo-Counterfactual Pair Construction<br/>Same cluster + Different sensitive group = Positive sample"]
    C --> G
    A --> G
    G --> H["In-Batch Contrastive Loss L_within"]
    G --> I["Cross-Batch Queue<br/>FIFO stores history (z, k̂, s)"]
    I --> J["Cross-Batch Contrastive Loss L_cross"]
    H --> K["L_CF = L_within + L_cross"]
    J --> K
    C --> L["Base SSL Loss L_SSL<br/>(SimCLR/SupCon/BYOL...)"]
    K --> M["Total Loss L = L_SSL + λ·L_CF"]
    L --> M

Key Designs

1. Momentum-Updated Clustering Prototypes: Acquiring Semantic Content Proxies Without Labels

The construction of pseudo-counterfactual pairs in ProtoFair hinges on a premise—knowing which samples are semantically similar. In self-supervised scenarios without target labels, this work utilizes the assignment results of unsupervised clustering prototypes as a proxy for content similarity. Specifically, \(K\) prototype vectors \(\{c_k\}_{k=1}^K\) are maintained in the clustering embedding space (output by \(h_\psi\)), which are not parameters learned via backpropagation but are maintained as non-learnable running estimates. After a warmup period of several epochs (where the encoder is trained exclusively with \(\mathcal{L}_{\text{SSL}}\)), K-Means is executed on the entire training set for clustering initialization: \(\{c_k\} \leftarrow \text{K-Means}(\{\bar{h}_i = \frac{h_\psi(f_\theta(x_i))}{\|h_\psi(f_\theta(x_i))\|}\}_{i=1}^N)\). Subsequently, a full K-Means initialization is re-executed every \(R\) epochs to prevent prototype drift. Between re-initializations, each iteration smoothly tracks the evolution of the feature space using an exponential moving average (EMA): \(c_k \leftarrow \text{normalize}(m \cdot c_k + (1-m) \cdot \frac{\sum_{i:\hat{k}_i=k} \bar{h}_i}{|\{i:\hat{k}_i=k\}|})\), where \(m \in [0,1)\) is the momentum coefficient. Each sample is assigned to the nearest prototype via cosine similarity: \(\hat{k}_i = \arg\max_k \bar{h}_i^\top c_k\).

The ingenuity of this design lies in a triple decoupling: (i) the clustering prototypes are decoupled from the encoder parameters (prototypes do not update via gradients), avoiding the trivial solution of collapsing all samples into a single cluster; (ii) the clustering head \(h_\psi\) is separated from the contrastive head \(g_\phi\), ensuring that base SSL quality and fairness regularization operate independently in separate subspaces; (iii) detaching the clustering assignment during backpropagation forms an EM alternating optimization—the clustering structure in the E-step reflects "content similarity under the current representation" rather than a fake structure artificially shaped by the fairness loss.

2. Pseudo-Counterfactual Pair Construction: Cross-Group Content Matching Mechanism

With clustering assignments and sensitive attributes, ProtoFair defines three types of pairwise relationships. The condition for positive pairs (pulled together) is same cluster but different sensitive group: \(\mathcal{P}_i = \{j \neq i \mid \hat{k}_j = \hat{k}_i \text{ 且 } s_j \neq s_i\}\). This condition directly operationalizes the intuition of counterfactual fairness—if two samples share the same semantic content (same cluster) but have different sensitive attributes, their representations should be consistent. The other two types of pairs do not appear in the numerator: (i) samples from different clusters are pushed apart as negative samples in the denominator; (ii) samples in the same cluster and from the same sensitive group are also not treated as positive pairs—this exclusion is crucial, as it ensures the loss specifically targets cross-group alignment rather than general similarity within the cluster, preventing the model from merely learning tighter clusters for within-group samples without eliminating cross-group disparities.

The in-batch contrastive fairness loss follows the standard contrastive learning framework:

\[\mathcal{L}_{\text{within}} = -\frac{1}{|\mathcal{V}|}\sum_{i\in\mathcal{V}}\frac{1}{|\mathcal{P}_i|}\sum_{j\in\mathcal{P}_i}\log\frac{\exp(z_i^\top z_j / \tau)}{\sum_{k=1,k\neq i}^{B}\exp(z_i^\top z_k / \tau)}\]

where \(\mathcal{V} = \{i : |\mathcal{P}_i| > 0\}\) is the set of samples with at least one pseudo-counterfactual partner in the batch, and \(\tau\) is the temperature coefficient. Note that the features \(z_i\) used here originate from the contrastive projection head \(g_\phi\) instead of the clustering head \(h_\psi\), meaning that the fairness gradient propagates only through the highest-quality representation space and does not interfere with the clustering subspace.

3. Cross-Batch Queue: Overcoming Pairing Bottlenecks Under Small Batches/Imbalance

The in-batch loss has a practical bottleneck: pseudo-counterfactual pairs require samples of the same cluster but different sensitive groups to co-occur in the same mini-batch. When the batch size is small or the sensitive group distribution is highly imbalanced, qualified positive pairs can be extremely scarce, diluting the fairness signal. To this end, ProtoFair borrows the queue design from MoCo, maintaining a FIFO queue \(\mathcal{Q}\) to store sample information from the recent \(M\) batches. Each record contains a triplet of the feature vector, cluster assignment, and sensitive attribute \((z_j^q, \hat{k}_j^q, s_j^q)\), with a queue capacity of \(Q = M \times B\). The queue entries are detached tensors and do not participate in gradient updates.

In each training step, the current batch samples are matched against the entire queue to discover additional cross-group positive samples: \(\mathcal{P}_i^q = \{j \in \mathcal{Q} \mid \hat{k}_j^q = \hat{k}_i \text{ 且 } s_j^q \neq s_i\}\). The cross-batch loss takes the same form as the in-batch loss, but positive samples are retrieved from the queue and the denominator iterates over the entire queue to provide a large number of diverse negative samples:

\[\mathcal{L}_{\text{cross}} = -\frac{1}{|\mathcal{V}^q|}\sum_{i\in\mathcal{V}^q}\frac{1}{|\mathcal{P}_i^q|}\sum_{j\in\mathcal{P}_i^q}\log\frac{\exp(z_i^\top z_j^q / \tau)}{\sum_{k\in\mathcal{Q}}\exp(z_i^\top z_k^q / \tau)}\]

The practical effect of this design is to dramatically expand the search space for pseudo-counterfactual pairs—evolving from "occasional occurrence within the same batch" to "continuous tracking across the most recent \(M\) batches", which significantly enhances the fairness gradient signal, especially in scenarios with imbalanced sensitive groups. The queue starts empty and is naturally filled during training; when the queue has not accumulated enough entries, the contribution of \(\mathcal{L}_{\text{cross}}\) is zero, which does not affect training stability.

Loss & Training

The complete ProtoFair regularization term is the sum of the in-batch and cross-batch components: \(\mathcal{L}_{\text{CF}} = \mathcal{L}_{\text{within}} + \mathcal{L}_{\text{cross}}\). The total training objective is \(\mathcal{L} = \mathcal{L}_{\text{SSL}} + \lambda \mathcal{L}_{\text{CF}}\), where \(\lambda > 0\) controls the strength of fairness regularization. ProtoFair is not applied from the very beginning; instead, a pure SSL warmup phase is executed for several epochs to let the encoder learn meaningful representations, after which the clustering prototypes are initialized and the fairness loss is activated. The clustering prototypes are fully re-initialized via K-Means every \(R\) epochs, with EMA smoothly tracking them in between. Training is performed using SGD (momentum 0.9, weight decay \(10^{-4}\), initial learning rate 0.1 + cosine annealing), using ResNet-18 as the backbone, and two independent MLP projection heads serving the contrastive and clustering losses respectively. Downstream evaluation follows the linear probing protocol: a linear classifier is trained on top of the frozen encoder representations.

Key Experimental Results

Main Results

CelebA (SupCon + ProtoFair): The table below extracts representative target-sensitive attribute combinations from Table 1. ProtoFair substantially reduces EO across all scenarios with minimal accuracy loss. On (T:e, S:y), EO drops from 10.8 to 1.9, matching the optimal result of the specialized fairness method FSCL (1.8) but yielding a 1.3 percentage point higher accuracy. The most prominent fairness improvement occurs on (T:a, S:m): EO drops from 30.5 to 14.2 (a 53% reduction), while accuracy remains virtually unchanged (80.5 vs 80.3).

Method	T:a,S:m ACC	T:a,S:m EO	T:b,S:y ACC	T:b,S:y EO	T:e,S:y ACC	T:e,S:y EO
CE	79.6	27.8	84.5	14.7	83.8	12.7
GRL	77.2	24.9	83.3	10.0	82.3	5.9
LNL	79.9	21.8	82.3	6.8	80.3	3.3
FSCL	79.1	11.5	83.8	6.4	82.0	1.8
SupCon	80.5	30.5	84.4	16.9	84.0	10.8
SupCon + ProtoFair	80.3	14.2	83.5	6.9	83.3	1.9

Self-Supervised Scenarios (SimCLR / BarlowTwins / BYOL): ProtoFair is equally applicable to purely self-supervised scenarios without target labels. On top of SimCLR, ProtoFair reduces EO from 29.4 to 21.9 (on par with SimCLR+GRL) while maintaining higher accuracy (73.3 vs 72.3). Consistent EO improvements are observed on BarlowTwins and BYOL, verifying its plug-and-play capability across different SSL frameworks.

Method	Base Framework	T:e,S:y EO	T:e,S:m EO	T:b,S:y EO	T:b,S:m EO
SimCLR	SimCLR	-	-	-	-
SimCLR + ProtoFair	SimCLR	-	-	-	-
BarlowTwins	BarlowTwins	1.23	1.64	6.72	12.07
BarlowTwins + ProtoFair	BarlowTwins	1.16	1.36	1.15	5.98
BYOL	BYOL	11.21	17.04	8.90	13.54
BYOL + ProtoFair	BYOL	8.98	7.11	4.59	6.83

UTKFace Robustness to Imbalance: Across different data imbalance ratios \(\alpha \in \{2,3,4\}\), ProtoFair consistently reduces EO (from 10.6 to 6.6 when \(\alpha=4\), and from 4.5 to 2.8 when \(\alpha=2\)), and the relative improvement remains stable in high-bias scenarios, indicating that the method scales gracefully with the degree of imbalance.

NIH Chest X-rays Cross-Domain Generalization: In the medical imaging field, ProtoFair reduces the gender gap in AUROC for pneumothorax classification from 0.03 to 0.01, while slightly improving the overall AUROC (from 0.70 to 0.72), verifying the effectiveness of the method in non-facial scenarios.

Ablation Study

Configuration	Key Observation
\(\lambda=0.3\), Different \(K\)	Accuracy remains stable at 83.0–84.5 under \(K \in \{5,10,15,20,30\}\). EO is insensitive to the number of clusters, and the default \(K=10\) is reasonable.
\(K=10\), Different \(\lambda\)	\(\lambda \in [0.1, 0.3]\) is the optimal range, yielding significant fairness improvements with minimal accuracy loss; for \(\lambda \geq 0.7\), accuracy drops noticeably, over-constraining the representations.
Removing Cross-Batch Queue	Fairness improvement degrades in imbalanced sensitive group scenarios, validating the critical role of the queue in expanding the positive sample search space.
Removing detach Operation	The clustering structure is run-away shaped by the fairness loss, potentially causing collapse and degeneration, validating the necessity of the EM alternating optimization.

Key Findings

• Clustering prototype quality is a core premise: The warmup period and periodic K-Means re-initialization are key to ensuring semantically meaningful clustering. Insufficient warmup leads to poor-quality pseudo-counterfactual pairs and the disappearance of fairness gains.
• A sweet spot exists for \(\lambda\): Within \([0.1, 0.3]\), fairness improves significantly with almost no loss in accuracy; values that are too large pull cross-group samples excessively, declining representation discriminability. The optimal \(\lambda\) varies slightly across different base SSL methods and target-sensitive attribute combinations (dropping to 0.2 in BYOL experiments).
• The cross-batch queue contributes most in imbalanced scenarios: When the proportions of sensitive groups are balanced, in-batch positive pairs are sufficient, making the marginal gain of the queue small. However, under severe group imbalance, the queue becomes the primary source of fairness gradient signals.
• Only 5 extra epochs are needed to take effect: In most CelebA experiments, ProtoFair requires only 5–10 epochs of additional training to substantially reduce EO, incurring minimal extra computational overhead.
• Predictability of sensitive attributes decreases: The linear probing accuracy for predicting sensitive attributes drops from 87.76% to 81.02% (on the Big Nose task) and from 82.41% to 76.07% (on the Bags Under Eyes task), aligning with t-SNE visualizations showing a more uniform mixing of cross-group samples.

Highlights & Insights

• Philosophical height of the plug-and-play design: The design of not modifying the base SSL objective makes ProtoFair naturally compatible with future SOTA self-supervised methods, requiring it simply to be appended as an extra regularization term. This "leave the core untouched, only add regularization" philosophy can be extended to other scenarios where constraints (such as robustness or OOD generalization) need to be layered onto mature systems.
• Using clustering for fairness is "leveraging existing strength": Unsupervised clustering in SSL is already widely used to enhance representation quality (e.g., DeepCluster, SwAV, PCL). This paper ingeniously redirects it towards fairness objectives—since clustering structures naturally capture content similarity, they seamlessly satisfy the "same content" condition of pseudo-counterfactual pairs without requiring additional modeling.
• Deliberately not treating same-cluster same-group samples as positives: This seemingly minor design choice is actually profound—if same-group pairs were allowed in the numerator, the loss would degenerate into "enhancing cluster tightness" instead of "eliminating cross-group variations," severely undermining the fairness effect. This detail demonstrates a clear understanding of the fundamental difference between "pan-clustering compaction" and "cross-group alignment."
• The detach design to prevent collapse is transferable: Detaching the clustering assignment to form an EM-style alternating optimization is a critical technique to prevent "the fairness loss from backward polluting the clustering structure." This strategy can be adopted by any method that dynamically constructs pseudo-labels based on current representations to apply auxiliary losses.

Limitations & Future Work

• Dependence on clustering semantic quality: The effectiveness of ProtoFair relies on the premise that clustering can capture meaningful semantic content. If the data itself is unsuitable for prototype-based clustering (such as in low-resolution or high-noise scenarios), the quality of pseudo-counterfactual pairs will degrade. Although the authors prevent this through a warmup period and periodic K-Means re-initialization, they do not provide a clear fallback strategy for when clustering fails.
• Currently validated only on binary sensitive attributes: All validated sensitive attributes (gender, age bracket binary, race binarized) are binary, leaving the performance on multi-class sensitive attributes (e.g., multi-ethnic, multi-age segments) or continuous sensitive attributes (e.g., income levels) unexplored.
• Downstream tasks limited to attribute classification: Fairness evaluation is conducted only on linear probing classification accuracy and Equalized Odds, without verifying representation fairness transferability to more complex downstream tasks (such as object detection, segmentation, or retrieval).
• Room for integration with stronger data augmentations: Currently, ProtoFair relies only on standard data augmentations, without considering generative augmentations or sensitive-attribute-aware data resampling. Combining the queue mechanism with targeted data augmentations for minority groups could further alleviate fairness issues under extreme imbalances.
• Specific improvement directions: (i) Introduce prototype confidence weighting to downweight pseudo-counterfactual pairs of samples located near fuzzy cluster boundaries; (ii) explore soft clustering assignments instead of hard assignments to allow boundary samples to contribute partial fairness signals; (iii) extend ProtoFair to multi-modal self-supervised scenarios (such as CLIP-style image-text contrast) to examine the cross-modal effects of the fairness regularization term.

Related Work & Insights

• vs FSCL (Park et al. 2022): FSCL achieves fairness by modifying the negative sample selection strategy in supervised contrastive loss (excluding negative samples from the same sensitive group), which is essentially "pushing away from the denominator" rather than "pulling together in the numerator"—the former constrains the model from exploiting sensitive attributes, while the latter actively encourages cross-group invariance. ProtoFair is complementary to FSCL and can be combined: FSCL provides a fairness constraint from the denominator side, and ProtoFair provides a cross-group alignment signal from the numerator side. Additionally, FSCL requires target labels to define positive pairs, whereas ProtoFair only requires sensitive attributes.
• vs Adversarial de-biasing methods (GRL, LNL): Adversarial methods train sensitive attribute discriminators with gradient reversal layers to "erase" sensitive information from representations, following a "what not to learn" paradigm; ProtoFair represents "what to learn"—actively pulling together cross-group samples of the same content. The instability and hyperparameter sensitivity of adversarial training have always been pain points in practical deployment; ProtoFair's standard contrastive loss formulation naturally avoids minimax optimization.
• vs PCL (Li et al. 2021): PCL uses momentum-updated prototypes for prototype contrastive learning to improve representation quality. ProtoFair directly reuse its prototype maintenance mechanism but with an entirely different objective—PCL pulls samples of the same prototype together to improve representations, whereas ProtoFair further filters a subset of "same prototype + different sensitive group" to improve fairness. This indicates that the reusability of clustering infrastructure in SSL far exceeds its original design objectives.
• Theoretical connection to counterfactual fairness: Counterfactual fairness by Kusner et al. (2017) requires structural causal models and causal graph knowledge, which is barely feasible in practical visual data. ProtoFair uses clustering as a "good enough but imperfect" causal proxy—assuming that clustering captures non-sensitive causal upstream variables. Although less rigorous than true causal inference, empirical results demonstrate that this weak assumption is sufficient to yield substantial fairness improvements on standard benchmarks.

Rating

• Novelty: ⭐⭐⭐⭐ Utilizing unsupervised clustering for fairness to construct pseudo-counterfactual pairs is a fresh perspective. The plug-and-play design concept is clean and distinct from existing coupled schemes, although the core components (prototype clustering, queue) are drawn from existing work, rendering the combination novel but without fundamentally new mechanisms.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively validated across three datasets, four base SSL methods, and both supervised and unsupervised scenarios. Ablation studies cover key hyperparameters \(K\) and \(\lambda\), and t-SNE visualizations alongside sensitive attribute predictability analyses are provided. Preliminary cross-domain validation (from faces to medical imaging) is also conducted.
• Writing Quality: ⭐⭐⭐⭐ The problem statement is clear, and the methodological motivation chain is complete (contradiction-driven narrative). Each step in the design explains "why" rather than just "what was done," keeping the logical relationships among the three components highly transparent.
• Value: ⭐⭐⭐⭐ The plug-and-play design ensures ProtoFair can be readily adopted by any SSL pipeline without redesigning training processes or loss functions, translating to high practical utility. The limitation lies in validating only binary sensitive attributes and attribute classification downstream tasks, with clustering semantic quality being an implicit prerequisite, leaving its generalization in more complex scenarios to be verified.

Related Papers

• [ICCV 2025] Backdooring Self-Supervised Contrastive Learning by Noisy Alignment
• [CVPR 2025] INACTIVE: Invisible Backdoor Attack against Self-supervised Learning
• [ECCV 2026] Improving Adversarial Robustness via Activation Amplification and Attenuation
• [CVPR 2026] ProxyFL: A Proxy-Guided Framework for Federated Semi-Supervised Learning
• [ICLR 2026] Fair Reinforcement Learning for Just AI