ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

Conference: ECCV 2026
arXiv: 2605.01971
Code: None
Area: AI Safety / Fairness / Self-Supervised Learning
Keywords: Self-Supervised Learning, Contrastive Learning, Fair Representation, Pseudo-Counterfactual Pairs, Clustering Prototypes

TL;DR

ProtoFair proposes a plug-and-play fairness regularization term. Without modifying existing self-supervised contrastive learning objectives, it discovers "same semantic content but different sensitive group" pseudo-counterfactual pairs via momentum-updated unsupervised clustering prototypes. By pulling these cross-group samples closer in the embedding space, it forces the encoder to learn representations invariant to sensitive attributes. When paired with SimCLR / SupCon / BarlowTwins / BYOL, it significantly reduces Equalized Odds on CelebA, UTKFace, and NIH Chest X-rays while maintaining competitive accuracy.

Background & Motivation

Self-supervised contrastive learning (such as SimCLR, SupCon, and BYOL) has become a mainstream paradigm for learning high-quality visual representations without relying on annotations, performing comparably to or even surpassing supervised methods on multiple downstream tasks. However, recent studies have revealed a concerning fact: even though the training process does not access explicit labels, these self-supervised representations still encode demographic biases present in the training data. For example, in facial attribute classification, models systematically exploit sensitive information like gender and race to make predictions, leading to significantly discrepant error rates across different demographic groups.

To address this issue, two mainstream approaches exist, but both possess fundamental limitations. Adversarial debiasing methods (e.g., GRL, LNL) train auxiliary discriminators to remove sensitive information from representations, but they introduce minimax optimization, which is unstable during training, sensitive to hyperparameters, and inherently modifies the training process itself rather than supplementing the original objective. Methods modifying the contrastive objective (e.g., FSCL) redesign the positive and negative sample sampling strategies within the loss function to reduce bias, but these methods are tightly coupled with specific SSL frameworks. Whenever a new SOTA self-supervised method emerges, the fairness scheme must be redesigned from scratch. The common pain point of both approaches is: they require replacing or fundamentally altering the base training objective, which yields extremely poor portability in the currently rapid iteration of SSL methods.

This paper raises a different question: Is it possible to make existing SSL methods naturally fairer by introducing a lightweight auxiliary regularization term, rather than redesigning the SSL objective? The core idea stems from the intuition of counterfactual fairness—if an individual's sensitive attribute changes but their representation remains unchanged, then the representation is fair. ProtoFair uses unsupervised clustering as a proxy for semantic content, constructs pseudo-counterfactual pairs (samples in the same cluster but belonging to different sensitive groups), and pulls them together in the embedding space to actively encourage the encoder to learn features invariant to sensitive attributes.

Method

Overall Architecture

The core mechanism of ProtoFair is to formulate fairness regularization as a plug-and-play auxiliary loss \(\mathcal{L}_{\text{CF}}\) added on top of any existing SSL loss \(\mathcal{L}_{\text{SSL}}\), with the total loss being \(\mathcal{L} = \mathcal{L}_{\text{SSL}} + \lambda \mathcal{L}_{\text{CF}}\), where \(\lambda\) controls the intensity of the fairness regularization. The base SSL objective remains completely unchanged, and ProtoFair influences the encoder solely through the gradients of \(\mathcal{L}_{\text{CF}}\).

The overall pipeline is divided into three parallel branches: after receiving input samples, the shared encoder \(f_\theta\) feeds them through a contrastive projection head \(g_\phi\) (producing L2-normalized features \(z_i\), used by both the base SSL loss and the ProtoFair loss) and a clustering projection head \(h_\psi\) (producing clustering space features \(\bar{h}_i\), dedicated to prototype assignment). The clustering side maintains \(K\) momentum-updated prototype vectors \(\{c_k\}_{k=1}^K\), which assign hard cluster labels \(\hat{k}_i\) to each sample through K-Means initialization and EMA tracking. These cluster labels, combined with the sensitive attributes \(s_i\), define the pseudo-counterfactual positive sample set \(\mathcal{P}_i = \{j \mid \hat{k}_j = \hat{k}_i \text{ and } s_j \neq s_i\}\). Positive pairs are retrieved both within the current batch (\(\mathcal{L}_{\text{within}}\)) and from a cross-batch FIFO queue (\(\mathcal{L}_{\text{cross}}\)). Adding these two components yields \(\mathcal{L}_{\text{CF}}\), which is then weighted and summed with \(\mathcal{L}_{\text{SSL}}\). The cluster assignments are detached from the computation graph before being passed to the ProtoFair loss, establishing an EM-style alternating optimization: the E-step fixes the clustering pseudo-labels, and the M-step optimizes the encoder and the contrastive head.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input batch x<br/>with sensitive attribute s"] --> B["Shared encoder fθ"]
    B --> C["Contrastive projection head gφ<br/>→ Normalized feature z"]
    B --> D["Clustering projection head hψ<br/>→ Clustering feature h̄"]
    D --> E["Momentum-updated clustering prototypes<br/>K-Means initialization + EMA tracking"]
    E --> F["Hard cluster assignment k̂ = argmax(h̄ᵀc)"]
    F --> G["Pseudo-counterfactual pair construction<br/>Same cluster + different sensitive group = Positive sample"]
    C --> G
    A --> G
    G --> H["In-batch contrastive loss L_within"]
    G --> I["Cross-batch queue<br/>FIFO stores history (z, k̂, s)"]
    I --> J["Cross-batch contrastive loss L_cross"]
    H --> K["LCF = L_within + L_cross"]
    J --> K
    C --> L["Base SSL loss LSSL<br/>(SimCLR/SupCon/BYOL...)"]
    K --> M["Total loss L = LSSL + λ·LCF"]
    L --> M

Key Designs

1. Momentum-Updated Clustering Prototypes: Acquiring Semantic Content Proxies Without Labels

ProtoFair requires a prerequisite to construct pseudo-counterfactual pairs: knowing which samples are semantically similar. In self-supervised scenarios lacking target labels, this work utilizes unsupervised clustering prototype assignments as a proxy for content similarity. Specifically, it maintains \(K\) prototype vectors \(\{c_k\}_{k=1}^K\) in the clustering embedding space (output by \(h_\psi\)), which are not parameters learned through backpropagation, but are instead maintained as non-learnable running estimates. After a warm-up period of several epochs (where the encoder is trained solely using \(\mathcal{L}_{\text{SSL}}\)), K-Means is executed on the entire training set to initialize the clustering prototypes: \(\{c_k\} \leftarrow \text{K-Means}(\{\bar{h}_i = \frac{h_\psi(f_\theta(x_i))}{\|h_\psi(f_\theta(x_i))\|}\}_{i=1}^N)\). Subsequently, a complete K-Means re-initialization is executed every \(R\) epochs to prevent prototype drift. Between these re-initializations, at each step, the evolution of the feature space is smoothed via Exponential Moving Average (EMA) tracking: \(c_k \leftarrow \text{normalize}(m \cdot c_k + (1-m) \cdot \frac{\sum_{i:\hat{k}_i=k} \bar{h}_i}{|\{i:\hat{k}_i=k\}|})\), where \(m \in [0,1)\) is the momentum coefficient. Each sample is assigned to its nearest prototype based on cosine similarity: \(\hat{k}_i = \arg\max_k \bar{h}_i^\top c_k\).

The elegance of this design lies in its triple decoupling: (i) prototype vectors are decoupled from encoder parameters (prototypes are not updated via gradients), preventing the trivial solution where all samples collapse into a single cluster; (ii) the clustering head \(h_\psi\) and contrastive head \(g_\phi\) are separated, ensuring that base SSL quality and fairness regularization function in separate, independent subspaces; (iii) detaching the cluster assignments formulates an EM alternating optimization—the clustering structure in the E-step reflects "content similarity under the current representation" rather than a spurious structure backward-shaped by the fairness loss.

2. Pseudo-Counterfactual Pair Construction: Cross-Group Content Alignment Mechanism

With the cluster assignments and sensitive attributes, ProtoFair defines three types of pairwise relationships. Positive pairs (to be pulled closer) are conditional on being in the same cluster but different sensitive groups: \(\mathcal{P}_i = \{j \neq i \mid \hat{k}_j = \hat{k}_i \text{ and } s_j \neq s_i\}\). This condition directly operationalizes the intuition of counterfactual fairness—if two samples share the same semantic content (same cluster) but have different sensitive attributes, their representations should behave consistently. The remaining two types of pairs do not appear in the numerator: (i) samples from different clusters are pushed apart as negative samples in the denominator; (ii) samples from the same cluster and same sensitive group are also not treated as positive samples—this exclusion is crucial because it ensures the loss specifically targets cross-group alignment rather than general intra-cluster similarity, avoiding cases where the model merely learns a tighter cluster for intra-group samples without resolving cross-group discrepancies.

The in-batch contrastive fairness loss follows the standard contrastive learning framework:

\[\mathcal{L}_{\text{within}} = -\frac{1}{|\mathcal{V}|}\sum_{i\in\mathcal{V}}\frac{1}{|\mathcal{P}_i|}\sum_{j\in\mathcal{P}_i}\log\frac{\exp(z_i^\top z_j / \tau)}{\sum_{k=1,k\neq i}^{B}\exp(z_i^\top z_k / \tau)}\]

where \(\mathcal{V} = \{i : |\mathcal{P}_i| > 0\}\) is the set of samples in the batch that have at least one pseudo-counterfactual partner, and \(\tau\) is the temperature coefficient. Note that the features \(z_i\) utilized here originate from the contrastive projection head \(g_\phi\) instead of the clustering head \(h_\psi\), meaning that fairness gradients propagate only through the space with the highest representation quality, without interfering with the clustering subspace.

3. Cross-Batch Queue: Overcoming Pairing Bottlenecks Under Small Batches/Imbalance

In-batch loss has a practical bottleneck: pseudo-counterfactual pairs require samples of the same cluster but different sensitive groups to co-occur within the same mini-batch. When the batch size is small or the sensitive group distribution is highly imbalanced, qualifying positive pairs can be extremely scarce, diluting the fairness signal. To address this, drawing inspiration from MoCo's queue design, ProtoFair maintains a FIFO queue \(\mathcal{Q}\) that stores sample information from the most recent \(M\) batches. Each record contains a triplet of the feature vector, cluster assignment, and sensitive attribute \((z_j^q, \hat{k}_j^q, s_j^q)\), with a queue capacity of \(Q = M \times B\). The queue entries are detached tensors and do not participate in gradient updates.

In each training step, the current batch samples are matched with the entire queue to discover additional cross-group positive pairs: \(\mathcal{P}_i^q = \{j \in \mathcal{Q} \mid \hat{k}_j^q = \hat{k}_i \text{ and } s_j^q \neq s_i\}\). The formulation of the cross-batch loss is identical to that of the in-batch loss, except that positive samples are selected from the queue, and the denominator traverses the entire queue to supply a broad array of diverse negative samples:

\[\mathcal{L}_{\text{cross}} = -\frac{1}{|\mathcal{V}^q|}\sum_{i\in\mathcal{V}^q}\frac{1}{|\mathcal{P}_i^q|}\sum_{j\in\mathcal{P}_i^q}\log\frac{\exp(z_i^\top z_j^q / \tau)}{\sum_{k\in\mathcal{Q}}\exp(z_i^\top z_k^q / \tau)}\]

The practical effect of this design is to substantially expand the search space for pseudo-counterfactual pairs—evolving from "occasional occurrence within the same batch" to "continuous tracking across the most recent \(M\) batches." This significantly strengthens the fairness gradient signal, especially in highly imbalanced sensitive group scenarios. The queue is initially empty and naturally populates as training progresses; before the queue accumulates enough entries, the contribution of \(\mathcal{L}_{\text{cross}}\) is zero, safeguarding training stability.

Loss & Training

The complete ProtoFair regularization term is the sum of the in-batch and cross-batch components: \(\mathcal{L}_{\text{CF}} = \mathcal{L}_{\text{within}} + \mathcal{L}_{\text{cross}}\). The overall training objective is \(\mathcal{L} = \mathcal{L}_{\text{SSL}} + \lambda \mathcal{L}_{\text{CF}}\), where \(\lambda > 0\) controls the strength of the fairness regularization. Instead of imposing ProtoFair from the beginning, a pure SSL warm-up phase is first conducted for several epochs to allow the encoder to learn meaningful representations, after which the clustering prototypes are initialized and the fairness objective is activated. Clustering prototypes undergo a comprehensive K-Means re-initialization every \(R\) epochs, with EMA smooth tracking in between. Training uses SGD (momentum 0.9, weight decay \(10^{-4}\), initial learning rate 0.1 with cosine annealing) with a ResNet-18 backbone. Two independent MLP projection heads serve the contrastive and clustering losses, respectively. Downstream evaluation follows the linear probing protocol: freeze the encoder and train a linear classifier on top of the learned representations.

Key Experimental Results

Main Results

CelebA (SupCon + ProtoFair): The table below extracts representative target-sensitive attribute combinations from Table 1. ProtoFair drastically reduces EO across all scenarios with minimal accuracy trade-offs. On (T:e, S:y), EO drops from 10.8 to 1.9, matching the optimal result of the specialized fairness method FSCL (1.8) while achieving a 1.3 percentage point higher accuracy. The most notable fairness improvement occurs on (T:a, S:m): EO drops from 30.5 to 14.2 (a 53% reduction) with almost no impact on accuracy (80.5 vs 80.3).

Method	T:a,S:m ACC	T:a,S:m EO	T:b,S:y ACC	T:b,S:y EO	T:e,S:y ACC	T:e,S:y EO
CE	79.6	27.8	84.5	14.7	83.8	12.7
GRL	77.2	24.9	83.3	10.0	82.3	5.9
LNL	79.9	21.8	82.3	6.8	80.3	3.3
FSCL	79.1	11.5	83.8	6.4	82.0	1.8
SupCon	80.5	30.5	84.4	16.9	84.0	10.8
SupCon + ProtoFair	80.3	14.2	83.5	6.9	83.3	1.9

Self-Supervised Scenarios (SimCLR / BarlowTwins / BYOL): ProtoFair is equally applicable to pure self-supervised scenarios lacking target labels. Building on SimCLR, ProtoFair reduces EO from 29.4 to 21.9 (on par with SimCLR+GRL) while retaining higher accuracy (73.3 vs 72.3). Consistent EO improvements are observed across BarlowTwins and BYOL, verifying the plug-and-play capability across various SSL frameworks.

Method	Base Framework	T:e,S:y EO	T:e,S:m EO	T:b,S:y EO	T:b,S:m EO
SimCLR	SimCLR	-	-	-	-
SimCLR + ProtoFair	SimCLR	-	-	-	-
BarlowTwins	BarlowTwins	1.23	1.64	6.72	12.07
BarlowTwins + ProtoFair	BarlowTwins	1.16	1.36	1.15	5.98
BYOL	BYOL	11.21	17.04	8.90	13.54
BYOL + ProtoFair	BYOL	8.98	7.11	4.59	6.83

UTKFace Robustness to Imbalance: Across different data imbalance ratios \(\alpha \in \{2,3,4\}\), ProtoFair consistently reduces EO (from 10.6 to 6.6 when \(\alpha=4\), and from 4.5 to 2.8 when \(\alpha=2\)), and the relative improvement remains robust under high-bias scenarios, indicating that the method scales gracefully with the degree of imbalance.

NIH Chest X-rays Cross-Domain Generalization: In medical image analysis, ProtoFair shrinks the gender gap in AUROC for pneumothorax classification from 0.03 to 0.01 while slightly improving overall AUROC (0.70 to 0.72), validating the effectiveness of the method in non-facial domains.

Ablation Study

Configuration	Key Observation
\(\lambda=0.3\), different \(K\)	Accuracy remains stable at 83.0–84.5 under \(K \in \{5,10,15,20,30\}\); EO is insensitive to the number of clusters, making the default \(K=10\) a reasonable choice.
\(K=10\), different \(\lambda\)	\(\lambda \in [0.1, 0.3]\) is the optimal interval where fairness is significantly improved with negligible accuracy loss; accuracy drops noticeably when \(\lambda \geq 0.7\), indicating over-constrained representations.
Remove cross-batch queue	Fairness improvement degrades in sensitive group imbalance scenarios, validating the crucial role of the queue in expanding the search range of positive samples.
Remove detach operation	Clustering structures are backward-shaped by the fairness loss, potentially causing collapse. This validates the necessity of the EM alternating optimization.

Key Findings

• The quality of clustering prototypes is a core prerequisite: The warm-up phase and periodic K-Means re-initialization are key to ensuring meaningful cluster semantics. Insufficient warm-up leads to poor pseudo-counterfactual pair quality and deprives the model of fairness improvements.
• There exists a sweet spot for \(\lambda\): Within \([0.1, 0.3]\), fairness improves significantly with almost zero accuracy loss. Exceeding this range overly squeezes cross-group differences, degrading the representation's discriminative power. The optimal \(\lambda\) slightly varies across different base SSL methods and target-sensitive attribute combinations (e.g., dropping to 0.2 in BYOL experiments).
• The cross-batch queue contributes most in imbalanced settings: When the ratios of sensitive groups are balanced, in-batch matching provides sufficient positive pairs, making the marginal gain of the queue minor. However, under high group imbalance, the queue becomes the primary driver of the fairness gradient signal.
• Effective in as few as 5 additional epochs: In most CelebA experiments, ProtoFair significantly reduces EO with only 5–10 additional epochs of training, introducing negligible extra computational overhead.
• Predictability of sensitive attributes drops: The accuracy of linear probing for sensitive attributes drops from 87.76% to 81.02% (on the Big Nose task) and from 82.41% to 76.07% (on the Bags Under Eyes task), consistent with the t-SNE visualization showing cross-group samples mixing more uniformly.

Highlights & Insights

• Philosophical height of plug-and-play design: The approach of not altering the base SSL objective enables ProtoFair to naturally align with future SOTA self-supervised methods by merely appending an extra regularization term. This "leave-the-core-alone, add-regularization" philosophy can be extended to other scenarios where one wishes to superimpose constraints (such as robustness or OOD generalization) onto mature systems.
• Using clustering for fairness leverages existing strengths: Unsupervised clustering is already widely utilized within SSL to enhance representation quality (e.g., DeepCluster, SwAV, PCL). This work ingeniously redirects it to a fairness goal—clustering structures inherently capture content similarity, which naturally satisfies the "same content" condition of pseudo-counterfactual pairs without requiring additional modeling.
• Deliberate exclusion of intra-group, intra-cluster positive pairs: This seemingly minor design choice is actually profound. If intra-group pairs were allowed in the numerator, the loss would degenerate into "tightening intra-cluster density" rather than "eliminating cross-group discrepancies," severely compromising the fairness effect. This detail shows a clear conceptual distinction between "general clustering contraction" and "cross-group alignment."
• Transferable detach design preventing collapse: Detaching the cluster assignments to construct an EM alternating optimization is a critical technique to prevent the "fairness loss from backward-contaminating the clustering structure." This strategy can be readily adopted by any method that dynamically constructs pseudo-labels based on current representations to define auxiliary losses.

Limitations & Future Work

• Dependence on clustering semantic quality: The effectiveness of ProtoFair relies on the premise that clustering captures meaningful semantic content. If the data is inherently unsuited for prototype-based clustering (e.g., low-resolution, high-noise scenarios), the quality of pseudo-counterfactual pairs will degrade. Although the authors mitigate this via a warm-up phase and periodic K-Means re-initialization, they do not offer an explicit fallback strategy for clustering failure.
• Currently validated only on binary sensitive attributes: All evaluations utilize binary sensitive attributes (gender, young/old, binarized race), leaving the performance on multi-class sensitive attributes (e.g., multi-ethnic, multiple age bins) or continuous sensitive attributes (e.g., income level) unexplored.
• Downstream tasks confined to attribute classification: Fairness evaluations are only measured on linear probing classification accuracy and Equalized Odds, without verifying if representation fairness translates to more complex downstream tasks like object detection, segmentation, or retrieval.
• Room for combining with stronger data augmentations: Currently, ProtoFair relies strictly on standard data augmentations without considering generative augmentations or sensitive-attribute-aware data resampling. Combining the queue mechanism with targeted data augmentation for minority groups could further alleviate fairness drop-offs under extreme imbalance.
• Specific refinement directions: (i) Introduce prototype confidence weighting to downscale the weights of pseudo-counterfactual pairs near cluster boundaries; (ii) explore soft cluster assignments instead of hard assignments, allowing boundary samples to contribute partially to fairness signals; (iii) extend ProtoFair to multimodal self-supervised learning (e.g., CLIP-style vision-language contrastive learning) to test the cross-modal impact of the fairness regularization.

Related Work & Insights

• vs FSCL (Park et al. 2022): FSCL achieves fairness by modifying the negative sample selection strategy in supervised contrastive loss (excluding the same sensitive group from negative samples). Its nature is "excluding from the denominator" rather than "pulling together in the numerator"—the former constrains the model from exploiting sensitive attributes, while the latter actively encourages cross-group invariance. ProtoFair and FSCL are complementary and can be combined: FSCL imposes fairness constraints from the denominator side, and ProtoFair provides cross-group alignment signals from the numerator side. Additionally, FSCL requires target labels to define positive pairs, whereas ProtoFair only requires sensitive attributes.
• vs Adversarial debiasing (GRL, LNL): Adversarial methods train sensitive attribute discriminators with gradient reversal layers to "erase" sensitive information, representing a "what not to learn" paradigm. ProtoFair is a "what to learn" paradigm—actively pulling together cross-group, same-content samples. Training instability and hyperparameter sensitivity in adversarial training have long been major deployment obstacles; ProtoFair, formulated as a standard contrastive loss, naturally bypasses minimax optimization.
• vs PCL (Li et al. 2021): PCL performs prototype contrastive learning using momentum-updated prototypes to improve representation quality. ProtoFair direct-repurposes this prototype maintenance mechanism but with an entirely different target: PCL pulls samples of the same prototype closer to improve representation, while ProtoFair filters the "same prototype + different sensitive group" subset to improve fairness. This indicates that the reusability of clustering infrastructure in SSL far exceeds its original design intent.
• Theoretical connection to counterfactual fairness: Counterfactual fairness (Kusner et al. 2017) requires structural causal models and causal graph knowledge, which are highly impractical for raw visual data. ProtoFair uses clustering as a "good-enough but imperfect" causal proxy, assuming that clusters capture non-sensitive causal upstream variables. Though not as mathematically rigorous as formal causal inference, empirical results demonstrate that this weaker assumption is sufficient to yield substantial fairness improvements on standard benchmarks.

Rating

• Novelty: ⭐⭐⭐⭐ Utilizing unsupervised clustering to construct pseudo-counterfactual pairs for fairness introduces a fresh perspective. The plug-and-play design is clear and distinct from existing coupled schemes. However, its core components (prototype-based clustering, queue) are adapted from prior work; the integration is novel, but it does not introduce a fundamentally new mechanism.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively verified across three datasets, four base SSL methods, in both supervised and unsupervised regimes. Ablation studies comprehensively cover key hyperparameters \(K\) and \(\lambda\). The paper also provides t-SNE visualizations and sensitive attribute predictability analyses, with a preliminary cross-domain (face to medical imaging) validation.
• Writing Quality: ⭐⭐⭐⭐ The problem definition is clear, and the method's motivational chain is well-articulated (contradiction-driven narrative). Each design step explains why it was done rather than merely what was done, clarifying the logical interactions among the three components.
• Value: ⭐⭐⭐⭐ The plug-and-play design allows ProtoFair to be immediately adopted by any SSL pipeline without necessitating a redesign of training processes or loss functions, offering high practical value. The main limitations include evaluating only binary sensitive attributes, property classification downstream tasks, and relying on implicit cluster semantic quality, which leaves generalization to more complex scenarios yet to be verified.

Related Papers

• [ICCV 2025] Backdooring Self-Supervised Contrastive Learning by Noisy Alignment
• [CVPR 2025] INACTIVE: Invisible Backdoor Attack against Self-supervised Learning
• [ECCV 2026] Improving Adversarial Robustness via Activation Amplification and Attenuation
• [CVPR 2026] ProxyFL: A Proxy-Guided Framework for Federated Semi-Supervised Learning
• [ICLR 2026] Fair Reinforcement Learning for Just AI