Skip to content

ARCA: Attribution-Regularized Causal Alignment for Task-Aware Bias Mitigation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/WUT-IDEA/ARCA
Area: LLM Alignment
Keywords: Task-Aware Debiasing, Causal Alignment, Differential Gradient Attribution, Counterfactual Generation, Masked Language Models

TL;DR

ARCA addresses the issue of social bias re-emergence during downstream fine-tuning by automatically extracting task-specific bias-propagating words via differential gradient attribution and jointly optimizing task classification with representation-level causal alignment on dynamically generated counterfactual pairs.

Background & Motivation

Pretrained masked language models (MLMs), such as BERT and RoBERTa, remain pivotal backbones for natural language understanding (NLU) tasks due to their favorable trade-off between task performance and inference latency compared with massive decoder-only architectures. However, these models systematically encode societal prejudices and stereotypes present in massive web corpora. When adapted to specific downstream tasks, these implicit biases are often reinforced and amplified by dataset-level statistical co-occurrences, leading to discriminatory predictions in critical, high-stakes domains such as automated resume screening, loan qualification, and legal case assessment.

Existing debiasing paradigmsโ€”encompassing counterfactual data augmentation (CDA), linear nullspace projections (e.g., INLP, Sentence-Debias), and contrastive objective tuning (e.g., FairFil, TriCon-Fair)โ€”suffer from a core structural flaw: debiasing is detached from downstream task adaptation. Pre-finetuning bias mitigation rarely survives subsequent parameter updates driven entirely by task classification loss, as downstream objectives readily re-learn spurious demographic shortcuts. Furthermore, traditional approaches rely heavily on static, manually compiled gender lexicons. Such lexicons are expensive to curate, fail to scale, and cannot anticipate implicit, task-specific lexical carriers (e.g., descriptive traits or social roles) whose attribution shifts dramatically in specific downstream contexts. This disconnect obstructs end-to-end gradient flow and prevents the joint optimization of fairness and task utility.

From a structural causal perspective, downstream social bias arises because the model exploits protected demographic attributes as a spurious causal shortcut to prediction, rather than relying exclusively on true semantic indicators. Effective mitigation therefore requires directly intervening on these spurious paths within the downstream fine-tuning loop itself. The intervention space must be localized dynamically using internal gradient signals from the adapting model, while representation alignment must ensure invariant latent geometry under counterfactual attribute replacement. Core idea: treat social bias as a spurious causal pathway from protected attributes to task predictions, construct a task-specific intervention lexicon via differential gradient attribution on a warmed-up model, and jointly optimize classification loss with representation-level causal alignment on on-the-fly counterfactual pairs.

Method

Overall Architecture

ARCA is designed as an end-to-end debiasing framework that unifies downstream task tuning and causal representation alignment. The pipeline operates in two sequential stages: Stage 0: Warm-up & Lexicon Construction and Stage 1: Downstream Fine-tuning. In Stage 0, the model undergoes a single warm-up epoch on downstream training data. A compact seed set of canonical attribute pairs is used to perform initial substitutions on a sampled probing subset. Token-level differential gradient attributions are aggregated into word-level scores to automatically identify task-specific lexical carriers that disproportionately affect predictions when attributes shift. Candidate words are validated against semantic similarity and attribute opposition constraints to produce an expanded causal intervention lexicon. In Stage 1, the model is re-initialized to pretrained weights and fine-tuned; training batches are scanned on the fly against the constructed lexicon to produce counterfactual pairs, concurrently optimizing the task cross-entropy loss and a representation-level causal divergence loss between original and counterfactual sentence embeddings.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Downstream Training Data & Seed Lexicon"] --> B["Task-Aware Lexicon Construction<br/>Differential Gradient Attribution & Expansion"]
    B --> C["On-the-Fly Task-Aware Causal Intervention<br/>Dynamic Attribute Substitution Generating Pairs"]
    C --> D["Causal Alignment Fine-Tuning<br/>Joint Task Prediction & Representation Alignment"]
    D --> E["Debiased Downstream Model"]

Key Designs

1. Task-Aware Lexicon Construction: Static Lexicon Gaps -> Differential Gradient Attribution -> Task-Specific Intervention Space
Manual lexicons only cover explicit demographic keywords and fail to account for context-dependent words that transmit bias in specific downstream tasks. ARCA addresses this by defining an intervention lexicon \(\mathcal{A} = \mathcal{A}_{\text{seed}} \cup \mathcal{A}_{\text{discovered}}\), where \(\mathcal{A}_{\text{seed}}\) is a compact anchor set of unambiguous pronouns and nouns (e.g., he/she, father/mother). Using a small probing set \(\mathcal{D}_{\text{probe}}\) (\(M=1000\)) from downstream training data, ARCA computes Gradient \(\times\) Input attribution scores \(a_i(x) = |e_i \odot \frac{\partial \mathcal{L}_{\text{task}}(x)}{\partial e_i}|_2\) on both original inputs and counterfactuals constructed with \(\mathcal{A}_{\text{seed}}\). The differential attribution at each token position measures how heavily the model shifts its reliance on that token when the protected attribute is altered: $$ \Delta a_i(x) = |a_i(x) - a_i(x')| $$ These scores are aggregated across the probing corpus into word-level scores \(\Delta a(w)\). Candidate words exceeding a dynamic threshold \(\delta = \mu_{\Delta a} + k \cdot \sigma_{\Delta a}\) (with minimum frequency \(n_{\min}\)) are retained. For each candidate, ARCA projects pretrained vocabulary embeddings along the attribute direction \(d_{\text{attr}}\) derived from seed pairs, retrieves nearest semantic neighbors with opposite attribute polarity (\(\cos(e_w, e_v) > \tau_{\text{sim}}\) and \(p(w)p(v) < 0\)), and validates them through bidirectional round-trip substitution in sentence templates. This discovers task-relevant carriers such as businessman/businesswoman and contextual evaluative terms like handsome/beautiful.

2. On-the-Fly Task-Aware Causal Intervention: Offline Data Bloat -> Linear Stream Scanning -> Semantically Preserved Counterfactual Pairs
Offline dataset augmentation doubles storage and slows convergence, often introducing grammatical distortion. In Stage 1, ARCA operates directly on mini-batches during fine-tuning with linear time complexity \(\mathcal{O}(n)\). Each tokenized sentence \(x_{\text{orig}}\) is scanned for string matches in the validated causal intervention lexicon \(\mathcal{A}\). If matched terms are found, they are deterministically replaced with their opposite-attribute counterparts while preserving capitalization, yielding a counterfactual variant \(x_{\text{cf}}\). Because counterparts are pre-verified for vocabulary compatibility, surface length parity, and template consistency, the intervention \(do(A = a')\) preserves the ground-truth downstream semantic label \(y\) while altering only the protected attribute path.

3. Causal Alignment Fine-Tuning: Representation Destruction -> Latent Causal Divergence -> Joint Optimization Weakening Shortcut Paths
Hard projections can discard task-critical features that happen to share variance with demographic attributes. Instead of post-hoc projection, ARCA imposes a continuous representation-level alignment constraint directly within the downstream training loop. For all original inputs, standard cross-entropy classification loss \(\mathcal{L}_{\text{task}}\) is computed against true labels. For matched intervention pairs, the [CLS] latent representations \(h_{\text{orig}}, h_{\text{cf}} \in \mathbb{R}^d\) are extracted from the transformer encoder, and the normalized Euclidean divergence is minimized: $$ \mathcal{L}{\text{align}} = \frac{1}{d} |h|_2^2 $$ By penalizing latent discrepancy between counterfactual representations, the model is compelled to map inputs differing only in protected attributes to invariant semantic representations, effectively severing the spurious shortcut pathway }} - h_{\text{cf}\(A \to \hat{Y}\) before signals reach the classification head.

Loss & Training

The overall training objective is a weighted combination of task utility and causal alignment: $$ \mathcal{L}{\text{total}} = \mathcal{L} $$ where }} + \alpha \cdot \mathcal{L}_{\text{align}\(\alpha \ge 0\) controls the strength of the debiasing regularization (default \(\alpha=2.0\)). For samples without attribute terms, \(\mathcal{L}_{\text{align}}\) is inactive (\(0\)). Models are trained using the AdamW optimizer with learning rates searched in \([2 \times 10^{-5}, 5 \times 10^{-5}]\) with a linear warmup-decay schedule over 10% of total steps. The preliminary lexicon extraction in Stage 0 requires only 1 warm-up epoch (\(E_w=1\)) on 1,000 probe instances, incurring less than 5% computational overhead relative to the full fine-tuning routine.

Key Experimental Results

Main Results

ARCA was evaluated across three core GLUE benchmark tasks: MNLI (matched split), SST-2, and MRPC. Conventional intrinsic bias metrics from StereoSet (Stereotype Score SS, Language Modeling LM Score, and Idealized CAT ICAT) were monitored as reference baselines. Task evaluation incorporates Classification Accuracy (Acc), Prediction Flip Rate (PFR, percentage of label shifts under attribute substitution), and Probability Divergence Score (PDS, mean absolute probability shift across classes).

Backbone Method StereoSet SS (Gender) \(\to\) 50 StereoSet ICAT \(\uparrow\) MNLI Acc (%) \(\uparrow\) MNLI PFR (%) \(\downarrow\) MNLI PDS \(\downarrow\)
BERT-base Original (Standard FT) 60.28 69.02 84.5 4.2 0.085
BERT-base CDA 59.61 69.39 83.9 2.8 0.052
BERT-base INLP 57.25 67.28 84.3 2.3 0.045
BERT-base Sentence-Debias 59.37 69.67 84.4 2.7 0.053
BERT-base TriCon-Fair 55.68 72.05 84.3 1.8 0.035
BERT-base DBR โ€” โ€” 84.2 2.5 0.048
BERT-base ARCA (Ours) โ€” โ€” 84.4 0.9 0.018
RoBERTa-base Original (Standard FT) 59.93 73.16 87.6 3.8 0.072
RoBERTa-base CDA 55.85 68.01 87.1 2.4 0.045
RoBERTa-base INLP 58.05 71.10 87.2 2.0 0.038
RoBERTa-base Sentence-Debias 58.38 75.69 87.4 2.3 0.044
RoBERTa-base TriCon-Fair 56.33 76.11 87.3 1.5 0.029
RoBERTa-base DBR โ€” โ€” 87.3 2.1 0.041
RoBERTa-base ARCA (Ours) โ€” โ€” 87.5 0.7 0.014

On SST-2 (using BERT-base), ARCA maintains competitive accuracy at 92.7% (vs. 92.8% standard fine-tuning) while reducing PFR from 3.5% to 0.3% and PDS from 0.068 to 0.006. On MRPC, ARCA achieves 87.26% accuracyโ€”substantially outperforming standard fine-tuning at 84.55%โ€”while attaining the lowest PDS of 0.0178 and cutting PFR from 3.05% to 2.10%.

Ablation Study

On SST-2 with BERT-base (\(\alpha=2.0\)), the impact of different lexicon construction configurations was systematically isolated:

Lexicon Variant Lexicon Size (Pairs) Acc (%) \(\uparrow\) PFR (%) \(\downarrow\) PDS \(\downarrow\) Observation
S.L. (Seed-Only) \(\sim 8\) 92.6 2.1 0.040 Pronoun-only substitutions fail to neutralize contextual bias carriers
M.L. (Manual High-Coverage) \(\sim 60\) 92.5 0.9 0.017 Extensive human curation; achieves strong debiasing but lacks task specificity
R.L. (Random Lexicon) \(\sim 60\) 92.4 3.3 0.063 Arbitrary word swaps corrupt semantics without debiasing benefit
C.L. (ARCA Discovered) \(\sim 55\) 92.7 0.8 0.015 Attribution-driven expansion yields the best debiasing and highest accuracy

Evaluating the sensitivity of the candidate-filtering threshold scaling factor \(k \in \{1.0, 1.5, 2.0\}\) demonstrated that \(k=1.5\) yields the optimal trade-off (93.12% Acc, 0.00% PFR, 0.0064 PDS). Lowering \(k\) to 1.0 admits noisy candidate pairs (Acc 92.20%, PDS 0.0790), while increasing \(k\) to 2.0 discards informative task-specific bias words.

Key Findings

  • Decoupling Intrinsic and Extrinsic Bias: Superior performance on static intrinsic benchmarks (e.g., TriCon-Fair achieving 55.68 SS and 72.05 ICAT on BERT) fails to guarantee fairness post-fine-tuning, where it still exhibits 1.8% PFR and 0.035 PDS on MNLI.
  • Attribution Discovers Non-Obvious Bias Channels: Differential gradient attribution successfully surfaced implicit bias mediators (e.g., businessman/businesswoman, hero/heroine, handsome/beautiful), enabling ARCA's 55 discovered pairs to outperform large 60-pair manual wordlists.
  • Regularization Window: Varying the weight \(\alpha\) revealed an optimal stability band \(\alpha \in [2.0, 2.5]\). Setting \(\alpha=0\) leaves models vulnerable to shortcut learning; pushing \(\alpha \ge 3.0\) forces over-alignment of distinct semantic structures, causing accuracy degradation and a slight rebound in PFR.

Highlights & Insights

  • Attribution-Guided Causal Discovery: Exploiting differential gradient signals as an automated lens to discover contextual bias words bypasses expensive manual annotation while adapting directly to downstream task distributions.
  • Single-Stage Pipeline Synergy: Unifying counterfactual generation and representation-level divergence into standard task fine-tuning eliminates the vulnerability where fine-tuning overrides prior debiasing interventions.
  • Regularization Dividend in Low-Resource Regimes: On small datasets like MRPC (3,668 instances), causal representation alignment acts as a beneficial manifold consistency regularizer, boosting task accuracy from 84.55% to 87.26%.

Limitations & Future Work

  • Language and Morphological Scope: Validated primarily on English corpora. Grammatical gender systems (such as those in Romance or Slavic languages) and agglutinative morphologies require complex syntactic agreement propagation beyond token-level substitutions.
  • Encoder-Centric Architecture: ARCA relies on the [CLS] token pooling in bidirectional MLMs. Adapting this causal alignment strategy to autoregressive decoder-only LLMs during extended generative trajectories remains non-trivial.
  • Controlled Counterfactual Metric Bounds: PFR and PDS measure local counterfactual robustness to attribute substitution, which does not guarantee comprehensive group-level fairness or mitigation of intersectional biases.
  • vs. CDA (Lu et al., 2020): CDA requires static, offline dataset regeneration that bloats corpus size and training time; ARCA uses an on-the-fly streaming substitution mechanism guided by model-internal attribution.
  • vs. INLP (Ravfogel et al., 2020) & Sentence-Debias (Liang et al., 2020): Linear projection methods assume bias subspace linearity and risk eroding non-linear task features; ARCA applies soft geometric divergence regularization to preserve task-relevant semantics.
  • vs. DBR (Li et al., 2025): While DBR optimizes prediction distribution divergence, ARCA grounds counterfactual generation in attribution-discovered task vocabularies and penalizes latent representation drift directly, achieving markedly lower probability divergence (0.018 vs. 0.048 on MNLI).

Rating

  • Novelty: โญโญโญโญโ˜† Combines differential gradient attribution with causal intervention to automatically construct task-specific debiasing lexicons.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous validation across three GLUE benchmarks, dual backbone models, intrinsic comparisons, ablations, and sensitivity studies.
  • Writing Quality: โญโญโญโญโญ Clear causal formulation, cohesive motivation, and meticulous empirical analysis.
  • Value: โญโญโญโญโ˜† Offers an efficient, highly practical blueprint for deploying robust, bias-mitigated MLMs in enterprise NLU applications.