DecSelfMask: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ferrazzipietro/DecSelfMask
Area: LLM Reasoning
Keywords: Self-Supervised Learning, Relevance Attribution, Masked Reconstruction, Decoder Adaptation, Clinical Text Classification
TL;DR¶
Addressing the severe scarcity of expert annotations in specialized domains, DecSelfMask leverages attention-aware layer-wise relevance propagation (AttnLRP) on task descriptors to locate and mask task-critical text spans under a "two peaks" causality criterion, transforming raw hospital notes into self-supervised training signals that surpass standard continual pretraining by +6.3 F1 and baseline SFT by up to +27.8 F1 across 136 clinical classification tasks.
Background & Motivation¶
In specialized domains such as biomedicine and clinical healthcare, deploying deep natural language processing models faces an acute annotation bottleneck. Hospitals routinely generate millions of unstructured patient records, emergency department encounter summaries, and doctor discharge notes. However, converting these raw observations into gold-standard classification labels demands significant domain expertise from certified physicians. The resulting manual labeling process is prohibitively expensive, time-consuming, and fundamentally difficult to scale. Consequently, modern medical NLP systems must operate in an asymmetric regime characterized by an overwhelming abundance of unannotated raw text alongside a scarce handful of labeled task examples.
To leverage raw data in few-label regimes, existing research predominantly follows two divergent paradigms. The first is semi-supervised self-training, where a downstream model fine-tuned on initial labeled examples generates synthetic pseudo-labels over unlabeled corpora to augment subsequent training. While self-training provides task alignment, it suffers from pervasive confirmation bias: noisy or hallucinated labels on complex clinical narratives compound errors during training. The second approach is continual pretraining (CPT), which adapts generative decoders by exposing them to raw clinical corpora via standard next-token prediction. Although CPT avoids pseudo-labeling noise, it operates entirely agnostic of downstream targets, learning generic textual statistics that frequently yield marginal or even negligible improvements on specific clinical decision tasks.
DecSelfMask resolves this tension by exploring the intersection of task-guided self-learning and self-supervised domain adaptation. In clinical classification, only sparse, specific text spans carry direct diagnostic or status evidence. When conditioned on a target task descriptor, the decoder's internal attention mechanisms naturally attribute varying relevance across the input context. If multiple semantically related evidence spans co-occur within an unannotated document, masking out one span and compelling the model to reconstruct it conditioned on the preceding context turns raw text into a self-supervised training signal tightly aligned with downstream tasks. Core idea: exploit attention-aware layer-wise relevance propagation (AttnLRP) on task descriptions to pinpoint task-relevant text spans in unlabeled corpora, then apply a "two peaks" causal masking criterion so the decoder learns task-relevant semantic structures via autoregressive reconstruction.
Method¶
Overall Architecture¶
The DecSelfMask pipeline consists of four sequential stages: relevance calculation via gradient-based attribution, smoothing and multi-peak span extraction, causal masked sequence generation, and multi-task downstream adaptation. Given a large corpus of unlabeled domain-specific notes and short textual strings defining downstream tasks, the framework appends each task descriptor to raw notes and computes input token relevancy using AttnLRP. A 1D Gaussian filter smoothes token scores before a dual-threshold filter isolates distinct high-relevance evidence spans. Documents satisfying the "two peaks" condition—containing at least two disjoint relevant spans—are converted into training instances by masking a downstream span that is preceded by an intact upstream span. The decoder is trained to reconstruct the masked span using standard next-token prediction, after which it is adapted to target classification tasks via linear probing heads or instruction tuning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw Clinical Notes + Target Task Descriptors"] --> B["AttnLRP Middle-Token Relevance Attribution"]
B --> C["Gaussian-Smoothed Dual-Threshold Peak Extraction"]
C --> D["Two-Peaks Causal Preceding-Context Masking"]
D --> E["Autoregressive Mask Reconstruction Pretraining"]
E --> F["Downstream Multi-Task Probing & Fine-Tuning"]
Key Designs¶
1. AttnLRP Middle-Token Relevance Attribution: efficient extraction of task-salient context from raw text To identify which portions of an unannotated clinical note \(x_i\) carry semantic relevance for a downstream classification task \(T\), the method appends a brief textual task descriptor \(s_T\) (e.g., "asthma", "heart rate") to the note. Standard input attribution techniques such as perturbation removal or contrastive explanations incur prohibitive inference overhead. DecSelfMask instead utilizes gradient-based Attention-aware Layer-wise Relevance Propagation (AttnLRP) to propagate attribution signals backward from the generated tokens to the input tokens. Because \(s_T\) consists of multiple sub-word tokens, computing attributions across all generated tokens would scale computation linearly with sequence length. Through validation experiments on a clinical Named Entity Recognition dataset, the authors found that computing AttnLRP relevance specifically from the middle token of \(s_T\) provides the sharpest localization signal, outperforming first- or last-token attributions. Special model tokens are excluded from relevance calculations to prevent sink-token distortion, yielding an efficient, high-fidelity relevance vector \(\text{rel}_{x_{i,j}|s_T}\) across input tokens.
2. Gaussian-Smoothed Dual-Threshold Peak Extraction: isolating coherent clinical spans without sub-word fragmentation Raw token-level relevance scores frequently exhibit high-frequency variance and fragment across sub-word boundaries. DecSelfMask applies a 1D Gaussian filter (\(G\)) with kernel size 3 and \(\sigma=1\) to smooth the score trajectory. Candidate spans \(r_{x_i|T}(s, e)\) are extracted using an upper threshold \(t_u = 0.4\) and a lower threshold \(t_l = 0.2\). A valid span requires all constituent tokens to maintain smoothed relevance above \(t_l\) (\(\forall j \in [s, e], G(\text{rel}_{x_{i,j}|s_T}) \ge t_l\)), with at least one token exceeding the peak threshold \(t_u\) (\(\exists j \in [s, e], G(\text{rel}_{x_{i,j}|s_T}) \ge t_u\)). To preserve semantic integrity and prevent partial-token slicing, extracted token intervals are expanded to cover the complete words intersecting the boundaries.
3. Two-Peaks Causal Preceding-Context Masking: ensuring adequate inference context while preventing label collapse A fundamental failure mode in masked auto-encoding occurs when masking a singular task-relevant span strips the entire document of contextual clues, forcing the model into blind hallucination. DecSelfMask addresses this via the "two peaks" framework: only documents exhibiting two or more disjoint high-relevance spans \(r_{x_i|T}(s_1, e_1)\) and \(r_{x_i|T}(s_2, e_2)\) are retained. In accordance with the autoregressive causal nature of decoder-only architectures, training sequences are constructed by masking a downstream span only if another unmasked relevant span occurs strictly to its left. This setup forces the model to leverage preceding clinical evidence (e.g., "shortness of breath upon exertion") to reconstruct masked corroborating details (e.g., "bronchodilator administration"), thereby embedding domain dependencies directly into representation space.
4. Probing-Based Representation Decoupling: resolving generation bottlenecks in small decoders When adapting DecSelfMask-trained models to downstream classification tasks, the authors observed an unexpected scaling behavior: while larger models (4B-8B) improved under standard supervised fine-tuning (SFT), smaller decoders (\(\sim\)1B) exhibited an average performance drop of \(-5.4\) Macro F1 under generative SFT. Diagnostic probing—attaching a lightweight classification head (a linear projection to 256 dimensions, ReLU, 0.1 dropout, and an output layer) to the final hidden state—revealed that DecSelfMask successfully injects rich downstream-relevant representations into small models, but their limited parameter capacity impedes generating aligned token labels autoregressively. Adapting via classification head probing yields a +13.2 average Macro F1 gain across all small models, confirming that representation learning can be decoupled from token generation limitations.
Loss & Training¶
The DecSelfMask pretraining objective is standard causal language modeling (next-token prediction) evaluated strictly over the masked span tokens: $$ \mathcal{L}{\text{DSM}} = - \sum \mid x'}^{|r_{x_i|T}|} \log P_{\theta}(r_{x_i|T, ki, r) $$ Pretraining utilizes DeepSpeed ZeRO-2 and FlashAttention-2 with an effective batch size of 128, AdamW optimizer, initial learning rate of \(4 \times 10^{-5}\) with a cosine decay schedule, weight decay of 0.001, and a 10% warmup ratio. Training over 1.03 million extracted clinical sequences required 68 GPU hours across two NVIDIA H200 GPUs for one full epoch. For downstream evaluation across the 136 Case Report Form (CRF) classification tasks, generative SFT uses LoRA (\(r=32\), \(\alpha=64\), learning rate \(3 \times 10^{-4}\)), while probing heads are optimized with AdamW at a learning rate of \(1 \times 10^{-3}\) for 20 epochs.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on the San Giovanni Bosco (SGB) dataset comprising 1.9M raw clinical records from an Italian emergency department, evaluated on 136 distinct Case Report Form (CRF) medical classification tasks (61k annotated note-item pairs, 85-15 train-test split). Performance is reported as Macro F1 with 95% confidence intervals.
| Model | Size | SFT Base | SFT + DSM | Probing Base | Probing + DSM | Max Gain (\(\Delta_{\text{best}}\)) |
|---|---|---|---|---|---|---|
| Llama3-Instruct | 1B | \(21.9 \pm 1.0\) | \(19.69 \pm 1.0\) | \(34.1 \pm 1.1\) | \(47.0 \pm 1.3\) | +25.2 |
| Qwen3-Instruct | 1.7B | \(30.9 \pm 1.2\) | \(22.3 \pm 1.0\) | \(33.3 \pm 1.2\) | \(46.8 \pm 1.3\) | +15.9 |
| Gemma3-Instruct | 4B | \(37.5 \pm 1.2\) | \(44.4 \pm 1.3\) | \(36.1 \pm 1.2\) | \(47.5 \pm 1.3\) | +10.0 |
| Llama3-Instruct | 8B | \(35.3 \pm 1.0\) | \(42.9 \pm 1.2\) | \(55.6 \pm 1.3\) | \(55.8 \pm 1.3\) | +20.5 |
| Qwen3-Instruct | 8B | \(32.5 \pm 1.2\) | \(40.4 \pm 1.2\) | \(52.5 \pm 1.3\) | \(\mathbf{60.3 \pm 1.2}\) | +27.8 |
Comparative baselines spanning bidirectional encoders, generative few-shot prompting, and continual pretraining on the 136 clinical tasks highlight DecSelfMask's consistent superiority:
| Method / Baseline | Architecture Class | Macro F1 | Gap vs. Ours (\(\Delta\)) |
|---|---|---|---|
| ClinicalBERT | Bidirectional Encoder (110M) | \(34.1 \pm 1.2\) | -26.2 |
| multilingual-BERT | Multilingual Encoder (178M) | \(34.2 \pm 1.2\) | -26.1 |
| BERT-cased | General Encoder (110M) | \(36.8 \pm 1.2\) | -23.5 |
| BERT-Italian | In-language Encoder (110M) | \(38.2 \pm 1.3\) | -22.1 |
| ModernBERT | Long-context Modern Encoder | \(44.1 \pm 1.3\) | -16.2 |
| Qwen3-8B (0-shot) | Generative Decoder | \(27.7 \pm 0.6\) | -32.6 |
| Qwen3-8B (2-shot) | Few-shot Decoder | \(30.7 \pm 0.7\) | -29.6 |
| Qwen3-8B + SFT | Supervised LoRA Fine-Tuning | \(32.5 \pm 1.1\) | -27.8 |
| Qwen3-8B + CPT + SFT | Continual Pretraining + SFT | \(33.5 \pm 1.3\) | -26.8 |
| Qwen3-8B + CPT + Probing | Continual Pretraining + Linear Head | \(54.0 \pm 1.4\) | -6.3 |
| Qwen3-8B + Synthetic SFT | Self-Training (Pseudo-Labels) | \(47.8 \pm 1.3\) | -12.5 |
| DecSelfMask-Qwen3-8B + Probing (Ours) | Task-Guided Masking + Head | \(\mathbf{60.3 \pm 1.2}\) | Best Result |
Ablation Study¶
Component ablations on the best-performing Qwen3-8B base dissect the critical contributions of the relevance attribution strategy and the two-peaks criterion:
| Configuration | Adaptation Strategy | Macro F1 | Note |
|---|---|---|---|
| DecSelfMask Full Model | Classification Head (CH) | \(\mathbf{60.3 \pm 1.2}\) | Standard configuration |
| Random Masking (Two Peaks) | Classification Head (CH) | \(56.4 \pm 1.1\) | -3.9 without AttnLRP |
| Random Masking (Two Peaks) | Generative SFT | \(30.4 \pm 1.2\) | Severe generation degradation |
| One Peak Only (Trained 5 Epochs) | Classification Head (CH) | \(47.2 \pm 1.3\) | -13.1 without dual peaks |
| One Peak Only | Generative SFT | \(30.0 \pm 1.2\) | Severe generation degradation |
Additionally, evaluating transferability on two held-out tasks unencountered during DecSelfMask sequence construction (Chronicity and Tloc-Dyspnea) revealed average F1 gains of +3.0 over the base model with classification head, and +18.3 over base model generative SFT, validating cross-task generalizability.
Key Findings¶
- The two peaks criterion is foundational: Training on single-peak instances induces a dramatic 13.1 point drop in Macro F1. Without an unmasked antecedent evidence span in the preceding context, causal infilling lacks necessary guidance, degrading self-supervised signal quality.
- AttnLRP attribution outperforms random masking: Replacing relevance-directed spans with random masking drops performance by 3.9 points, proving that model-derived relevance isolates genuinely informative clinical relations rather than arbitrary vocabulary in-filling.
- Attribution alignment shifts toward factual concepts: Evaluating model attributions against gold-standard annotated clinical spans showed that DecSelfMask increased Qwen3-8B's relevance attribution to true target spans by an average of 37.2%, shifting focus away from generic complaints (e.g., dyspnea) toward specific diagnostic indicators (e.g., pacemaker mentions).
Highlights & Insights¶
- Inverting interpretability into self-supervision: Rather than treating attention attribution purely as an ex-post diagnostic artifact, DecSelfMask transforms AttnLRP into an active, automated supervision signal for unlabeled data curation.
- Bridging continual pretraining and self-training: The framework avoids the unfocused statistical smoothing of standard CPT while sidestepping the hallucinated label cascades characteristic of pseudo-label self-training.
- Unlocking latent representations in small models: Probing experiments expose a stark dichotomy: sub-2B decoders acquire robust domain-task representations under masked pretraining, but fail when coerced into complex generative instruction formatting.
Limitations & Future Work¶
- Domain and linguistic specificity: The empirical evaluation is constrained to Italian emergency department clinical narratives; cross-lingual validation on English corpora (e.g., MIMIC-IV) and non-medical domains remains to be established.
- Attribution compute overhead: Generating AttnLRP maps across millions of prompt-note combinations required ~30 GPU hours on an L40S, presenting an operational bottleneck when scaling to larger general-domain corpora.
- Dependence on concise task descriptors: The method relies on short, semantically dense textual anchors (1-5 tokens) defining the classification tasks, which may not readily extend to fuzzy, complex, or open-ended multi-hop reasoning formulations.
Related Work & Insights¶
- vs. Continual Pretraining (e.g., PMC-LLaMA, MedGemma): CPT ingests uncurated raw text uniformly without task orientation; DecSelfMask introduces task-grounded attribution guidance, outperforming in-domain CPT by +6.3 F1 points.
- vs. Self-Training & Synthetic Labeling (e.g., Zero-shot Self-Training): Conventional self-learning models generate discrete pseudo-labels that risk reinforcement of false positives; DecSelfMask reconstructs ground-truth verbatim tokens from the text itself, eliminating label hallucination.
- vs. Masked Language Modeling (e.g., BERT, ModernBERT, InforMask): Rather than altering the underlying decoder causal attention mask during downstream inference, DecSelfMask injects bidirectional-like understanding during pretraining while preserving the standard autoregressive generative interface.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant, creative concept leveraging internal attribution dynamics to guide self-supervised mask generation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 136 clinical tasks, 5 model sizes, probing studies, and rigorous ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, transparent mathematical definitions, and illuminating attribution visualizations.
- Value: ⭐⭐⭐⭐⭐ Offers a high-impact, practical recipe for adapting generative decoders to data-rich, label-poor enterprise domains.