Distributional Stabilization for Zero-Shot Relation Extraction via Structural Margin Contrast¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qinglushengliangye/SCAR
Area: LLM Agent
Keywords: Zero-Shot Relation Extraction / Joint-Encoding / Coarse-to-Fine Cascade / Interactive Contrastive Learning / Distributional Stabilization
TL;DR¶
Under a full-vocabulary diagnostic, joint-encoding zero-shot relation extraction suffers from unnormalized dot-product recall collapse and instance-level BCE confidence ambiguity; SCAR introduces a gradient-isolated Z-score cascade alongside an inference-removed interactive contrastive branch, boosting Macro F1 by up to +9.7 pp and slashing zero-recall types from 35 to 1.
Background & Motivation¶
Relation extraction (RE) is fundamental to knowledge graph construction, biomedical knowledge discovery, and structured question answering. Real-world relation inventories are inherently large, open-ended, and continuously evolving, which renders the static enumeration and annotation of all relation categories at training time practically impossible. Zero-shot relation extraction (ZSRE) bypasses this bottleneck by treating candidate relation types as natural language descriptions provided dynamically during inference. Among existing paradigms, the joint-encoding framework—exemplified by GLiREL—concatenates candidate relation label tokens with the input sentence and processes them in a single Transformer forward pass. This single-pass paradigm achieves state-of-the-art Macro F1 while maintaining inference throughputs of 250 to 300 sentences per second on modern GPUs, drastically outperforming traditional pairwise cross-encoders and NLI-based formulations that scale linearly with the candidate pool size \(|R|\).
However, standard ZSRE benchmarking conventions obscure critical failure modes by evaluating models on small held-out candidate pools of only 5 to 15 relation types (\(m \in \{5, 10, 15\}\)). When the authors establish a controlled full-vocabulary diagnostic that evaluates against all 113 Wiki-ZSL relation types simultaneously, the baseline exhibits two catastrophic distributional pathologies. First, the model undergoes severe recall collapse: 35 out of 113 relation types (accounting for 24% of all gold-positive test instances) receive zero recall, and 63 types drop below 10% recall, depressing the average per-relation recall to an abysmal 24.1%. Second, the model exhibits pervasive confidence ambiguity: predicted scores for true positive instances and negative pairs heavily overlap (with an average score gap of merely 0.296), causing the best globally searched threshold to yield only 18.2% Macro F1 compared to 75.9% on the smaller \(m=15\) candidate pool.
Investigating the structural causes reveals that these two pathologies stem from distinct, orthogonal mechanisms. Recall collapse is driven by the fact that unnormalized dot-product scores lack scale-invariance: absolute scoring magnitudes learned across small training batches (\(\le 25\) candidate types) fail to generalize to 113 candidate types, systematically drowning out low-magnitude relations. Confidence ambiguity arises because the standard per-instance binary cross-entropy (BCE) objective imposes no global geometric constraints on the representation manifold, leaving semantically similar relation types (such as instance of versus part of) hopelessly entangled. Directly applying Z-score normalization to fine-grained dot-products turns independent binary decisions into zero-sum competitions and breaks no-relation handling; similarly, replacing BCE with standard supervised contrastive learning eliminates per-pair decision thresholds and destroys pair-label inner product alignment. Core idea: keep the fine-grained joint-encoding baseline pathway completely intact and attach two stabilized auxiliary branches—a gradient-isolated coarse-to-fine cascade with Z-score normalization (CCA) to restore cross-relation score comparability, and an interactive supervised contrastive learning module (ISCL) with label-aware hard-negative weighting that reshapes representation geometry during training and is removed at inference.
Method¶
Overall Architecture¶
SCAR operates on the design philosophy of "preserving the baseline inference pathway while attaching defensively stabilized auxiliaries". Built upon a shared DeBERTa-v3-large backbone, input text and concatenated relation labels [REL] r_l are jointly encoded to produce entity-pair representations \(H_{\text{pair}} \in \mathbb{R}^{P \times d}\) and relation label representations \(H_{\text{type}} \in \mathbb{R}^{|R| \times d}\). Alongside the primary unnormalized dot-product scoring matrix \(S_{\text{fine}}\), SCAR attaches two lightweight branches: the Coarse-to-Fine Cascade Augmentation (CCA) branch running parallel to scoring, and the Interactive Supervised Contrastive Learning (ISCL) branch active exclusively during training.
The following flowchart outlines SCAR's data processing stream, gradient-isolation barrier, and inference-time pruning:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Input Sentence + [REL] Candidate Relation Tokens"] --> ENC["Shared Backbone Encoder<br/>DeBERTa-v3-large"]
ENC --> REP["Extract Pair Reps H_pair and Label Reps H_type"]
REP --> BASE["Baseline Fine Scorer<br/>Dot Product S_fine"]
REP --> CCA["Coarse-to-Fine Cascade Augmentation (CCA)<br/>Dual FFN + Cosine Scaling + Sample-wise Z-score Norm"]
REP --> ISCL["Interactive Supervised Contrastive Learning (ISCL)<br/>Pair-to-Label Cross-Attention + Label-Weighted Hard Negatives"]
CCA -->|Gradient Isolation Barrier| AUX_RET["Auxiliary Retrieval Loss L_ret (Detached Gradients)"]
CCA --> FUSE["Dynamic Curriculum Soft Fusion<br/>S_final = S_fine + alpha(t) * S_norm_coarse"]
BASE --> FUSE
ISCL -->|Active in Training Only| LOSS_SC["Structural Contrastive Loss L_SC"]
FUSE --> LOSS_REL["Primary Weighted BCE Loss L_rel"]
FUSE -->|Removed at Inference| OUT["Zero-Shot Relation Predictions (Sigmoid > Threshold t)"]
Key Designs¶
1. Coarse-to-Fine Cascade Augmentation (CCA): Unnormalized dot-product scale mismatch -> Dual-encoder cosine scoring with sample-wise Z-score calibration -> Converts absolute magnitudes into relative rankings and restores collapsed recall
To eliminate recall collapse caused by the scale mismatch between small training candidate pools and large evaluation inventories, CCA introduces a parallel coarse scoring stream that introduces fewer than 0.1% additional backbone parameters. Two independent two-layer FFNs (\(d \to 2d_r \to d_r\), with LayerNorm, GELU, and Dropout of 0.1) project entity-pair representations \(h_{\text{pair}}\) and relation label representations \(h_{\text{type}}^{(l)}\) into a shared \(d_r\)-dimensional space. Using a learnable CLIP-style inverse temperature parameter \(\exp(\gamma)\) clamped to \([1.0, 30.0]\), coarse cosine similarity scores are computed: $\(S^{(b,p,l)}_{\text{coarse}} = \left( \frac{f_{\text{rel}}(h_{\text{pair}})}{\|f_{\text{rel}}(h_{\text{pair}})\|_2} \cdot \frac{f_{\text{label}}(h_{\text{type}}^{(l)})}{\|f_{\text{label}}(h_{\text{type}}^{(l)})\|_2} \right) \cdot \exp(\gamma)\)$ To enforce scale-invariance, CCA standardizes the coarse scores across the candidate relation axis via sample-wise Z-score normalization with symmetrical clipping: $\([S^{\text{norm}}_{\text{coarse}}]_{b,p,l} = \text{clamp}\left( \frac{S^{(b,p,l)}_{\text{coarse}} - \mu_{bp}}{\sigma_{bp} + \epsilon}, -6, 6 \right)\)$ where \(\mu_{bp}\) and \(\sigma_{bp}\) denote the empirical mean and standard deviation across all candidate relation types for entity pair \((b, p)\). Z-score normalization transforms absolute score magnitudes into relative intra-sample rankings within the candidate set. Consequently, a relation whose raw dot-product score would otherwise be submerged below the decision threshold can resurface if it ranks prominently among its competing candidates. The calibrated coarse score is additively injected into the fine pathway via a curriculum-warmed weighting function: \(S_{\text{final}} = S_{\text{fine}} + \alpha(t) S^{\text{norm}}_{\text{coarse}}\), with \(\alpha(t) = \min(t/T_w, 1)\alpha_{\max}\).
2. Interactive Supervised Contrastive Learning (ISCL): Instance-level BCE confidence ambiguity -> Label-aware cross-attention and embedding-weighted hard negatives -> Reshapes representation geometry and widens the gold-to-negative margin
To address confidence ambiguity stemming from BCE's failure to regulate global representation geometry, ISCL establishes a training-only contrastive branch that repels semantically confusable relation types. Unlike standard supervised contrastive learning that directly projects static entity embeddings, ISCL first applies a single-head residual cross-attention mechanism allowing each entity-pair representation to interact with all candidate label tokens in the batch:
$\(A = \text{softmax}\left(\frac{W_Q h_{\text{pair}} \cdot (W_K H_{\text{type}})^\top}{\sqrt{d}}\right), \quad \tilde{h}_{\text{pair}} = \text{LN}\left(h_{\text{pair}} + W_O A W_V H_{\text{type}}\right)\)$
Projecting \(\tilde{h}_{\text{pair}}\) yields an interactive anchor vector \(z_i\). When constructing contrastive pairs, positive instances across the batch are aligned using a global relation ID mapping to prevent fragmentation caused by lexical variations. Crucially, to repel fine-grained confusions (such as misclassifying part of or use as instance of), ISCL computes dynamic negative sample weights \(w_{ia}\) directly from the cosine similarity between natural language relation label embeddings:
$\(w_{ia} = \begin{cases} 1 + \beta \max\left(0, z_{\text{label}}^{(y_i)} \cdot z_{\text{label}}^{(y_a)}\right), & a \in \mathcal{N}(i) \\ 1, & \text{otherwise} \end{cases}\)$
with \(\beta=1.0\). This explicit margin penalty pushes confusable relation types apart on the unit hypersphere, substantially increasing the margin between gold-positive and negative predictions.
3. Defensive Stabilization Architecture (Gradient Isolation & Warmup): Optimization instability from naive multi-task coupling -> Gradient detachment and multi-stage curriculum scheduling -> Prevents representation collapse and preserves baseline performance
Empirical investigations reveal that naive integration of auxiliary dual-encoder and contrastive objectives causes catastrophic training instability, degrading Macro F1 by up to \(-23.3\) pp in the worst-case run. SCAR enforces two strict defensive guard-rails: First, in the coarse-to-fine branch, the auxiliary InfoNCE retrieval loss \(\mathcal{L}_{\text{ret}}\) receives detached inputs, completely blocking gradient back-propagation from \(\mathcal{L}_{\text{ret}}\) into any model parameters. The dual-encoder projection heads are updated exclusively through the main BCE loss via the additive fusion equation. Second, both auxiliary streams employ phased curriculum warmup: the contrastive loss weight is annealed linearly over \(T_w^{\text{sup}}=1500\) steps (\(\lambda_{\text{sup}}(t) = \min(t/T_w^{\text{sup}}, 1)\lambda_{\text{sup}}^{\max}\), with \(\lambda_{\text{sup}}^{\max}=0.05\)), while coarse fusion weight \(\alpha(t)\) warms up over 1,000 to 3,000 steps. Crucially, ISCL is entirely removed at inference time, resulting in zero extra inference parameters and less than 5% latency overhead from CCA.
Loss & Training¶
The overall training objective of SCAR combines three balanced terms: $\(\mathcal{L} = \mathcal{L}_{\text{rel}} + \lambda_{\text{ret}} \mathcal{L}_{\text{ret}} + \lambda_{\text{sup}}(t) \mathcal{L}_{\text{SC}}\)$ 1. Primary classification loss \(\mathcal{L}_{\text{rel}}\): weighted binary cross-entropy computed over \(S_{\text{final}}\) with positive weight \(w_+=2.0\) and negative weight \(w_-=1.0\). 2. Auxiliary retrieval monitoring loss \(\mathcal{L}_{\text{ret}}\): $\(\mathcal{L}_{\text{ret}} = -\frac{1}{|P|} \sum_{(b,p)} \log \frac{\exp(S^{(b,p,y_{bp})}_{\text{cn}})}{\sum_l \exp(S^{(b,p,l)}_{\text{cn}})}\)$ where coarse inputs are detached, with weight \(\lambda_{\text{ret}}=0.01\) on Wiki-ZSL (0.05 on FewRel). 3. Interactive supervised contrastive loss \(\mathcal{L}_{\text{SC}}\): $\(\mathcal{L}_{\text{SC}} = \sum_{i \in \mathcal{I}^+} \frac{-1}{|\mathcal{P}(i)|} \sum_{j \in \mathcal{P}(i)} \log \frac{\exp(z_i \cdot z_j / \tau)}{\sum_{a \ne i} w_{ia} \exp(z_i \cdot z_a / \tau)}\)$ with temperature \(\tau=0.1\). The model is optimized using AdamW with cosine learning rate scheduling. On Wiki-ZSL, the backbone learning rate is set to \(2 \times 10^{-6}\) and auxiliary projection heads to \(2 \times 10^{-5}\) to ensure numerical stability; early stopping patience is set to 12 epochs based on validation Macro F1.
Key Experimental Results¶
Main Results¶
Experiments were conducted on Wiki-ZSL and FewRel across candidate pool sizes \(m \in \{5, 10, 15\}\), repeating each configuration over 5 independent random relation splits (30 independent runs in total). Macro Precision (P), Recall (R), and F1 (mean \(\pm\) standard deviation) are reported below:
| Dataset | Held-out Pool \(m\) | Method | Macro P (%) | Macro R (%) | Macro F1 (%) | Speed (sent/s) |
|---|---|---|---|---|---|---|
| Wiki-ZSL | \(m=5\) | GLiREL* (SOTA baseline) | 89.41 | 80.67 | 83.28 | - |
| Baseline (reproduced) | 90.68 | 82.63 | 85.60 ± 7.1 | - | ||
| CCA (Ours) | 96.93 | 92.65 | 94.41 ± 2.9 | - | ||
| SCAR (Ours) | 96.04 | 92.31 | 93.95 ± 1.8 | - | ||
| \(m=10\) | GLiREL* (SOTA baseline) | 89.87 | 81.56 | 83.67 | 47.60 | |
| Baseline (reproduced) | 84.34 | 79.17 | 80.14 ± 2.3 | 299.8 | ||
| CCA (Ours) | 92.27 | 86.73 | 88.88 ± 3.7 | 289.9 | ||
| SCAR (Ours) | 93.71 | 87.81 | 89.87 ± 2.1 | 285.0 | ||
| \(m=15\) | GLiREL* (SOTA baseline) | 79.44 | 74.81 | 73.91 | - | |
| GPT-4o zero-shot prompt | 81.04 | 32.06 | 41.57 | - | ||
| Baseline (reproduced) | 83.79 | 74.47 | 75.88 ± 5.1 | - | ||
| CCA (Ours) | 86.86 | 81.13 | 82.40 ± 4.4 | - | ||
| SCAR (Ours) | 85.84 | 81.67 | 83.04 ± 4.1 | - | ||
| FewRel | \(m=5\) | Baseline (reproduced) | 93.82 | 84.78 | 86.89 ± 7.3 | - |
| CCA (Ours) | 95.71 | 93.67 | 94.48 ± 3.6 | - | ||
| SCAR (Ours) | 97.03 | 95.14 | 96.02 ± 2.1 | - | ||
| \(m=10\) | Baseline (reproduced) | 88.90 | 84.84 | 86.10 ± 2.2 | 265.5 | |
| CCA (Ours) | 91.89 | 88.54 | 89.52 ± 3.3 | 259.6 | ||
| SCAR (Ours) | 91.94 | 88.89 | 89.73 ± 2.5 | 254.2 | ||
| \(m=15\) | Baseline (reproduced) | 84.66 | 80.32 | 80.50 ± 6.3 | - | |
| CCA (Ours) | 90.41 | 86.55 | 87.39 ± 3.6 | - | ||
| SCAR (Ours) | 88.68 | 86.06 | 86.57 ± 2.5 | - |
(Note: GLiREL* pretrains on synthetic data, whereas our variants train strictly on supervised splits; inference speed was benchmarked on a single NVIDIA GeForce RTX 4090 D with batch size 24.)
Ablation Study & Full-Vocabulary Diagnostic¶
Paired ablation tests were conducted under the challenging \(m=15\) setting across 3 independent runs (\(\Delta \text{Macro F1} = \text{ablated} - \text{full}\), negative values denote performance drops):
| Ablated Component / Variant | Wiki-ZSL \(\Delta\)F1 (mean ± std) | Wiki-ZSL Worst Run Drop | FewRel \(\Delta\)F1 (mean ± std) | FewRel Worst Run Drop |
|---|---|---|---|---|
| CCA Components | ||||
| w/o Z-score normalization | -2.82 ± 1.15 | -3.65 | -8.73 ± 5.65 | -14.53 |
| w/o gradient isolation (\(\mathcal{L}_{\text{ret}}\) backprops) | -13.23 ± 9.27 | -23.26 | -5.81 ± 5.08 | -11.42 |
| ISCL Components (atop CCA) | ||||
| w/o label-aware cross-attention | -6.15 ± 3.68 | -10.16 | -9.46 ± 8.67 | -15.11 |
| w/o in-batch global ID alignment | -11.73 ± 8.17 | -19.35 | -8.61 ± 7.08 | -15.37 |
| w/o curriculum warmup (\(\lambda_{\text{sup}}\) at step 0) | -11.51 ± 11.96 | -20.35 | -5.28 ± 6.32 | -10.52 |
| Diagnostic Metric on Full Wiki-ZSL (113 types, 134K triplets) | Baseline | CCA | SCAR | Mechanism Impact |
|---|---|---|---|---|
| Zero-recall relation types (Recall = 0%) | 35 types (24% gold) | 22 types | 1 type (near zero) | CCA breaks absolute dot-product suppression |
| Types with Recall < 10% | 63 types (49% gold) | - | Substantially reduced | Under-represented long-tail relations recover |
| Mean Gold-Positive score (\(\text{Gold}^+ \mu\)) | 0.392 | 0.462 | 0.652 | Gold distribution shifts strongly upward |
| Gold-to-Negative score gap (\(\mu^+ - \mu^-\)) | 0.296 | 0.370 | 0.560 (+89%) | Distinct separation between positive & negative scores |
| Best-threshold Macro F1 (oracle sweep) | 18.2% | 18.2% | 42.9% (+24.7 pp) | Massive leap in discriminative potential |
| Usable threshold window (\(F_1 > 15\%\) count $ | W | $) | 11 thresholds | 33 thresholds |
Key Findings¶
- Independence of the two distributional pitfalls: Full-vocabulary diagnostics reveal that CCA alone reduces zero-recall types from 35 to 22 (mean recall climbing from 24.1% to 33.9%), but leaves the best-threshold Macro F1 completely unchanged at 18.2%. Conversely, only the addition of ISCL's structural contrastive alignment widens the score margin from 0.296 to 0.560 and boosts best-threshold F1 to 42.9%. Recovering recall does not fix score overlap; the two issues require distinct, tailored remedies.
- Defensive guard-rails are indispensable for convergence: Naively coupling multi-task auxiliary losses to joint-encoding networks is perilous. Omitting gradient isolation allows the coarse retrieval loss to destabilize the shared backbone representation, causing a massive \(-13.23\) pp drop (\(-23.26\) pp in the worst split). Similarly, omitting curriculum warmup exposes the encoder to chaotic initial contrastive gradients, dropping F1 by \(-11.51\) pp.
- Dataset density governs architectural trade-offs: On Wiki-ZSL, where sentences average \(\approx 5\) entity pairs, the effective contrastive batch size reaches \(\approx 120\) anchors, allowing ISCL to maximally refine decision boundaries. On FewRel, where sentences contain exactly 1 entity pair (\(\approx 24\) anchors), CCA alone matches or slightly exceeds the full SCAR model (87.39% vs. 86.57%), providing a practical rule of thumb for real-world deployments.
Highlights & Insights¶
- Label-embedding hard-negative weighting: Utilizing the cosine similarity between natural language relation embeddings as dynamic contrastive penalties provides an elegant, zero-overhead mechanism to separate intrinsically ambiguous relation pairs (such as
instance ofvs.part of) without expensive online negative mining. - Inference-time pruning with zero extra latency: Removing the contrastive branch entirely during evaluation leaves only CCA's light projection layers, preserving the native speed of joint-encoding (285 sent/s) with a negligible 5% throughput cost and 0.1% parameter overhead.
- Exposing the small-pool evaluation illusion: By contrasting small-pool benchmarks (\(m=5\sim 15\)) against full-vocabulary diagnostics (113 types), the paper reveals that high small-pool accuracy can hide catastrophic systemic failures in real-world large-vocabulary deployments.
Limitations & Future Work¶
- Regression on dominant relation types: Although SCAR improves recall on 101 out of 113 relation types, it regresses on 12 types (e.g.,
drafted by,educated at). These regressing relations previously monopolized entity-type signatures under the baseline (e.g., absorbing all athletic affiliation predictions); once SCAR restores the recall of previously suppressed sibling relations (member of sports teamrecovering from 0% to 94.8%), the dominant types naturally surrender part of their share. Designing adaptive loss weighting or type-conditional losses remains an open problem. - Scope limited to English sentence-level extraction: Evaluation was conducted on sentence-level benchmarks (Wiki-ZSL and FewRel). Extending the framework to document-level extraction and multilingual ontologies remains unverified.
- Hyperparameter sensitivity: SCAR introduces auxiliary hyperparameters (cascade weight \(\alpha\), contrastive temperature \(\tau\), curriculum schedules), which may require domain-specific calibration when deploying to new corpora.
Related Work & Insights¶
- vs GLiREL (Boylan et al., 2025): GLiREL established the single-pass joint-encoding paradigm for fast zero-shot relation extraction, but its unnormalized dot-product scoring and standard BCE objective trigger severe recall collapse and score entanglement under full-vocabulary deployment. SCAR maintains GLiREL's throughput while fixing its underlying distributional pathologies.
- vs LLM Prompting & Retrieval (GPT-4o / Li et al., 2025): While LLMs provide conversational flexibility, GPT-4o demonstrates severe precision-recall skew on Wiki-ZSL \(m=15\) (81.04% precision vs. 32.06% recall, yielding only 41.57% F1). SCAR outperforms GPT-4o by +41.5 pp at a fraction of the inference latency.
- vs NLI / QA Reformulations (Sainz et al., 2021; Levy et al., 2017): Textual entailment and question-answering formulations require verbalized templates and multiple forward passes per candidate relation, scaling poorly as candidate sets expand. SCAR proves that single-pass Transformer encoders can match or exceed NLI accuracy when paired with distributional stabilization.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ [Pioneering identification of recall collapse and confidence ambiguity in joint-encoding ZSRE, resolved via non-invasive, defensively stabilized cascade and contrastive branches.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive 30-run statistical validations across multiple pools, paired ablations, full-vocabulary diagnostics, error decomposition, and leakage-free three-way splits.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity and mathematical precision; deeply insightful narrative connecting empirical failure modes directly to structural mechanisms.]
- Value: ⭐⭐⭐⭐⭐ [Paves the way for deploying high-throughput joint-encoding relation extractors into large-scale real-world ontologies with full confidence in recall stability.]