Skip to content

Distributional Stabilization for Zero-Shot Relation Extraction

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/qinglushengliangye/SCAR
Area: LLM Reasoning
Keywords: Zero-Shot Relation Extraction, Joint Encoding, Distributional Stabilization, Recall Collapse, Confidence Ambiguity

TL;DR

Under a full-vocabulary diagnostic on joint-encoding zero-shot relation extraction, this paper identifies two independent distributional pathologies—recall collapse and confidence ambiguity—and resolves them with SCAR via gradient-isolated Z-score normalized coarse-to-fine cascade augmentation and training-only interactive contrastive alignment.

Background & Motivation

Relation extraction, which identifies semantic relations between entity pairs in unstructured text, underpins critical applications such as automated knowledge graph construction, question answering, and biomedical information extraction. In real-world environments, relation inventories are open-ended and continuously evolving, rendering full enumeration during pre-training or supervised fine-tuning infeasible. Zero-Shot Relation Extraction (ZSRE) addresses this challenge by providing candidate relations as natural language label strings or descriptive verbalizations dynamically at inference time. Recently, the joint-encoding paradigm—exemplified by GLiREL—has emerged as a dominant architecture. By concatenating all candidate relation labels into the input sequence for a single Transformer forward pass, joint encoding bypasses the prohibitive inference latency of traditional cross-encoders and textual entailment models (which require separate forward passes per relation), achieving exceptional throughput exceeding 250–300 sentences per second.

However, standard ZSRE benchmarking protocols rely almost exclusively on small held-out candidate pools of 5 to 15 unseen types, masking severe failure modes that emerge in large-scale deployments. When the authors administer a controlled full-vocabulary diagnostic evaluating against all 113 Wiki-ZSL relation types simultaneously, two independent distributional pathologies surface: (i) Recall collapse, where 35 out of 113 relation types (accounting for 24% of all gold instances) receive exactly zero recall predictions, dragging the mean per-relation recall down to 24.1%, with failures extending beyond unseen types to well-supported training relations such as occupation; and (ii) Confidence ambiguity, wherein predicted probability scores for gold-positive and negative relation candidates overlap drastically (with a mean gap of only 0.30), restricting best-threshold Macro F1 to a meager 18.2% and rendering decision threshold selection extremely brittle.

Investigating the structural causes reveals that recall collapse stems from unnormalized dot-product scoring lacking scale-invariance, as score scales calibrated on small training sets (\(\le 25\) candidate types) fail to generalize to larger candidate sets. Meanwhile, confidence ambiguity arises because per-instance Binary Cross-Entropy (BCE) loss does not constrain the global geometry of the representation space, leaving semantically similar relation types (such as instance of vs. part of) heavily entangled. Naively normalizing raw fine scores or directly swapping BCE for supervised contrastive loss destroys per-instance multi-label decisions and NoRel rejection. Core idea: preserve the joint-encoding backbone and BCE classification pathway completely intact, while attaching two modular, stabilized auxiliary branches—a gradient-isolated coarse-to-fine cascade (CCA) with Z-score normalization to restore cross-relation comparability, and a training-only interactive supervised contrastive learning (ISCL) module to reshape global representation geometry.

Method

Overall Architecture

SCAR is constructed atop the DeBERTa-v3-large backbone, sharing contextual representations between input sentences and candidate labels. For a given text and candidate entity pairs, candidate relations are concatenated as \([REL]\ r_l\) prefixes to the sentence and processed in a single joint self-attention pass, producing entity-pair representations \(H_{pair}\) and label representations \(H_{type}\). The primary pathway computes fine-grained unnormalized dot products \(S_{fine}\) supervised by BCE loss. Concurrently, SCAR deploys two stabilized auxiliary branches: the Coarse-to-Fine Cascade Augmentation (CCA) branch projects representations into a shared retrieval space, calculates cosine similarity with a learnable temperature, standardizes scores via per-sample Z-score normalization along the relation axis, and injects the calibrated relative ranking into the primary score stream via curriculum warmup; the Interactive Supervised Contrastive Learning (ISCL) branch conditions entity pairs on candidate labels via cross-attention and enforces global geometric separation using label-embedding hard-negative weighting, being completely removed at inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input Text & Candidate Labels<br/>[REL] r1 ... [SEP] text"] --> B["DeBERTa-v3-large Backbone<br/>Joint Self-Attention Encoding"]
    B --> C["Extract Entity-Pair H_pair & Label Reps H_type"]
    C --> D["Primary Pathway: Unnormalized Dot Product S_fine<br/>S_fine = H_pair · H_type^T"]
    C --> E["Coarse-to-Fine Cascade Augmentation (CCA)<br/>Lightweight Projections & Scaled Cosine S_coarse"]
    E --> F["Per-Sample Z-Score Normalization<br/>S_norm = clamp((S_coarse - μ) / σ, -6, 6)"]
    F -->|Gradient Detached| G["Auxiliary InfoNCE Loss L_ret Monitoring"]
    D --> H["Curriculum Warmed Additive Fusion<br/>S_final = S_fine + α(t) · S_norm"]
    C --> I["Interactive Supervised Contrastive Learning (ISCL)<br/>Pair-to-Label Cross-Attention Interaction h_pair"]
    I --> J["Label-Embedding Hard-Negative Weighted SupCon L_SC<br/>Active during Training, Removed at Inference"]
    H --> K["Inference Thresholding & NoRel Decision"]

Key Designs

1. Coarse-to-Fine Cascade Augmentation (CCA): Gradient-isolated Z-score ranking to prevent recall collapse

To resolve recall collapse caused by absolute dot-product scale degradation under candidate set expansion, CCA constructs an auxiliary dual-encoder pathway. Entity-pair representations and label representations are projected into a low-dimensional shared retrieval space (\(d_r=128\) or \(256\)) using independent two-layer FFNs with LayerNorm and GELU, and normalized to compute cosine similarity scaled by a learnable temperature \(\gamma\): $\(S^{(b,p,l)}_{coarse} = \left(v^{(b,p)}_{rel} \cdot v^{(b,l)}_{label}\right) \cdot \exp(\gamma)\)$ To convert absolute scores into scale-invariant relative rankings, CCA applies per-sample Z-score normalization along the candidate relation dimension: $\([S^{norm}_{coarse}]_{bpl} = \text{clamp}\left(\frac{S^{(b,p,l)}_{coarse} - \mu_{bp}}{\sigma_{bp} + \epsilon}, -6, 6\right)\)$ The resulting calibrated ranking signal is additively fused into the fine pathway via curriculum schedule \(\alpha(t) = \min(t / T_w, 1) \cdot \alpha_{max}\). Crucially, CCA enforces strict gradient isolation: \(S^{norm}_{coarse}\) is detached before entering the auxiliary retrieval loss \(\mathcal{L}_{ret}\), guaranteeing that \(\mathcal{L}_{ret}\) backpropagates no gradients into the backbone or projection heads. The auxiliary dual encoders are updated exclusively through the main classification loss via fused scores, eliminating the risk of coarse retrieval objectives corrupting fine-grained inner-product alignment.

2. Interactive Supervised Contrastive Learning (ISCL): Label-aware geometric reshaping to eliminate confidence ambiguity

To tackle the confidence ambiguity caused by BCE's inability to penalize confusable negative classes globally, ISCL reshapes the representation space during training. Rather than applying contrastive learning on isolated entity-pair embeddings, ISCL introduces single-head cross-attention from entity pairs to candidate label representations: $\(\tilde{h}_{pair} = \text{LayerNorm}\left(h_{pair} + W_O \text{Softmax}\left(\frac{W_Q h_{pair} \cdot (W_K H_{type})^\top}{\sqrt{d}}\right) W_V H_{type}\right)\)$ Positive pairs are aligned across the batch using global relation label IDs, preventing positive set fragmentation. Furthermore, hard-negative weights are derived directly from the cosine similarity of static label embeddings: $\(w_{ia} = \begin{cases} 1 + \beta \max(0, z_{lab}^{y_i} \cdot z_{lab}^{y_a}), & a \in \mathcal{N}(i) \\ 1, & \text{else} \end{cases}\)$ This explicitly magnifies the repulsive gradient against semantically close relation types (e.g., distinguishing instance of from part of). Governed by curriculum warmup schedule \(\lambda_{sup}(t)\) to protect early optimization stability, ISCL is entirely discarded at inference time, achieving substantial gold-negative score separation with zero inference parameter or latency overhead.

Loss & Training

The complete SCAR training objective integrates three complementary terms: the primary asymmetric-weighted binary cross-entropy loss \(\mathcal{L}_{rel}\) (\(w_+=2.0\), \(w_-=1.0\)), the auxiliary InfoNCE retrieval loss \(\mathcal{L}_{ret}\) weighted by \(\lambda_{ret}\) (\(0.01\) or \(0.05\)), and the curriculum-scheduled interactive supervised contrastive loss \(\mathcal{L}_{SC}\): $\(\mathcal{L}_{total} = \mathcal{L}_{rel} + \lambda_{ret}\mathcal{L}_{ret} + \lambda_{sup}(t)\mathcal{L}_{SC}\)$ where \(\lambda_{sup}(t) = \min(t/T_w^{sup}, 1) \cdot \lambda_{sup}^{max}\) warms up smoothly to \(0.05\) over \(1,500\) steps. Optimization utilizes AdamW with cosine decay and early stopping (patience 12 on validation Macro F1). On Wiki-ZSL, lower learning rates are employed (\(2\times 10^{-6}\) for the encoder and \(2\times 10^{-5}\) for auxiliary heads) to stabilize multi-branch interaction, while on FewRel both the baseline and SCAR share standard rates (\(1\times 10^{-5}\) and \(1\times 10^{-4}\)).

Key Experimental Results

Main Results

Evaluations on the Wiki-ZSL and FewRel benchmarks under varying held-out candidate sizes \(m \in \{5, 10, 15\}\) are summarized below. All metrics represent the mean and standard deviation across 5 independent random relation splits.

Dataset Unseen Pool \(m\) Model Macro Precision (%) Macro Recall (%) Macro F1 (%)
Wiki-ZSL \(m=5\) Baseline (GLiREL reproduction) 90.68 82.63 85.60 ± 7.13
CCA (ours) 96.93 92.65 94.41 ± 2.95
SCAR (ours) 96.04 92.31 93.95 ± 1.82
\(m=10\) Baseline (GLiREL reproduction) 84.34 79.17 80.14 ± 2.25
CCA (ours) 92.27 86.73 88.88 ± 3.67
SCAR (ours) 93.71 87.81 89.87 ± 2.11
\(m=15\) Baseline (GLiREL reproduction) 83.79 74.47 75.88 ± 5.11
CCA (ours) 86.86 81.13 82.40 ± 4.39
SCAR (ours) 85.84 81.67 83.04 ± 4.10
FewRel \(m=5\) Baseline (GLiREL reproduction) 93.82 84.78 86.89 ± 7.27
CCA (ours) 95.71 93.67 94.48 ± 3.55
SCAR (ours) 97.03 95.14 96.02 ± 2.14
\(m=10\) Baseline (GLiREL reproduction) 88.90 84.84 86.10 ± 2.18
CCA (ours) 91.89 88.54 89.52 ± 3.25
SCAR (ours) 91.94 88.89 89.73 ± 2.49
\(m=15\) Baseline (GLiREL reproduction) 84.66 80.32 80.50 ± 6.32
CCA (ours) 90.41 86.55 87.39 ± 3.55
SCAR (ours) 88.68 86.06 86.57 ± 2.53

Note: In the challenging \(m=15\) setting, SCAR outperforms the strong prior system TMC-BERT by +9.3 pp (Wiki-ZSL) and +5.6 pp (FewRel), and outperforms GPT-4o by +41.5 pp (Wiki-ZSL) and +15.9 pp (FewRel).

Ablation Study

Paired ablation differences (\(\Delta = \text{ablated} - \text{full model}\)) across 3 independent runs at \(m=15\) on both benchmarks. More negative values indicate greater indispensability.

Ablation Configuration Wiki-ZSL Mean \(\Delta\) F1 (%) Wiki-ZSL Worst \(\Delta\) F1 (%) FewRel Mean \(\Delta\) F1 (%) FewRel Worst \(\Delta\) F1 (%)
Full SCAR Model 0.00 0.00 0.00 0.00
CCA Components
w/o Z-score normalization -2.82 ± 1.15 -3.65 -8.73 ± 5.65 -14.53
w/o gradient isolation -13.23 ± 9.27 -23.26 -5.81 ± 5.08 -11.42
ISCL Components (atop CCA)
w/o cross-attention -6.15 ± 3.68 -10.16 -9.46 ± 8.67 -15.11
w/o global label alignment -11.73 ± 8.17 -19.35 -8.61 ± 7.08 -15.37
w/o curriculum warmup -11.51 ± 11.96 -20.35 -5.28 ± 6.32 -10.52

Key Findings

  • Stabilization guardrails are critical: Removing gradient isolation causes performance to crash by up to -23.26 pp on Wiki-ZSL; eliminating contrastive curriculum warmup leads to a -20.35 pp drop on the worst split. These catastrophic degradations prove that unconstrained multi-task coupling destabilizes the joint-encoder backbone, whereas the proposed isolation and warmup guardrails are essential.
  • Strict independence between pathologies: The full-vocabulary diagnostic reveals that CCA alone reduces zero-recall relation types from 35 to 22, yet leaves best-threshold Macro F1 unchanged at 18.2%. Confidence ambiguity is resolved only when ISCL is engaged, widening the gold-negative score margin from 0.296 to 0.560 (+89%) and propelling best-threshold Macro F1 to 42.90%.
  • Single-pair vs. multi-pair corpus behavior: Wiki-ZSL sentences contain an average of ~5 candidate entity pairs (~120 contrastive anchors per batch), enabling ISCL to deliver robust geometric improvements. FewRel contains only 1 pair per sentence (~24 anchors per batch); here CCA alone suffices, matching full SCAR within statistical variation.

Highlights & Insights

  • Full-vocabulary diagnostic exposes small-pool evaluation illusions: Challenging models against all 113 ontology classes simultaneously unmasks profound structural failures (recall collapse on common relations and confidence ambiguity) that standard 5–15 candidate pool evaluations completely conceal.
  • Zero-cost inference design via training-only geometric reshaping: ISCL operates solely during training to align representation manifolds and is discarded at test time; CCA introduces only lightweight projections (~0.1% parameter increase). Consequently, inference throughput remains above 250–285 sent/s on a single RTX 4090 D.
  • Scale invariance via relative rank projection: By combining learnable-temperature cosine similarity with cross-relation Z-score standardization, CCA transforms unnormalized dot products into calibrated per-sample rankings without disturbing independent multi-label classification logic.

Limitations & Future Work

  • Intra-family recall regressions on dominant relations: In the full-vocabulary evaluation, 12 of 113 relation types experience recall regression (e.g., drafted by dropping to 0%). Detailed error analysis reveals that these relations previously monopolized recall within shared entity-type signature families; SCAR restores the recall of previously suppressed sibling relations, inevitably redistributing recall within the family. Designing adaptive relation-specific weighting or type-conditional losses represents an important future direction.
  • Evaluation restricted to sentence-level English benchmarks: Empirical validation is currently confined to English sentence-level benchmarks (Wiki-ZSL and FewRel) built upon DeBERTa-v3-large. Evaluating SCAR across document-level relation extraction, multilingual environments, and diverse foundation backbones remains to be investigated.
  • vs. GLiREL (Boylan et al., 2025): GLiREL establishes the fast joint-encoding ZSRE paradigm but relies on unnormalized dot products and unconstrained BCE, leading to 35 zero-recall classes under full-vocabulary evaluation. SCAR preserves its single-pass speed while cutting zero-recall classes from 35 to 1 and more than doubling full-vocabulary Macro F1.
  • vs. Dual-Encoders & Cross-Encoders (Chen and Li, 2021; Lan et al., 2023): Dual-encoders lack deep token-level cross-attention between entity context and candidate relations, while multiple-choice cross-encoders suffer from severe latency (1.4–27.8 sent/s). SCAR achieves the best of both worlds—deep joint interaction with high industrial throughput (254–285 sent/s).
  • vs. LLMs & Prompting (GPT-4o / Li et al., 2025): Prompted LLMs exhibit a severe precision-recall skew on Wiki-ZSL (81% precision vs. 32% recall) alongside substantial latency and cost. SCAR surpasses GPT-4o by over +41 pp Macro F1 at \(m=15\) while maintaining high efficiency and balanced recall.

Rating

  • Novelty: ⭐⭐⭐⭐ [Identifies two paradigm-level distributional pitfalls in joint-encoding ZSRE and designs complementary, stabilized auxiliary mechanisms]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated over 30 independent runs, comprehensive full-vocabulary diagnostics, three-way leakage-free splits, and detailed error profiling]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear technical exposition, insightful diagnosis, and rigorous ablation of every stabilizing component]
  • Value: ⭐⭐⭐⭐⭐ [Provides actionable engineering solutions and deployment guidelines for high-throughput relation extraction in open-world knowledge engineering]