REPAIR: Resolving Long-Tail Confusion in Scientific Retrievers via Fact-Verified Iterative Refinement¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/yerimoh/REPAIR
Area: Information Retrieval & RAG
Keywords: Scientific Information Retrieval, Long-Tail Concept Confusion, Fact-Verified Data Augmentation, Hard Negative Mining, Iterative Contrastive Self-Evolution
TL;DR¶
Addressing structural hallucinations and catastrophic semantic collapse caused by heavy-tailed entities and extreme fact-sensitivity in scientific corpora, REPAIR introduces a three-stage self-evolving framework—diagnosing low-margin confused queries, expanding fact-verified evidence via external scientific APIs, and differentiating model-aware hard negatives—outperforming 19 competitive baselines across nine materials science and biomedical benchmarks using only 4M training pairs.
Background & Motivation¶
In specialized domains such as materials science and biomedicine, the relentless influx of new scientific literature positions efficient and reliable knowledge discovery as an existential challenge. Retrieval-augmented generation (RAG) relies fundamentally on underlying dense retrievers to dynamically ground generation in up-to-date, factual domain contexts, thereby powering mission-critical applications including biomedical question answering, molecular property discovery, and clinical decision support. However, while state-of-the-art dense retrievers demonstrate remarkable efficacy across general-domain text, they suffer catastrophic performance degradation when transferred to specialized scientific corpora—a failure mode frequently obscured by conventional domain-adaptation paradigms.
Scientific literature retrieval is intrinsically constrained by two structural characteristics: an extreme long-tailed concept distribution and heightened fact-sensitivity. Empirical statistical analyses reveal that scientific terminology (chemical formulas, rare crystalline lattices, specific gene and protein variants) exhibits a pronounced heavy-tailed Zipfian distribution, where over 62.4% of scientific concepts occur exactly once. Unlike general vocabulary, these domain entities possess no interchangeable semantic substitutes. Crucially, scientific validity is hypersensitive to minute textual perturbations: an unconstrained LLM synthesizing training data frequently generates plausible yet fatal structural hallucinations—swapping biological counter-actors such as BCL1 and BCL2, reversing clinical cause-and-effect conclusions, or substituting incompatible chemical functional groups. Such spurious supervisory signals contaminate retriever representations.
Prior methodologies attempted to remedy data scarcity by indiscriminately scaling LLM-generated synthetic data or applying parameter-heavy reranker distillations, yet internal resampling fails to synthesize concepts outside the training distribution while generative hallucinations inject destructive noise. Core idea: instead of unconstrained generative scaling, build a self-evolving closed loop that diagnoses model-specific low-margin confused queries, grounds long-tail concepts via external authoritative scientific knowledge base APIs to formulate verified queries, and refines discriminative boundaries using model-aware fact-contrastive hard negatives.
Method¶
Overall Architecture¶
REPAIR (Retriever via Epistemic API-Guided Iterative Refinement) initializes a dense retriever on compact, high-density scientific seed corpora \(T_0\), and subsequently executes an iterative, data-centric self-evolution procedure across three interconnected stages: Stage I (Diagnosis) inspects the retriever's internal scoring margins to identify vulnerable long-tail concepts driving retrieval failures; Stage II (Expansion) anchors these confusing concepts to verifiable external facts via domain APIs (Semantic Scholar, PubChem, Materials Project), mining genuine scientific literature to construct factual query-document pairs; Stage III (Differentiation) filters out false negatives and extracts the single most challenging, model-aware hard negative to sharpen the decision boundary. The retriever is updated via an InfoNCE contrastive objective, which reshapes the margin landscape and initializes the subsequent diagnosis cycle.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Seed Scientific Corpora T0 Initialization"] --> B["Self-Diagnosed Long-Tail Concept Confusion"]
B --> C["Verifiable Evidence Augmentation"]
C --> D["Hard Negative Refinement"]
D --> E["Iterative Contrastive Optimization"]
E -->|Margin Landscape Reshaped / Next Iteration Loop| B
E --> F["Deploy High-Precision Scientific Retriever"]
Key Designs¶
1. Self-Diagnosed Long-Tail Concept Confusion: Low-margin query selection and dual-track distractor mining Traditional active learning frameworks sample queries based purely on loss or global uncertainty, failing to pinpoint the specific semantic distractors corrupting scientific representations. REPAIR diagnoses failure points by calculating the positive-negative separation margin for each training query: \(\Delta_\theta(q) = s_\theta(q, d^+(q)) - \max_{d \in \text{TopK}_\theta(q) \setminus \{d^+(q)\}} s_\theta(q, d)\), isolating the bottom \(p = 40\%\) queries to construct the confusion set \(\mathcal{Q}_{conf}\). Candidate entities are extracted using domain tools (MATDETECTOR and CHEMDATAEXTRACTOR). To isolate systemic distractors, REPAIR executes dual-track mining: Intra-Query mining scores each concept via the Confusing Concept Score \(CCS(e) = \frac{df(e; \mathcal{D}^-)}{df(e; \mathcal{D}^+) + \epsilon}\), capturing entities that appear frequently in top-ranked false negatives but remain absent in the true positive; Inter-Query mining employs parameter-free FINCH clustering on query embeddings to group shared confusion patterns and extract top recurring concepts. The union yields the targeted distractor set \(\mathcal{C}_{conf} = \mathcal{C}_{intra} \cup \mathcal{C}_{inter}\).
2. Verifiable Evidence Augmentation: API-grounded metadata retrieval and zero-hallucination query construction Because long-tail concepts lack internal semantic support within generative priors, purely generative synthesis inevitably hallucinates plausible falsehoods. In this stage, REPAIR grounds every concept in \(\mathcal{C}_{conf}\) by querying authoritative external databases (PUBCHEM, Materials Project) to retrieve grounded chemical and physical attributes (such as molecular formulas, space groups, and valid synonyms). Conditioned on 1 to 3 grounded concepts, an LLM generates a candidate query \(q_{model}\) engineered to be semantically ambiguous or demanding. This candidate is immediately dispatched to scientific APIs (Semantic Scholar, PubChem, Materials Project). If no literature matches, the hallucinated query is discarded; if verified, the authentic paper title defines the ground-truth query \(q_{new}\), the actual paper body becomes positive document \(d^+\), and the remaining top-ranked API results constitute the candidate hard negative pool \(\mathcal{D}^-_{cand}\), entirely bypassing generative fabrication.
3. Hard Negative Refinement: Consistency filtering and model-aware boundary sharpening Naive inclusion of hard negatives often risks false negatives or out-of-domain semantic drift. REPAIR applies consistency filtering over the local pool \(\mathcal{D}_{pool} = \{d^+\} \cup \mathcal{D}^-_{cand}\), retaining a tuple only if the current retriever ranks the true positive \(d^+\) within the top \(\kappa = 2\) positions. This guarantees that the generated query is empirically answerable and not corrupted by semantic drift. From the validated pool, REPAIR selects the single hardest negative that maximizes the current retriever's dot-product score: \(d^- = \arg\max_{d \in \mathcal{D}^-_{cand}} s_\theta(q_{new}, d)\). Selecting this single factually plausible yet model-confusing negative establishes a substantially sharper, more robust scientific decision boundary than flooding the contrastive pool with undifferentiated negatives.
Loss & Training¶
The parameters \(\theta\) of the decoder-only backbone (Qwen2.5-0.5B, 1.5B, 7B) are optimized by minimizing the InfoNCE objective over verified triplets \((q_{new}, d^+, d^-)\): $$ \mathcal{L}(q_{new}) = -\log \frac{\exp(s_\theta(q_{new}, d^+) / \tau)}{\exp(s_\theta(q_{new}, d^+) / \tau) + \sum_{d \in \mathcal{N}} \exp(s_\theta(q_{new}, d) / \tau)} $$ where \(\tau\) is the temperature hyperparameter and \(\mathcal{N}\) comprises the mined hard negative \(d^-\), in-batch negatives, and cross-device negatives synchronized across Distributed Data Parallel (DDP). Backbones are fine-tuned using LoRA (\(r=16, \alpha=32\), dropout \(0.05\)) on two NVIDIA H200 GPUs in bfloat16 precision with gradient checkpointing. All experiments default to two refinement iterations, dynamically updating \(\Delta_\theta(q)\) and achieving rapid convergence without overfitting.
Key Experimental Results¶
Main Results¶
REPAIR was evaluated against 19 competitive baselines across four BEIR scientific information retrieval datasets (NFCorpus, SciFact, SciDocs, TREC-COVID) and one sentence similarity benchmark (BIOSSES), as well as downstream materials and biomedical tasks.
| Model | Scale | #Pairs | NFCorpus (nDCG@10) | SciFact (nDCG@10) | SciDocs (nDCG@10) | COVID (nDCG@10) | BIOSSES (Spearman) | AVG. |
|---|---|---|---|---|---|---|---|---|
| BM25 | - | - | 0.325 | 0.665 | 0.158 | 0.656 | - | 0.451 |
| Contriever | 110M | 1.5B | 0.328 | 0.677 | 0.165 | 0.596 | 0.833 | 0.520 |
| SPECTER 2.0 | 110M | 3.3M | 0.228 | 0.671 | - | 0.584 | - | - |
| MedCPT | 220M | 255M | 0.340 | 0.724 | 0.123 | 0.697 | 0.837 | 0.532 |
| BGE-Large | 895M | 2.8B | 0.345 | 0.723 | 0.222 | 0.753 | 0.804 | 0.560 |
| BMRETRIEVER-410M | 410M | 11.4M | 0.321 | 0.711 | 0.167 | 0.831 | 0.840 | 0.563 |
| REPAIR-500M (Ours) | 500M | 4.0M | 0.376 | 0.680 | 0.196 | 0.812 | 0.853 | 0.583 |
| InstructOR-XL | 1.5B | 1.24M | 0.360 | 0.646 | 0.174 | 0.713 | 0.842 | 0.547 |
| BMRETRIEVER-2B | 2B | 10.0M | 0.351 | 0.760 | 0.199 | 0.863 | 0.828 | 0.600 |
| REPAIR-1.5B (Ours) | 1.5B | 4.0M | 0.376 | 0.757 | 0.201 | 0.853 | 0.849 | 0.607 |
| RepLLaMA | 7B | 500K | 0.378 | 0.756 | 0.181 | 0.847 | - | - |
| LLM2Vec | 7B | 2.7M | 0.393 | 0.788 | 0.225 | 0.776 | 0.852 | 0.606 |
| E5-Mistral | 7B | 1.8M | 0.386 | 0.764 | 0.162 | 0.872 | 0.855 | 0.608 |
| BMRETRIEVER-7B | 7B | 11.4M | 0.364 | 0.778 | 0.201 | 0.861 | 0.847 | 0.610 |
| Promptriever | 7B | 1.0M | 0.376 | 0.760 | 0.176 | 0.835 | 0.861 | 0.602 |
| REPAIR-7B (Ours) | 7B | 4.0M | 0.413 | 0.789 | 0.227 | 0.842 | 0.846 | 0.623 |
In specialized downstream benchmarks (ChemLit-QA materials and biomedical subsets, MeSH entity linking, and RELISH paper recommendation), REPAIR-7B consistently establishes state-of-the-art results: ChemLit-QAmat nDCG@20 reaches 71.5 (vs. 67.2 for BMRETRIEVER-7B), ChemLit-QAbiomed R@5 attains 91.5 (vs. 86.2), MeSH MRR@5 reaches 62.0 (vs. 61.1), and RELISH MAP hits 87.4 (vs. 86.7).
Ablation Study¶
Extensive ablations evaluate iteration depth, selection ratio \(p\), negative analysis depth \(k\), and the impact of fact-verified training data against alternative synthetic corpora on identical backbones.
| Evaluation Dimension | Configuration | NFCorpus | SciFact | SciDocs | COVID | BIOSSES | AVG. | Key Empirical Insight |
|---|---|---|---|---|---|---|---|---|
| Iteration Depth (0.5B) | Iter 0 (Seed Corpus) | 0.321 | 0.623 | 0.173 | 0.751 | 0.781 | 0.530 | Severely constrained by initial long-tail confusion |
| Iter 1 (Initial Refinement) | 0.343 | 0.651 | 0.187 | 0.749 | 0.803 | 0.547 | +3.2% gain, resolves high-frequency distractors | |
| Iter 2 (Default Setting) | 0.376 | 0.680 | 0.196 | 0.812 | 0.853 | 0.583 | +10.0% relative gain; optimal compute-performance trade-off | |
| Iter 3 (Extended Round) | 0.378 | 0.693 | 0.200 | 0.814 | 0.855 | 0.588 | Gains saturate with marginal +0.005 increase | |
| Iter 4 (Deeper Loop) | 0.382 | 0.697 | 0.199 | 0.813 | 0.852 | 0.589 | Diminishing returns (+0.001) at constant compute | |
| Selection Ratio \(p\) (0.5B) | \(p = 10\%\) | 0.321 | 0.630 | 0.180 | 0.730 | 0.788 | 0.530 | Captures only extreme failures; misses broader long-tail |
| \(p = 40\%\) (Default) | 0.343 | 0.651 | 0.187 | 0.749 | 0.803 | 0.547 | Captures 422,183 unique concepts before diminishing returns | |
| \(p = 70\%\) | 0.342 | 0.653 | 0.191 | 0.757 | 0.809 | 0.550 | Marginal discovery of novel concepts drops sharply | |
| \(p = 100\%\) (Full) | 0.328 | 0.664 | 0.201 | 0.760 | 0.817 | 0.554 | Injects resolved queries, wasting API calls and compute | |
| Data Quality (Qwen2.5-0.5B) | + BMRetriever Data | 0.297 | 0.678 | 0.181 | 0.793 | 0.841 | 0.558 | Corrupted by unverified generative hallucinations |
| + REPAIR Data | 0.376 | 0.680 | 0.196 | 0.812 | 0.853 | 0.583 | Proves verified data quality outweighs backbone capacity |
Key Findings¶
- Iterative refinement actively expands knowledge frontiers without bias stagnation: Of the confusing concepts diagnosed in Iteration 1, 48.6% are completely resolved after a single refinement step. Furthermore, 83.1% of the concepts identified in Iteration 2 are entirely novel (Jaccard similarity of only 0.146), confirming the model progressively expands its knowledge rather than reinforcing existing biases.
- Critical phase transition in negative mining occurs at \(k=30\): Across six diagnostic metrics (average similarity, marginal similarity, concept drift, hard-negative ratio, similarity spread, and positive-negative gap), \(k=30\) marks the definitive inflection point. Beyond \(k=30\), marginal negative similarity plunges below the positive baseline, and concept drift escalates rapidly, verifying that larger pools dilute the diagnostic focus with trivial out-of-domain noise.
- Empirically verified absence of false negatives: Semantic Scholar citation analysis reveals that the citation overlap between anchor positives and REPAIR-mined hard negatives is \(< 0.00001\%\), comparable to the random pair baseline (\(0.000\%\)) and vastly below established citation links in SciDocs (\(5.80\%\)). This rigorously validates that mined negatives are genuine semantic distractors rather than missed positives.
Highlights & Insights¶
- External API grounding eliminates LLM hallucination modes: Unconstrained LLM generation exhibits four chronic vulnerabilities in scientific domains: numerical entity swapping (BCL1 vs. BCL2), factual direction reversal (decreased vs. increased risk), disease entity context bleeding, and invalid chemical substitution. REPAIR categorically halts these structural failure modes by enforcing external API verification.
- Confusing Concept Score (CCS) provides mathematically grounded diagnosis: By formulating \(CCS(e) = \frac{df(e; \mathcal{D}^-)}{df(e; \mathcal{D}^+) + \epsilon}\), the framework cleanly measures the distribution discrepancy between confusing negatives and ground truth, precisely diagnosing why a query fails without relying on heuristic prompting.
- Exceptional compute and data efficiency: REPAIR-7B achieves state-of-the-art performance with only 4M pairs and ~52 GPU-hours on 2×H200 GPUs, dramatically undercutting the ~1,500 GPU-hours of RepLLaMA-7B and the 2.8B text pairs required by BGE-Large.
Limitations & Future Work¶
- Performance saturation beyond two iterations: Improvement tapers off rapidly after two rounds (average nDCG@10 increases by \(\le +0.005\) in round 3 and \(+0.001\) in round 4). Merely extending the iteration count provides diminishing utility; expanding the breadth of intra-iteration evidence represents a more promising future frontier.
- Dependency on public API coverage and accessibility: Rare or newly emergent concepts absent from Semantic Scholar, PubChem, or Materials Project are discarded rather than approximated. Consequently, the long-tail boundary is ultimately bounded by the richness of external knowledge bases.
- Domain extension requires specific API connectors: While the diagnosis and differentiation stages transfer seamlessly without modification, applying REPAIR to physics, engineering, or legal/finance domains necessitates integrating appropriate external databases (e.g., NASA ADS, USPTO, SEC EDGAR).
Related Work & Insights¶
- vs BMRETRIEVER: BMRETRIEVER relies on unconstrained LLM prompting for biomedical query generation, inadvertently ingesting structural factual errors and requiring complex multi-task prompt tuning. REPAIR employs API-verified grounding and a unified retrieval format, outperforming BMRETRIEVER while using fewer than half the training pairs.
- vs E5-Mistral / BGE-Large: General-domain LLM retrievers leverage web-scale pre-training or expensive reranker distillation, but struggle with fine-grained scientific distinctions due to unweighted long-tail representations. REPAIR demonstrates that data-centric fact verification yields superior specialized representation geometry.
- vs SPECTER 2.0 / SciMult: Static scientific retrievers rely on citation graphs or multi-task contrastive pre-training but cannot dynamically adapt to model-specific blind spots. REPAIR introduces an active, iterative diagnostic feedback loop that dynamically discovers and resolves long-tail confusion.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering data-centric self-evolution framework that replaces noisy generative scaling with verified API grounding and margin-guided diagnosis.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 9 primary benchmarks, 3 cross-domain generalization tasks, 19 strong baselines, and deep mathematical analyses of neighborhood stability.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured, insightful mathematical formulations, clear motivation, and complete transparency regarding error modes.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational standard for training reliable, hallucination-free dense retrievers in high-stakes scientific and biomedical domains.