OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cindyzhangxy/OntologyBench
Area: Medical NLP
Keywords: dense retrieval, biomedical ontologies, compositional retrieval, phenotype representations, evaluation benchmark
TL;DR¶
Constructs OntologyBench, a 597k-pair tiered benchmark across concept grounding, relational retrieval, and compositional phenotype constraints, revealing that dense retrievers suffer severe aggregation failures on joint constraints (scoring only 0.313 nDCG@10 vs. 0.739 for an ontology-aware baseline), which downstream late-interaction or LLM rerankers fail to resolve.
Background & Motivation¶
Dense retrieval models map queries and documents into a shared continuous embedding space, supporting flexible semantic matching across diverse natural-language expressions without relying on exact lexical overlap. This flexibility has made neural representations highly attractive for biomedical and clinical applications, where clinical findings and disease phenotypes are frequently documented using non-standardized terminology, colloquial phrasing, and complex synonymy that foil strict symbolic string-matching.
However, clinical decision-making and biomedical reasoning fundamentally depend on satisfying multiple interdependent structured constraints rather than computing independent unstructured semantic similarity. Existing general-purpose benchmarks (such as BEIR and MTEB) and conventional biomedical retrieval testbeds (like TREC-COVID and NFCorpus) primarily measure broad passage-level semantic relevance, offering little diagnostic clarity on whether models can verify joint compatibility across multiple entities. In real-world differential diagnosis, a single isolated symptom is compatible with hundreds of disparate disorders; a retrieval system must evaluate the conjunction of observed clinical features, where matching only an incomplete subset of phenotypes leads to dangerous misdiagnoses.
To systematically investigate this gap, the authors design a controlled evaluation framework grounded in curated biomedical knowledge graphs to decouple retrieval performance across escalating levels of constraint complexity. The core idea is to introduce OntologyBench, a tiered benchmark spanning concept grounding, relational retrieval, and multi-phenotype compositional retrieval, using decoupled two-stage evaluation and rule-based error categorization to diagnose why dense retrieval and reranking configurations fail to recover structured biomedical compatibility.
Method¶
Overall Architecture¶
OntologyBench is built upon authoritative, expert-curated biomedical ontologies to create a controlled evaluation universe: the Monarch Disease Ontology (MONDO, comprising 6,611 canonical disease concepts and synonyms), the Human Phenotype Ontology (HPO, 8,253 clinical phenotype terms with extensive aliases), and the HGNC/NCBI databases (2,345 canonical gene entities). The benchmark contains 471,854 training pairs and 125,744 evaluation pairs partitioned into three distinct tiers: Tier 1 single-hop concept grounding (mapping textual surface aliases of genes, phenotypes, and disorders to canonical definitions); Tier 2 pairwise relational retrieval (evaluating bidirectional Dis↔Phen, Phen→Gene, and Dis→Gene associations); and Tier 3 compositional retrieval (PhenTriplet→Dis, taking a combination of three observed phenotypes from a single disorder to retrieve fully compatible diseases).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Query Input<br/>(Alias / Single-Hop Relation / Phenotype Triplet)"] --> B["Tiered Biomedical Ontology Benchmark Space<br/>(Tier 1 Grounding / Tier 2 Relations / Tier 3 Compositional Constraints)"]
B --> C["Dual-Supervision Contrastive Learning Regimes<br/>(Concept Grounding Baseline vs Unified Multi-Tier Contrastive Fine-Tuning)"]
C --> D["Decoupled Two-Stage Retrieval Evaluation System<br/>(First-Stage Top-50 Recall vs Token-Interaction & Generative LLM Scoring)"]
D --> E["Four-Mode Diagnostic Error Characterization<br/>(Aggregation Failure / Generic Bias / Related Confusion / Semantic Drift)"]
Key Designs¶
1. Tiered Biomedical Ontology Benchmark Space: Decoupling Surface Normalization, Relational Traversal, and Compositional Cardinality Rather than collapsing entity normalization and multi-constraint reasoning into a single score, OntologyBench organizes tasks into three hierarchical tiers. Tier 1 evaluates surface alias-to-definition resolution with one-to-one mapping (AvgPos = 1.00) under moderate alias diversity (polysemy < 1.0%); Tier 2 measures pairwise associations across many-to-many biomedical relationships (Dis→Phen averages 19.81 positives, Phen→Dis averages 16.62); and Tier 3 evaluates compositional retrieval under incomplete observations using phenotype triplets (PhenTriplet→Dis). Empirical ontology analysis demonstrates that single phenotypes correspond to an average of 328.9 compatible diseases, phenotype pairs narrow this to 26.7, and triplets reduce the hypothesis space to an average of 5.0 compatible disorders (2.26 positive targets in the test set). This triplet design establishes a non-trivial regime requiring joint constraint satisfaction without collapsing into near-deterministic single-target lookups observed with 4-5 phenotypes. Strict partition audits ensure zero within-task overlap of directed query-target pairs.
2. Dual-Supervision Contrastive Learning Regimes: Isolating Concept-Grounding Emergence from Unified Multi-Task Optimization To establish whether relational and compositional competence can spontaneously emerge from lexical normalization or requires explicit relational supervision, the benchmark introduces two distinct fine-tuning paradigms. The "Concept Grounding Supervision (marked with \(\dagger\))" regime trains models exclusively on Tier 1 alias-definition pairs, withholding all relational and compositional signals. The "Unified Ontology Supervision (marked with \(\ddagger\))" regime trains models across all tiers jointly using softmax contrastive loss formulations: $\(L(q, d^+) = -\log \frac{\exp(\text{sim}(q, d^+))}{\sum_{d \in B} \exp(\text{sim}(q, d))}\)$ where bi-encoders (BioLORD, Qwen3-Embedding) optimize cosine similarity with Cached Multiple Negatives Ranking Loss (CachedMNRL) or InfoNCE over sequence lengths of 256 tokens, while late-interaction models (ModernColBERT) optimize PyLate CachedContrastive loss via ColBERT MaxSim aggregation. All models are trained for two epochs on an NVIDIA H100 MIG device to guarantee controlled comparisons.
3. Decoupled Two-Stage Retrieval Evaluation System: Distinguishing Candidate Omission from Downstream Reranking Ceilings To address the confounding of candidate recall failures and scoring reordering errors, the authors implement an explicit two-stage evaluation framework on the Tier 3 task. First, the strongest first-stage retriever generates the top-50 candidates from the full disease corpus, establishing that 82.1% of queries contain at least one relevant disease in their candidate set (Hit@50 = 0.821, 2,083/2,537 queries), while 17.9% of queries fail at candidate generation and receive zero credit end-to-end. Next, downstream rerankers are evaluated strictly over these fixed candidates. Evaluated systems include token-level late-interaction models (ModernColBERT across training regimes) and generative LLM candidate scorers (Qwen3-4B-Instruct, Qwen3-4B-Thinking, Qwen3.6-27B, Gemma4-31B, GPT-5.4-mini) using standardized zero-to-hundred integer compatibility prompts that instruct models to evaluate joint textual compatibility without external knowledge.
4. Four-Mode Diagnostic Error Characterization: Categorizing Neural Failure Modes Under Structured Constraints To understand why embedding models fail on multi-phenotype queries where ontology-aware symbolic baselines succeed, a sequential rule-based failure analysis is applied to all Tier 3 queries where the embedding model fails to rank a ground-truth disease at rank 1. Using predicted disease phenotype overlap (\(Ov_{\text{pred}}\)), ground-truth disease rank under the symbolic Phenomizer reference (\(r_{\text{Phenomizer}}\)), and mean query phenotype information content (\(IC_q\)), failures are partitioned into four mutually exclusive categories: - Aggregation Failure (58.5%): Occurs when the symbolic baseline ranks the true disease within the top 10 (\(r_{\text{Phenomizer}} \le 10\)) but the embedding model retrieves a disease matching at most one query phenotype (\(Ov_{\text{pred}} \le 1\)), highlighting the model's tendency to match phenotypes independently rather than enforcing joint intersection; - Generic Phenotype Bias (25.0%): Triggered when query phenotype information content falls into the bottom quartile (\(IC_q \le 5.949\)), where non-specific broad symptoms dominate the embedding calculation; - Related-Disease Confusion (11.8%): Near-misses where the retrieved disease matches two of the three phenotypes and ranks near the top symbolically (\(r_{\text{Phenomizer}} \le 10\)); - Semantic Drift (4.7%): Unrelated predictions showing poor phenotype overlap and failure under the symbolic baseline (\(r_{\text{Phenomizer}} > 10\)).
Key Experimental Results¶
Main Results¶
Retrieval performance across all eight tasks is evaluated using nDCG@10 across lexical, dense bi-encoder, and specialized biomedical embedding models under varied supervision regimes.
| Model Category / Model | Gene→Def (T1) | Phen→Def (T1) | Dis→Def (T1) | Dis→Phen (T2) | Phen→Dis (T2) | Phen→Gene (T2) | Dis→Gene (T2) | PhenTriplet→Dis (T3) |
|---|---|---|---|---|---|---|---|---|
| Ontology-Aware References | ||||||||
| Phenomizer-style (HPO) | — | — | — | 0.482 | 0.788 | — | — | 0.739 |
| Graph Traversal (Ontology Graph) | — | — | — | — | — | 0.694 | — | — |
| Lexical & Pretrained Baselines | ||||||||
| BM25 | 0.087 | 0.381 | 0.275 | 0.070 | 0.103 | 0.071 | 0.054 | 0.107 |
| all-MiniLM-L6-v2 | 0.263 | 0.629 | 0.314 | 0.106 | 0.153 | 0.109 | 0.058 | 0.123 |
| MedEmbed-0.1B | 0.440 | 0.738 | 0.454 | 0.130 | 0.169 | 0.134 | 0.068 | 0.143 |
| SapBERT-0.1B | 0.670 | 0.818 | 0.476 | 0.088 | 0.159 | 0.113 | 0.069 | 0.131 |
| BioLORD-0.1B | 0.290 | 0.807 | 0.426 | 0.091 | 0.178 | 0.105 | 0.058 | 0.151 |
| Qwen3-Embed-0.6B | 0.325 | 0.605 | 0.247 | 0.089 | 0.151 | 0.106 | 0.063 | 0.109 |
| Qwen3-Embed-4B | 0.526 | 0.616 | 0.330 | 0.119 | 0.188 | 0.129 | 0.073 | 0.080 |
| OpenAI text-embedding-3-large | 0.747 | 0.860 | 0.516 | 0.116 | 0.195 | 0.136 | 0.069 | 0.169 |
| Fine-Tuned Configurations | ||||||||
| BioLORD-0.1B† (Tier 1 only) | 0.513 | 0.869 | 0.562 | 0.112 | 0.175 | 0.091 | 0.048 | 0.171 |
| BioLORD-0.1B‡ (Unified multi-tier) | 0.455 | 0.668 | 0.532 | 0.160 | 0.166 | 0.191 | 0.089 | 0.244 |
| Qwen3-Embed-0.6B† (Tier 1 only) | 0.615 | 0.864 | 0.538 | 0.112 | 0.158 | 0.076 | 0.045 | 0.158 |
| Qwen3-Embed-0.6B‡ (Unified multi-tier) | 0.499 | 0.572 | 0.488 | 0.184 | 0.283 | 0.269 | 0.110 | 0.313 |
Ablation & Two-Stage Reranking Results¶
Evaluation of end-to-end performance on the Tier 3 PhenTriplet→Dis task (N = 2,537 complete query set), examining whether late-interaction or generative candidate scoring resolves first-stage retrieval limitations.
| Stage / System Configuration | MRR@10 | Hit@10 | nDCG@10 | Key Properties & Findings |
|---|---|---|---|---|
| First-Stage Retriever | ||||
| Qwen3-Embed-0.6B‡ (Full retriever) | 0.297 | 0.572 | 0.313 | Provides fixed top-50 pool (Hit@50 = 0.821) |
| Token-Interaction Rerankers | ||||
| ModernColBERT (Pretrained) | 0.168 | 0.384 | 0.180 | MaxSim scoring suffers token dispersion |
| ModernColBERT† (Tier 1 fine-tuned) | 0.168 | 0.386 | 0.183 | Lexical grounding yields no reranking gain |
| ModernColBERT‡ (Unified fine-tuned) | 0.135 | 0.384 | 0.163 | Unified contrastive tuning degrades token matching |
| LLM Candidate Scoring (0-100) | ||||
| Qwen3-4B-Instruct-2507 | 0.262 | 0.525 | 0.275 | Instruction tuning lacks structured multi-hop precision |
| Qwen3-4B-Thinking-2507 | 0.278 | 0.546 | 0.288 | Reasoning trace slightly improves compatibility calibration |
| Qwen3.6-27B | 0.306 | 0.571 | 0.315 | Marginal +0.009 MRR gain; Hit@10 slightly drops |
| Gemma4-31B | 0.284 | 0.524 | 0.287 | Overly conservative scores fail to distinguish candidates |
| OpenAI GPT-5.4-mini | 0.301 | 0.574 | 0.312 | Peak Hit@10 (+0.002) but fails to surpass first stage |
Key Findings¶
- Persistent Gap on Compositional Constraints: Dense retrieval models perform competitively on Tier 1 concept grounding (e.g., OpenAI text-embedding-3-large achieves 0.860 nDCG@10 on Phen→Def), but drop sharply on Tier 3 compositional phenotype retrieval (best zero-shot model reaches only 0.169, and best fine-tuned model reaches 0.313). In stark contrast, the ontology-aware Phenomizer baseline achieves 0.739, exposing a 57.6% performance gap.
- Model Scale Does Not Mitigate Compositional Degradation: Scaling pretrained embeddings from 0.6B to 4B parameters (Qwen3-Embed-4B) actually reduces Tier 3 performance from 0.109 to 0.080. Parameter scaling without structural inductive bias fits surface lexical co-occurrences rather than learning boolean constraint conjunctions.
- Rerankers Fail to Provide End-to-End Improvements: When evaluated across the complete query set where 17.9% of queries lack relevant candidates in the top-50 pool, neither late-interaction models (ModernColBERT nDCG@10 = 0.163–0.183) nor 27B-31B LLM scorers (Qwen3.6-27B nDCG@10 = 0.315 vs. retriever 0.313) deliver meaningful gains over the initial dense retriever.
- Aggregation Failure Is the Dominant Error Mode: Diagnostic categorization reveals that 58.5% of Tier 3 retrieval failures stem from aggregation failure, where models retrieve disorders that match only 0 or 1 query phenotype while completely missing the third, despite full phenotype coverage by the ground-truth disease.
Highlights & Insights¶
- Grounding IR Benchmarks in Curated Graph Ontologies: Elegantly bridges knowledge representation and neural retrieval by translating MONDO, HPO, and HGNC graph relationships into a multi-tiered information retrieval testbed with formal constraint guarantees.
- Rigorous Data Integrity Audits: Guarantees zero leakage of exact directed query-target pairs within tasks, while transparently quantifying transductive multi-task concept recurrence across tiers.
- Debunking Reranking Complacency in Constrained Retrieval: Provides empirical proof that downstream cross-encoders or LLM candidate scorers cannot rescue severe candidate omission or replace structured topological representation learning in first-stage retrieval.
Limitations & Future Work¶
- Simplification of Clinical Reality: Restricts compositional queries to fixed triplets, omitting temporal disease staging, age/sex demographic conditioning, and vital negative findings (ruled-out phenotypes).
- Transductive Concept Space: Evaluates combinations within a shared closed-vocabulary ontology, leaving inductive zero-shot generalization to unmapped novel diseases or newly discovered genetic variants unmeasured.
- Absence of Neuro-Symbolic Model Exploration: While diagnosing the failure of pure embedding and reranking paradigms, the paper does not propose or evaluate hybrid retrieval architectures combining dense vectors with explicit graph index traversal.
Related Work & Insights¶
- vs BEIR / MTEB: BEIR and MTEB primarily measure unstructured paragraph semantic similarity; OntologyBench isolates boolean multi-constraint satisfaction across controlled ontology tiers.
- vs QIME (Tang et al., 2026): QIME synthesizes ontology-grounded questions to enhance embedding interpretability; OntologyBench rigorously tests whether embedding geometry inherently preserves relational and compositional compatibility.
- vs Phenomizer (Köhler et al., 2009): Phenomizer relies on exact entity recognition followed by symbolic information-content calculation; OntologyBench demonstrates that neural models must incorporate symbolic topological priors to overcome greedy partial-match traps.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the first rigorous evaluation of dense retrieval under multi-constraint biomedical ontology structures.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 597k pairs, 12 model architectures, two fine-tuning regimes, two-stage reranking, and formal error taxonomy.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear framing, mathematically self-contained metrics, and restrained, objective scientific claims.
- Value: ⭐⭐⭐⭐⭐ Highly impactful diagnostic benchmark for clinical RAG, clinical decision support systems, and neuro-symbolic retrieval.