Skip to content

GutenBench: A Taxonomy-Based Dataset for Evaluation of Information Retrieval Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/contemmcm/gutenbench
Area: LLM Reasoning
Keywords: Information retrieval benchmark, Retrieval taxonomy, Cross-lingual retrieval, Zero-shot evaluation, Latent conceptual retrieval

TL;DR

This work introduces GUTENBENCH, a fine-grained zero-shot IR benchmark built upon a four-level capability taxonomy (lexical, semantic, explicit aggregation, and latent conceptual bridging) spanning 2,170 expert-adjudicated queries and 272 parallel Bible translations across 141 languages, revealing that state-of-the-art retrievers systematically degrade on high-level conceptual queries and suffer acute performance drops in low-resource cross-lingual scenarios.

Background & Motivation

Retrieval-augmented generation (RAG) has emerged as the cornerstone methodology for grounding large language models (LLMs) and curbing hallucinatory generation. However, the reliability of RAG systems is strictly bounded by the fidelity and precision of the retrieved evidence: when retrievers supply irrelevant distractors or semantically adjacent yet non-grounding context, the generative LLM often amplifies hallucinations rather than resolving them. Despite the critical reliance on retrieval quality, diagnosing precisely why and under what conditions retrieval fails remains an open challenge because contemporary benchmarks routinely collapse evaluation into a single macro-averaged metric across heterogeneous query sets.

This aggregate-centric evaluation paradigm conceals critical failure modes due to inherent construction biases. Prevalent IR benchmarks are frequently generated by prompting annotators or models to formulate queries conditioned directly on randomly sampled text passages, followed by candidate pooling against lexical baselines like BM25. This workflow artificially skews benchmarks toward lexically transparent queries possessing strong verbatim overlap, which in turn inflates the apparent competence of basic lexical matching and masks the catastrophic degradation of modern dense retrievers when confronted with queries requiring complex multi-passage synthesis or abstract conceptual inferencing.

To overcome the diagnostic opacity of existing benchmark suites, this paper argues for an orthogonal evaluation framework that decouples distinct retrieval competencies within a single, highly controlled, and massively parallel corpus. Leveraging the structural parallelism of Bible translations, the authors construct a stratified benchmark where evidence difficulty spans from trivial surface-form matching to latent conceptual reasoning. Core idea: disentangle information retrieval capabilities into a four-level hierarchical taxonomy—lexical, semantic, explicit aggregation, and latent conceptual bridging—and operationalize it over a massively parallel corpus across 141 languages to provide precise, diagnostic visibility into retriever failure modes.

Method

Overall Architecture

The construction and evaluation pipeline of GUTENBENCH is structured around a three-phase quality control framework paired with a four-tier capability taxonomy. The process initiates with a broad pool of candidate questions harvested from diverse real-world online sources, processed via active learning-inspired sampling (combining diversity and inter-retriever disagreement) and AI-assisted human-in-the-loop triage. Subsequently, candidate evidence passages are pooled from five frontier LLMs alongside BM25 and subjected to rigorous adjudication by human subject-matter experts. Finally, passages are mapped onto standardized NKJV pericope document units, classified into the lowest sufficient taxonomy level, and deployed for cross-lingual zero-shot benchmarking.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Heterogeneous Web Query Pool<br/>~100K natural user queries"] --> B["Dual Sampling & Triage<br/>Diversity + disagreement sampling & AI-human triage"]
    B --> C["Multi-Source Pooling & Expert Adjudication<br/>5 frontier LLMs + BM25 pooling with SME relevance labels"]
    C --> D["Document Mapping & Taxonomy Assignment<br/>NKJV pericope boundary alignment & 4-level categorization"]
    D --> E["Cross-Lingual Zero-Shot Benchmark<br/>272 translations · 141 languages · stratified evaluation"]

Key Designs

1. Dual Sampling & Triage: Mitigating Document-Conditioned Query Bias and Topic Skew

Traditional benchmarks that reverse-engineer questions from corpus passages introduce severe document-conditioned query bias, where queries unnaturally mirror the specific vocabulary, phrasing, and syntax of target passages. GUTENBENCH circumvents this by harvesting ~100,000 real-world candidate queries from 11 diverse online platforms, including topical indexes, study guides, and community Q&A forums. To extract a highly informative diagnostic subset without succumbing to distributional distortion, the authors design a three-component sampling strategy: (i) Diversity sampling via text embedding, UMAP dimensionality reduction, and HDBSCAN clustering to balance topic coverage; (ii) Disagreement sampling across a committee of dense retrievers, prioritizing queries where retriever candidates exhibit minimal top-rank overlap to maximize diagnostic discriminability; and (iii) Random sampling to preserve base distribution anchoring. The subsequent Triage phase employs an LLM assistant to flag out-of-domain or metadata-centric queries (e.g., counting chapters), while dataset maintainers enforce the final inclusion decision.

2. Multi-Source Pooling & Expert Adjudication: Overcoming LLM Positivity Bias

To ensure high evidence recall without burdening annotators with exhaustive corpus-wide reading, candidate passage pooling combines five frontier zero-shot LLM engines (ChatGPT, Claude, Gemini, Grok, and DeepSeek) leveraging parametric memory and search capabilities, complemented by BM25 top-5 candidates to introduce hard lexical negatives. In the relevance assessment phase, human assessors evaluate every candidate passage on a graded scale: 0 (not relevant), 1 (partially relevant), and 2 (definitely relevant), retaining the authority to manually expand the pool if relevant passages were missed. A preliminary validation study on 50 golden queries established that while frontier LLMs approach expert-level recall (0.941–0.985), their precision is significantly degraded (0.806–0.842 vs. 0.877 for human experts). This documented "positivity bias" in LLMs acts as an overly lenient filter; deploying human subject-matter experts is essential to prevent false-positive evidence from corrupting the RAG diagnostic benchmark.

3. Document Mapping & Four-Level Taxonomy: Disentangling Single-Document from Multi-Document Complexity

To map annotated passages into standardized retrieval chunks, the target text is segmented using the editorial section headings of the New King James Version (NKJV pericopes), yielding 2,813 cohesive, non-overlapping document units that preserve semantic context while remaining within model context windows. Queries are subsequently classified into four distinct levels of retrieval capability: - L1 (Lexical): Single-document sufficient; queries and relevant passages share high surface keyword overlap. - L2 (Semantic): Single-document sufficient; requires resolving semantic gaps such as entity aliases, paraphrases, synonyms, or hypernym-hyponym associations. - L3 (Explicit Aggregation): Multi-document necessary; the information need requires aggregating multiple distinct, explicitly stated facts scattered across complementary documents (e.g., counting, entity comparison, sequence tracking). - L4 (Latent Conceptual Bridging): Multi-document necessary; relevant passages cannot be identified via explicit claims alone, requiring latent conceptual inferencing, theological principles, analogical reasoning, or reconciling conflicting perspectives.

Each query is assigned the lowest taxonomy tier capable of recovering its complete supporting evidence, confirmed through LLM consensus and author adjudication.

Benchmark & Evaluation Protocol

Evaluations are conducted under a zero-shot, first-stage retrieval regime, using normalized Discounted Cumulative Gain at rank 10 (nDCG@10) as the primary evaluation metric, supplemented by MAP and Recall@100. In the English monolingual setting, models are evaluated across a parallel corpus of 27 distinct English Bible translations to average out stylistic idiosyncrasies. In the asymmetric cross-lingual setting, English queries are paired against 245 non-English translations across 140 target languages, partitioned according to linguistic diversity into high-resource (21 languages), mid-resource (15 languages), and low-resource (104 languages) tiers. The benchmark tests sparse models (BM25) alongside 16 open-weights dense retrievers spanning sub-1B architectures (SBERT, BGE, Instructor, Qwen3) to large foundation scale models (Nemotron, GritLM, INF-X-Retriever, KaLM-Gemma3).

Key Experimental Results

Main Results

The main evaluation results on the English monolingual benchmark across 27 translations (reported as nDCG@10, mean ± standard deviation across translations) are summarized in Table 1:

Model Category Model Name Parameter Size L1 (Lexical) L2 (Semantic) L3 (Aggregation) L4 (Latent)
Sparse model BM25 — 0.712±0.046 0.325±0.026 0.306±0.016 0.200±0.015
Open-weights (<1B) SBERT L6 20M 0.437±0.027 0.206±0.023 0.247±0.013 0.216±0.013
Open-weights (<1B) SBERT L12 30M 0.460±0.022 0.231±0.022 0.269±0.014 0.223±0.014
Open-weights (<1B) SBERT MPNet 110M 0.548±0.015 0.338±0.018 0.339±0.011 0.260±0.013
Open-weights (<1B) Instructor Base 130M 0.303±0.019 0.123±0.017 0.158±0.014 0.167±0.011
Open-weights (<1B) SBERT Multilingual MPNet 280M 0.175±0.022 0.098±0.014 0.135±0.016 0.143±0.015
Open-weights (<1B) BGE-large-en-v1.5 340M 0.648±0.019 0.370±0.015 0.375±0.014 0.256±0.014
Open-weights (<1B) Instructor Large 370M 0.380±0.017 0.198±0.016 0.217±0.011 0.227±0.012
Open-weights (<1B) Qwen3 0.6B 600M 0.715±0.010 0.459±0.010 0.413±0.009 0.326±0.014
Open-weights (≥1B) Nemotron 1B 1.24B 0.839±0.011 0.640±0.012 0.541±0.012 0.431±0.018
Open-weights (≥1B) Instructor XL 1.27B 0.524±0.016 0.297±0.019 0.291±0.016 0.273±0.014
Open-weights (≥1B) Qwen3 4B 4.02B 0.832±0.010 0.639±0.012 0.567±0.011 0.483±0.016
Open-weights (≥1B) INF-X-Retriever Pro 7.07B 0.846±0.011 0.679±0.009 0.598±0.011 0.516±0.014
Open-weights (≥1B) GritLM 7B 7.24B 0.860±0.008 0.728±0.012 0.605±0.013 0.492±0.016
Open-weights (≥1B) Nemotron 8B 7.50B 0.897±0.006 0.765±0.006 0.654±0.008 0.565±0.014
Open-weights (≥1B) Qwen3 8B 7.57B 0.859±0.008 0.686±0.010 0.603±0.011 0.504±0.014
Open-weights (≥1B) KaLM Gemma3 12B 11.77B 0.827±0.006 0.680±0.009 0.615±0.007 0.505±0.012

Assessor Qualification and Computational Overhead

Table 2 details the precision, recall, and F1 scores of human experts and frontier LLM annotators during the assessor qualification study on 50 golden queries, alongside average token consumption and estimated API costs per model per 10 queries across taxonomy levels:

Assessor / Complexity Level Precision Recall F1-score Avg. Input Tokens Avg. Output Tokens GPT-5.5 Est. Cost (/10 Q)
Human Expert 0.877 0.993 0.931 — — —
DeepSeek-V4 Pro 0.832 0.985 0.902 — — —
ChatGPT 5.5 0.821 0.978 0.893 — — —
Grok 4.2 0.842 0.941 0.889 — — —
Claude Opus 4.7 0.815 0.971 0.886 — — —
Gemini 3.1 Pro 0.806 0.949 0.872 — — —
L1 (Lexical Pooling) — — — 55,081 37,375 $1.34
L2 (Semantic Pooling) — — — 59,361 43,303 $1.54
L3 (Aggregation Pooling) — — — 64,593 54,707 $1.90
L4 (Latent Conceptual) — — — 84,975 95,053 $3.23

Key Findings

  • Universal Capability Degradation: Across all evaluated dense and sparse architectures, performance drops monotonically as taxonomy level increases. L4 represents a severe systemic bottleneck: even the top-performing model, Nemotron 8B, attains only 0.565 nDCG@10 on L4, which is approximately 63% of its corresponding L1 score (0.897).
  • The Lexical Illusion of BM25: BM25 demonstrates competitive performance on L1 queries (0.712), outscoring multiple sub-1B dense models; however, its effectiveness collapses immediately on L2 (0.325) and L4 (0.200). In aggregate benchmarks dominated by L1-like queries, BM25 artificially appears competitive, obscuring its complete inability to handle semantic or multi-step reasoning.
  • Compounding Degradation in Low-Resource Cross-Lingual Settings: High-resource cross-lingual transfer exhibits minimal drop relative to English, but low-resource languages suffer drastic performance degradation that compounds with retrieval complexity. GritLM 7B, while highly competitive in English, degrades sharply in multilingual settings, whereas KaLM Gemma3 12B achieves the highest robustness and top scores across all four levels in low-resource cross-lingual retrieval.

Highlights & Insights

  • Diagnostic Stratification Framework: Replaces opaque aggregate metrics with a clean four-tiered taxonomy, transforming IR evaluation into an actionable diagnostic tool that pinpoints whether a retriever struggles with semantic paraphrase, information synthesis, or latent concept bridging.
  • The Bible as a Universal Cross-Lingual Calibration Standard: Creatively exploits the biblical corpus as a natural, highly standardized, and parallel textual framework spanning 141 languages, enabling controlled cross-lingual comparisons uncorrupted by translation artifacts or domain distribution shifts.
  • Human-in-the-Loop Safeguard Against LLM Positivity Bias: Provides empirical proof that modern LLMs exhibit significant leniency in relevance labeling; utilizing LLMs for recall pooling while reserving final judgment for human domain experts establishes a gold-standard protocol for IR benchmark construction.

Limitations & Future Work

  • Pericope Chunking and Document Length Constraints: The corpus is divided into verse-range pericope chunks to preserve cross-lingual structural alignment, focusing strictly on passage retrieval; it does not evaluate extreme long-context "needle-in-a-haystack" scenarios.
  • Domain Specificity of Biblical Text: Although chosen as a controlled diagnostic instrument rather than an all-inclusive domain, the textual style and conceptual structure of scripture may not fully transfer to technical industrial domains such as finance, biomedical science, or legal discovery.
  • Absence of Unanswerable Queries: The current benchmark comprises fully supported queries. Future work plans to integrate unanswerable queries to evaluate retriever abstention and calibrated "I do not know" uncertainty estimation.
  • vs BEIR / MTEB: While BEIR and MTEB assess zero-shot transfer across heterogeneous domains, their aggregate metrics aggregate trivial lexical matches with deep semantic queries; GUTENBENCH isolates and stratifies fine-grained retrieval capabilities within a single controlled corpus.
  • vs BRIGHT / R2MED: While recent benchmarks explore reasoning-intensive retrieval, they often rely on complex synthetic logic chains limited to English; GUTENBENCH establishes an intuitive 4-tier taxonomy spanning 141 languages with massively parallel grounding.
  • vs MIRACL / NeuCLIR: Prior multilingual benchmarks focus primarily on high-to-mid-resource languages and frequently utilize machine translation; GUTENBENCH provides human-curated authentic translations across 104 low-resource languages.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Establishes a four-tier capability taxonomy that fundamentally disentangles lexical, semantic, aggregation, and latent conceptual retrieval.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 17 diverse models across 272 translations, 141 languages, with bootstrap confidence interval validation and detailed cost accounting.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorously articulated, logically cohesive narrative with pristine mathematical definitions and transparent empirical reporting.
  • Value: ⭐⭐⭐⭐⭐ Delivers an indispensable diagnostic instrument for RAG and dense retrieval architectures, exposing deep limitations in conceptual bridging and multilingual generalization.