Skip to content

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Benchmark Validity, Meta-Evaluation, Content Validity, Convergent Validity, LLM Evaluation

TL;DR

BenchBrowser is an interactive retrieval infrastructure that searches item-level evaluation evidence across 20+ benchmarks for practitioner-defined use-cases, systematically uncovering pervasive threats to content validity and convergent validity in language model benchmarks.

Background & Motivation

In the rapid development and real-world deployment of large language models, practitioners routinely rely on public benchmarks as proxies for assessing model competence. However, existing benchmarks exhibit substantial heterogeneity in both design scope and granular measurement. Some suites, such as InfoBench and AlpacaEval, evaluate distinct capabilities across individual test instances, whereas others, like GSM8K and NutriBench, focus on a single, narrowly constrained task. This coarseness and heterogeneity mean that benchmark-level metadata obscures how an advertised capability is operationalized at the individual item level. A benchmark titled "poetry generation" may omit haikus and sonnets entirely, while an "instruction-following" benchmark tests an arbitrary, idiosyncratic set of skills. Consequently, aggregate benchmark scores fail to reveal whether a test suite truly covers a practitioner's downstream construct.

This gap between benchmark claims and item-level content engenders two fundamental validity crises originating from psychometric measurement theory. The first is a threat to content validity—the extent to which the items in an evaluation adequately represent the entire multifaceted domain of the capability being measured. Content validity is frequently compromised by sparse coverage or scope-metadata mismatches, where a benchmark claims broad competence while evaluating only a tiny, unrepresentative sub-facet. The second is a threat to convergent validity, which requires that nominally similar benchmarks measuring the same underlying capability induce concordant model comparisons. In practice, models excelling on one reasoning benchmark often collapse on another, producing conflicting rankings and leaving practitioners with no grounded way to decipher which signal to trust.

Identifying these validity pitfalls at scale is prohibitively expensive because validity is both item-dependent and stakeholder-specific: static dataset cards cannot reveal latent capability operationalizations, and manual item audits do not scale to tens of thousands of benchmark examples. To overcome this impasse, the authors reframe validity inspection as a semantic retrieval task. Core idea: build an interactive retrieval infrastructure, BenchBrowser, that rewrites practitioner use-cases into synthetic anchors, retrieves semantically aligned test items across 70,000+ benchmark examples in seconds, and surfaces concrete item-level evidence to expose content disparities and ranking inversions.

Method

Overall Architecture

BenchBrowser operates as an end-to-end evidence retrieval and meta-evaluation pipeline. Given an arbitrary natural language use-case from a practitioner, the system executes three sequential stages: anchor generation, semantic dense retrieval, and LLM judge filtering. First, an anchor generator transforms the practitioner's abstract use-case into grounded query representations to bridge the format and phrasing gaps between high-level intent and concrete benchmark questions. Next, an instruction-tuned bi-encoder retrieves top-\(k\) nearest-neighbor candidates from a multi-benchmark index comprising over 70,000 items spanning 20+ popular NLP evaluation suites. Finally, an LLM judge conservatively filters out spurious candidates, presenting practitioners with a clean pool of relevant evaluation items in under 5.3 seconds. Based on this retrieved evidence, the system powers two core meta-evaluation workflows: facet analysis for auditing content validity disparities and rank divergence measurement for auditing convergent validity stability.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Practitioner Use-case<br/>(Natural Language Intent)"] --> B["Stage 1: Example Synthesis Anchoring<br/>(Format-Aware Synthetic Probes)"]
    B --> C["Stage 2: Semantic Dense Retrieval<br/>(Multi-Anchor Pooling via BGE-en-icl)"]
    C --> D["Stage 3: LLM Judge Filtering<br/>(Conservative Relevance Screening)"]
    D --> E["Validity Meta-Evaluation Auditing<br/>(Interactive Evidence Inspection)"]
    E --> F["Facet Content Validity Analysis<br/>(Quantifying Representational Skew)"]
    E --> G["Model Rank Divergence Analysis<br/>(Kendall Tau Stability vs. Gold Sets)"]

Key Designs

1. Example Synthesis Anchoring: Bridging Abstract Intent and Concrete Benchmark Item Formats

Directly embedding abstract practitioner queries (such as "creative writing" or "instruction following") causes severe retrieval failure because indexed benchmark items contain salient surface-level format cues (e.g., multiple-choice answer options, fill-in-the-blank brackets, or unit-test scaffolding) that are absent in high-level intent descriptions. To solve this representation mismatch, BenchBrowser implements Example Synthesis. For any incoming use-case, an LLM generates \(n=3\) synthetic evaluation test cases (1-2 lines long) in diverse standard evaluation formats (such as open-ended QA, MCQ, or fill-in-the-blank) containing only the question prompt without the answer. By matching the syntactic and structural distribution of indexed benchmark items, Example Synthesis substantially outperforms raw query baselines (Original), subtopic rephrasing (Rephrasing), and QLoRA-based structured slot extraction (Shorthand \(\langle\text{skill}\rangle \& \langle\text{key}_1\rangle \& \langle\text{key}_2\rangle\)), achieving a top-tier Prec sys@20 of 0.890 and NDCG@20 of 0.971 across evaluation sets.

2. Semantic Dense Retrieval: Balancing Multi-Format Recall with Interactive Low Latency

Searching over 70,000 heterogeneous benchmark items across 20+ datasets requires balancing instruction comprehension against strict latency budgets for interactive human auditing. Cross-encoder rerankers provide strong precision but require over 14.0 seconds per query, rendering interactive workflows impractical. BenchBrowser adopts BAAI/bge-en-icl paired with a Faiss index as its retrieval backbone. For each synthetic anchor generated in Stage 1, the retriever fetches nearest-neighbor items in embedding space using in-context learning query prompts. Candidates from all anchors are pooled, deduplicated, sorted by cosine similarity, and truncated to the top-\(k\) set. This configuration achieves superior precision (Prec sys@k = 0.694) and recall (0.218) over standard bi-encoders (bge-large-en-v1.5) and instruction-tuned embeddings (llm-embedder), while executing dense search in just 1.04 seconds and sustaining an end-to-end multi-tenant latency of \(5.28 \pm 1.48\) seconds.

3. LLM Judge Filtering: Conservative De-Noising While Preserving Construct Exploration

Top-\(k\) candidates returned from vector search can still contain items with spurious lexical overlap that fail to capture the target construct. Rather than relying on an uncalibrated hard threshold, BenchBrowser applies an LLM judge (gpt-5-mini-2025-08-07) to evaluate candidates across three structured criteria: Knowledge Overlap (domain topic alignment), Skill Transfer (applicability of underlying cognitive reasoning), and Practical Utility (whether solving the item provides credible evidence for the practitioner's goal). The judge assigns a 3-point score: Relevant (1), Partially Relevant (0), or Irrelevant (-1). Crucially, the filtering step is deliberately configured to eliminate only clearly irrelevant noise (-1), leaving the nuanced boundary between Relevant and Partially Relevant intact for practitioner exploration. Human evaluation across 4,260 annotations confirms that human-LLM agreement (Gwet's \(\text{AC}_2\) between 0.43 and 0.90) concentrates disagreements almost entirely at the partial-vs-full relevance threshold, confirming the safety and validity of this automated screening.

4. Dual Validity Auditing Protocols: Facet Dispersion and Rank Divergence Quantification

To operationalize psychometric validation for AI practitioners, BenchBrowser establishes two formal auditing protocols: - Content Validity Audit (Facet Analysis): The broad capability is held constant (e.g., code generation) while an orthogonal facet is varied across controlled variants (e.g., programming language: Python, C++, Rust, Go). The system tracks the percentage and volume of retrieved relevant items across facets, surfacing severe representational skew and capability blind spots. - Convergent Validity Audit (Rank Divergence \(\Delta\)): For use-cases possessing pre-existing gold benchmarks, the system measures whether model rankings induced by retrieved items agree with the gold benchmark ranking across 8 models. To separate true divergence from small-sample noise, it defines rank divergence as: $\(\Delta(u) = \tau_{\text{gold}}(u) - \tau_{\text{ret}}(u)\)$ where \(\tau_{\text{ret}}(u)\) is the Kendall's rank correlation between the model ranking on retrieved items \(R_u\) and a size-matched subsample from the gold benchmark \(G_u\), and \(\tau_{\text{gold}}(u)\) is the ceiling correlation obtained by subsampling the gold set against itself. A large positive \(\Delta(u)\) demonstrates that ranking instability is driven by construct mismatch rather than sample size limitations.

Benchmark & Evaluation Protocol

BenchBrowser indexes 20 prominent benchmarks (including MMLU, BIG-bench Hard, IFEval, InfoBench, GSM8K, HumanEval, AlpacaEval, PubMedQA, etc.) evaluating accuracy, exact match, Pass@1, and win-rate against reference models. The evaluation testbed covers: 1. Synthetic Use-cases (\(N=158\)): 58 broad topic domains from MMLU/BBH, 50 atomic skills derived from Claude conversation logs via Clio, and 50 concrete deployed application tasks; 2. Crowdsourced Novel Intent Queries (\(N=21\)): Elicited from 9 NLP practitioners reflecting non-standard evaluation intents (e.g., research visualization scripts); 3. Controlled Validation Queries (\(N=20\)): Paired 1:1 with existing gold benchmarks (e.g., NutriBench, SciQ, WMDP) with the gold sets explicitly excluded from the retrieval index to evaluate bounded recall and rank transfer.

Key Experimental Results

Main Results: Unified Retrieval Performance Across Query Types (@K=20)

The table below evaluates anchor generation methods across Topics, Skills, and Applications query sets against lexical and neural baselines on the full 70k benchmark database (quoted directly from Table 1 of the paper):

Method Category Retrieval Method Topics: Prec sys@20 Topics: NDCG@20 Topics: Recall@20 Skills: Prec sys@20 Skills: NDCG@20 Applications: Prec sys@20 Applications: NDCG@20
Non-Neural Baseline Random 0.304* 0.702* 0.029* 0.310* 0.628* 0.045* 0.231*
Non-Neural Baseline BM25 0.529* 0.850* 0.191* 0.654* 0.823* 0.320* 0.648*
Neural Dense Retrieval Original 0.763* 0.921* 0.350* 0.768* 0.911* 0.572 0.870
Neural Dense Retrieval Rephrasing 0.812* 0.923* 0.345* 0.775 0.908* 0.563 0.849
Neural Dense Retrieval Shorthand 0.884 0.955* 0.414 0.815 0.916 0.572 0.808*
Neural Dense Retrieval Example Synthesis 0.890 0.971 0.423 0.837 0.937 0.560 0.869

Note: * denotes methods performing significantly below the best method in that column based on paired sign-flip tests with Holm correction (\(p < 0.05\)).

Convergent Validity & Model Rank Divergence

Evaluating 8 models (Llama-3.1-8B-Instruct, Llama-3.1-Tulu-3-8B, Mistral-7B-Instruct-v0.2, Qwen1.5-14B-Chat, Qwen2.5-7B-Instruct, c4ai-command-r7b, gemma-2-9b-it, gemma-3-4b-it) across 100 Monte Carlo trials reveals severe rank divergence between gold benchmarks and retrieved relevant items (excerpted from Table 12):

Task Retrieved Items \(k\) \(\tau_{\text{ret}}\) (Mean) \(\tau_{\text{gold}}\) (Ceiling) Rank Divergence \(\Delta = \tau_{\text{gold}} - \tau_{\text{ret}}\) \(\tau_{\text{sanity}}'\) (Equal Budget) \(\Delta'\) (Equal Budget) Convergent Validity Finding
wmdp 60 0.17 0.39 0.21 [0.15, 0.28] 0.20 0.01 Moderate convergence; high intrinsic gold variance
pubmed 69 -0.04 0.20 0.25 [0.17, 0.31] 0.00 0.05 Near zero correlation; diverse question formulations
bb_cs_algo 73 0.10 0.47 0.36 [0.31, 0.42] 0.28 0.20 Moderate divergence; format sensitivity in algorithm QA
nutribench 62 0.04 0.45 0.41 [0.34, 0.48] 0.35 0.34 Significant divergence; nutrition facts vs calculations
sciq 78 0.12 0.71 0.58 [0.54, 0.62] 0.57 0.43 Severe rank divergence in general science reasoning
bb_social_support 60 -0.11 0.74 0.86 [0.82, 0.90] 0.61 0.76 Ranking inversion; conversational dialogue vs multiple choice
bb_hindu 48 -0.36 0.62 0.98 [0.93, 1.03] 0.36 0.73 Severe inversion; deity hierarchy vs scriptural context
bb_riddle_sense 24 -0.23 1.00 1.23 0.54 0.75 Massive inversion; verbal wordplay vs deductive riddles

Key Findings

  • Query rewriting eliminates format mismatch: Neural dense retrieval using synthesized example anchors yields gains of over 10 points in Prec sys@20 compared to raw queries on Topics and Skills sets, proving that providing explicit input format cues is essential for retrieving benchmark items.
  • Content validity exhibits acute facet skew: Examining 20 skill families across 120 facets reveals dramatic dispersion. In code generation, Python yields an 86.7% relevant retrieval rate while Go yields only 10.0%; in creative writing, fantasy achieves 75.9% while historical fiction reaches only 51.7%. Lexical BM25 baselines confirm this skew is an inherent artifact of benchmark curation rather than a retrieval bias.
  • Convergent validity fails frequently due to three distinct drivers:
  • Rubric Shift: Changing scoring paradigms (e.g., from multiple-choice accuracy on BIG-bench to generative win-rate on HellaSwag or AlpacaEval) systematically alters model rankings;
  • Divergent Operationalization: Nominal construct overlap conceals fundamentally different cognitive demands (e.g., solving riddles based on wordplay in bb_riddle_sense vs. constraint tracking in colored object puzzles);
  • Context Shift: Evaluation transitions from factual recall to applied execution in realistic scenarios (e.g., one-line Python syntax tests vs. full software debugging).

Highlights & Insights

  • Reframing meta-evaluation as semantic retrieval: BenchBrowser sidesteps the prohibitive expense and static obsolescence of manual item taxonomy annotations by creating an interactive, real-time search engine over 70k+ existing benchmark items.
  • Actionable attribution of ranking inversions: Rather than reporting aggregate rank instability as a black-box metric, BenchBrowser allows practitioners to inspect the exact test items that reverse model orderings, diagnosing whether discrepancies stem from spurious metric artifacts or legitimate capability gaps.
  • Empowering practitioner-centric construct inspection: Validity is fundamentally a property of how test scores are interpreted for a specific use-case. By enabling dynamic querying across arbitrary facets, the tool puts construct validation directly into the hands of downstream users.

Limitations & Future Work

  • Constrained human evaluation demographics: The human study was conducted with \(N=10\) NLP graduate practitioners all located in the United States, which may limit the cultural and linguistic diversity of judged use-cases.
  • Bound to indexed public benchmark inventory: The tool can only surface evidence that already exists within its 20-benchmark database; entirely unrepresented practical capabilities (such as multi-day autonomous enterprise workflows) cannot be retrieved or inspected.
  • Vulnerability to gaming and cherry-picking: If used adversarially, the retrieval engine could theoretically be co-opted to curate biased benchmark subsets that artificially flatter a specific model's architectural strengths.
  • vs. Psychometric Construct Validity Audits (Bean et al., 2025; Alaa et al., 2025): Prior audits established the critical need for construct validity in AI but relied on costly expert manual tagging or fixed taxonomies; BenchBrowser automates evidence collection dynamically via retrieval.
  • vs. Synthetic Benchmark Generation (AutoBencher, ONEbench): Fully synthetic benchmark generation often suffers from LLM generator biases, stylistic homogeneity, and unvetted task difficulty; BenchBrowser reuses established, peer-reviewed benchmark items to avoid synthetic degradation.
  • vs. Mechanistic Interpretability Audits (SAE-based Probing, Bohacek et al., 2026): SAE probes evaluate capability gaps inside model weights, which are inaccessible for proprietary models; BenchBrowser operates strictly at the behavioral interface, auditing empirical task validity directly.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering framework integrating information retrieval with psychometric content and convergent validity auditing for LLMs.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive experiments across 20+ benchmarks, 70k+ items, 8 models, and 4,260 human annotations with rigorous statistical testing.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally articulate narrative, clear conceptual framing, and elegant formalization of validity threats.
  • Value: ⭐⭐⭐⭐⭐ A vital diagnostic tool for mitigating benchmark saturation and misleading evaluation claims across academic and industry AI deployments.