Skip to content

Auditing MCQA Benchmarks through Probability Landscapes

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Evaluation
Keywords: MCQA benchmark auditing, probability landscapes, normalized residual entropy, mean pairwise distance, noise injection

TL;DR

Proposes a two-level MCQA benchmark auditing framework using model output probability distributions, characterizing macro benchmark structure via top prediction probability and normalized residual entropy summarized by Mean Pairwise Distance, and filtering distractor competition via noise injection to effectively prioritize flawed items for targeted human review.

Background & Motivation

Multiple-choice question answering (MCQA) benchmarks serve as the primary instrument for assessing the capabilities of Large Language Models (LLMs) due to their fixed response format and straightforward automated scoring. However, as frontier model capabilities continue to surge, established benchmarks such as MMLU, ARC, and HellaSwag are rapidly approaching performance saturation. While the research community has responded by developing increasingly demanding datasets such as GPQA, MMLU-Pro, and Humanity's Last Exam (HLE), constructing and validating new high-difficulty benchmarks requires substantial human expert labor. Meanwhile, auditing and cleaning flaws within existing benchmarks remains a costly and labor-intensive challenge.

Classic psychometric theory and educational measurement posit that a well-designed multiple-choice item must include plausible distractors that engage in meaningful competition with the correct answer. In contrast, modern LLM benchmark analysis predominantly relies on aggregate accuracy metrics or leaderboard score shifts, which discard fine-grained internal confidence distributions across alternative options. While input-removal baselines (e.g., question stem removal) can expose superficial dataset biases, they do not offer a systematic diagnostic lens into overall benchmark health or internal distractor efficacy. Relying strictly on comprehensive human re-annotation (as in MMLU-Redux) is financially prohibitive for wide-scale benchmark auditing.

The central angle of this paper is that output probability distributions across candidate tokens encode rich representations of model uncertainty and alternative competition. By analyzing the residual probability mass allocated among non-selected options and stripping away plausible distractor competition via semantic noise, benchmarks can be diagnosed efficiently without heavy annotation overhead. Core idea: construct a dual-level probabilistic auditing framework that characterizes macro-level option competition and structural heterogeneity using probability landscapes and Mean Pairwise Distance, while employing semantic noise injection to sharply isolate candidate flawed items for targeted human review.

Method

Overall Architecture

The framework comprises two complementary modules: benchmark-level probability landscape analysis and item-level perturbation diagnostics. At the macro level, model output logits on the choice tokens (A, B, C, D) are extracted and converted to probabilities via Softmax under standard greedy inference. Each question is represented in a two-dimensional coordinate space defined by the top prediction probability \(P_{\text{top1}}\) and the normalized residual entropy \(H_{\text{norm}}\), with overall benchmark dispersion and cohesion quantified by the Mean Pairwise Distance (MPD). At the micro level, semantic noise injection replaces original distractors with unrelated city names to eliminate distractor competition. Items that models continue to answer incorrectly under noise injection are flagged as residual failures and triaged by human auditors into perturbation artifacts (Excluded) and genuine benchmark defects (Actionable).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["MCQA input question and candidate options"] --> B["Benchmark probability extraction<br/>extract option token logits and compute Softmax"]
    B --> C["2D probability landscape representation<br/>compute top confidence P_top1 and residual entropy H_norm"]
    C --> D["Macro landscape aggregation and cohesion<br/>pool cross-model points to compute Mean Pairwise Distance MPD"]
    B --> E["Semantic noise injection perturbation<br/>substitute distractors with unrelated city names"]
    E --> F["Residual failure triage and targeted human review<br/>separate perturbation artifacts from actionable flaws"]

Key Designs

1. 2D Probability Landscape Representation: Joint Modeling of Model Confidence and Residual Distractor Competition

Standard accuracy discard how probability mass is distributed across non-selected options. To capture the internal distractor dynamics, each question is mapped to a two-dimensional vector \(z_i = (P_{\text{top1}}^{(i)}, H_{\text{norm}}^{(i)})\). Here, \(P_{\text{top1}} = \max_{x_i \in O} P(x_i \mid q)\) reflects the model's confidence in its top prediction. To measure whether the remaining distractors compete meaningfully with each other, the probability distribution over the non-selected options \(O \setminus \{y_{\text{pred}}\}\) is renormalized into \(P'(x_i) = \frac{P(x_i \mid q)}{\sum_{x_j \in O \setminus \{y_{\text{pred}}\}} P(x_j \mid q)}\), after which the Normalized Residual Entropy is computed:

\[H_{\text{norm}} = \frac{-\sum_{x_i \in O \setminus \{y_{\text{pred}}\}} P'(x_i) \log_2 P'(x_i)}{\log_2(|O| - 1)}\]

\(H_{\text{norm}}\) is strictly bounded between 0 and 1. A high \(H_{\text{norm}}\) indicates that the model distributes residual probability evenly across all non-selected distractors, signaling broad competition among alternatives. Conversely, \(H_{\text{norm}} \approx 0\) indicates concentrated competition where a single unchosen option absorbs nearly all remaining probability mass. Combining \(P_{\text{top1}}\) and \(H_{\text{norm}}\) provides a compact structural fingerprint of each benchmark question.

2. Macro Landscape Cohesion Quantification: Measuring Structural Heterogeneity via Mean Pairwise Distance

To summarize the global dispersion of points in the probability landscape, the framework introduces the Mean Pairwise Distance (MPD). Given a set of \(N\) question state vectors \(\{z_i\}_{i=1}^N\) pooled across models and items, MPD calculates the average Euclidean distance across all pairs:

\[\text{MPD} = \frac{2}{N(N - 1)} \sum_{i < j} \|z_i - z_j\|_2\]

In the primary evaluation, 1,000 sampled items per benchmark are evaluated across three distinct open-weight models (Qwen-2.5 7B, Gemma-2 9B, and Llama-3 8B), producing \(N=3,000\) pooled (item, model) points. A lower MPD signifies that items cluster tightly in a narrow region of the probability landscape (e.g., highly saturated benchmarks where most items crowd near \(P_{\text{top1}} \approx 1.0\)). A higher MPD indicates greater structural heterogeneity in item difficulty, model confidence, and distractor competition.

3. Semantic Noise Injection: Eliminating Distractor Competition to Narrow Candidate Flaws

Reviewing an entire benchmark manually is cost-prohibitive. The framework introduces noise injection, which replaces all original distractors with semantically unrelated city names (e.g., Paris, Seoul, Tokyo). Under sound test construction, removing meaningful semantic competition should render the correct option glaringly obvious, driving model accuracy close to 100%. Questions that models still fail to answer correctly under noise injection are marked as residual failures. This diagnostic filter narrows thousands of benchmark items down to a small, tractable pool of candidate flawed questions.

4. Fine-Grained Failure Taxonomy and Targeted Human Review: Separating Perturbation Artifacts from Benchmark Flaws

To handle the flagged residual failures rigorously, a two-level classification taxonomy is established through human review. The taxonomy partitions residual failures into two groups: Excluded cases (failures caused by the perturbation itself) and Actionable benchmark issues (true dataset flaws). Excluded types include Negation / EXCEPT format breakdown (X1) (where injected city names also satisfy the negative condition) and Semantic collision (X2) (where an injected city name happens to be a valid answer in context). Actionable types include Scope inconsistency (A1), Answer-key errors (A2), Encoding artifacts (A3, e.g., 1/6 auto-converted into '6-Jan' by Excel), Narrative ambiguity (A4), and Trivia-dependent underspecification (A5).

Key Experimental Results

Main Results

Evaluations were conducted using three open-weight instruction-tuned models under 10B parameters: Qwen-2.5 7B, Gemma-2 9B, and Llama-3 8B across four MCQA benchmarks: MMLU, ARC, HellaSwag, and CommonsenseQA (CSQA, standardized to 4 choices). Deterministic greedy decoding (temperature=0, top-p=1) was applied to extract the first-token logits for letters Aโ€“D.

Table 1 summarizes the baseline Mean \(H_{\text{norm}}\) and pooled MPD (\(N=3,000\)) alongside accuracy across perturbation conditions (100% question token masking, no-answer injection, noise injection, and score bias with '(Score: 10/10)').

Benchmark Mean \(H_{\text{norm}}\) Pooled MPD (N=3000) Baseline Acc (%) Noise Injection Acc (%) No Answer Acc (%)
ARC 0.572 0.404 91.4 99.7 (+8.3) 40.8 (-50.6)
CommonsenseQA (CSQA) 0.488 0.424 83.4 99.1 (+15.6) 32.4 (-51.0)
HellaSwag 0.552 0.433 73.3 98.5 (+25.2) 36.5 (-36.8)
MMLU 0.536 0.479 60.2 98.6 (+37.7) 17.1 (-43.1)

Note: In supplementary experiments on GPQA (448 items across 3 models, N=1,344), pooled MPD was 0.477, reflecting high structural dispersion comparable to MMLU.

Ablation Study

Among the 4,000 total benchmark instances evaluated across the four datasets, noise injection revealed exactly 30 residual failure cases. Targeted human review categorized these failures according to the proposed taxonomy:

Group Failure Subtype Core Mechanism & Manifestation Item Count Distribution
Excluded X1. Negation / EXCEPT Injected city names satisfy negative condition, destroying contrast 15 50.0% of failures
X2. Semantic collision Injected city name accidentally acts as a valid contextual answer 6 20.0% of failures
Actionable A1. Scope inconsistency Question stem asks about 'all' cases while answer fits only a subset 1 Verified dataset flaw
A2. Answer-key error Ground-truth label is factually incorrect or another choice is better 3 Verified dataset flaw
A3. Encoding artifact File conversion corrupted choices (e.g., fraction 1/6 converted to 6-Jan) 1 Verified dataset flaw
A4. Narrative ambiguity Context does not uniquely determine the designated continuation 3 Verified dataset flaw
A5. Trivia underspecification Answer hinges on unstated pop-culture/episode trivia (e.g., Seinfeld) 1 Verified dataset flaw
Total All Residual Failures 21 perturbation limitations excluded; 9 actionable flaws identified 30 Drastic reduction in audit effort

External validation on the 373-item exact-text overlap between the MMLU subset and MMLU-Redux expert annotations (containing 9 confirmed flaws): - Noise injection alone flagged 2 items, achieving a Precision of 1.000 (2/2 verified flaws), Recall of 0.222, and an Enrichment of 41.4ร— over random inspection (\(p = 0.0005\)). - Noise injection combined with an LLM structural classifier (GPT-4o-mini) flagged 11 items, reaching a Recall of 0.444 (4/9 flaws caught), Precision of 0.364, and an Enrichment of 15.1ร— (\(p < 0.0001\)), massively outperforming the 0.024 random baseline.

Key Findings

  • Consistent Macro Structural Rankings: Pooled MPD establishes a consistent dispersion ordering across benchmarks: ARC (0.404) < CSQA (0.424) < HellaSwag (0.433) < MMLU (0.479). ARC concentrates heavily near \(P_{\text{top1}} \approx 1.0\), signaling near-total saturation for modern LLMs, whereas MMLU exhibits the highest structural diversity.
  • Item-Level Entropy is Model-Dependent: While macro \(H_{\text{norm}}\) rankings are broadly stable across calibrated models, cross-model Spearman rank correlations of item-level \(H_{\text{norm}}\) hover near zero (small-model mean \(r \in [0.028, 0.177]\); all-pair mean \(r \in [-0.010, 0.087]\)). This indicates that individual item residual entropy is dominated by model-specific calibration and cannot serve as an autonomous flaw judge.
  • Noise Injection Drastically Shrinks Audit Scope: Noise injection elevates accuracy across all datasets to 98.5%โ€“99.7%, eliminating non-essential reasoning failures. The remaining 30 residual errors, after filtering out 21 perturbation-induced artifacts, directly surfaced 9 actionable benchmark defects with high efficiency.

Highlights & Insights

  • Probabilistic Signals as Diagnostic Probes: Moves beyond scalar accuracy metrics by using renormalized residual entropy \(H_{\text{norm}}\) and spatial cohesion MPD to characterize the internal geometry of distractor competition.
  • Two-Stage Cost-Effective Auditing: Combines semantic noise injection with targeted human review, achieving a 41.4ร— flaw enrichment ratio that replaces full-scale re-annotation with targeted micro-audits.
  • Self-Aware Taxonomical Distinction: Explicitly identifies perturbation failure modes (negation format collapse and semantic collisions), establishing clear epistemic boundaries between model quirks, perturbation artifacts, and actionable dataset errors.

Limitations & Future Work

  • Requirement of White-Box Option Logits: The framework relies on extracting output token probabilities, preventing direct application to closed-source commercial APIs that offer text-only responses.
  • Instability of Item-Level Residual Entropy: Single-item \(H_{\text{norm}}\) is sensitive to tokenizer nuances, prompting formats, and model calibration, making it suitable only as an aggregate descriptive tool rather than a standalone per-item filter.
  • Static Distractor Noise Strategy: Using uniform city names creates vulnerabilities in negative-format questions (NOT/EXCEPT). Developing context-aware and syntax-preserving synthetic noise could reduce false-positive perturbation artifacts.
  • vs MMLU-Redux (Gema et al., 2025): MMLU-Redux relies on exhaustive expert re-annotation across thousands of questions, which is too costly to scale across every benchmark. This work provides a two-stage triage funnel, shrinking the review surface by over 99% while identifying equivalent structural flaws.
  • vs Benchmark Saturation Studies (Akhtar et al., 2026): Prior saturation studies analyze leaderboard score ceilings and discriminative power statistically. This work investigates internal benchmark mechanics, explaining why datasets saturate through the lens of distractor competition and probability landscapes.

Rating

  • Novelty: โญโญโญโญโ˜† Creatively repurposes output probability landscapes and residual entropy for benchmark data quality auditing.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 4 major benchmarks, 6 model architectures, controlled perturbations, and external expert alignment validation.
  • Writing Quality: โญโญโญโญโญ Rigorous prose with clean conceptual boundaries between macro descriptive summaries and micro actionable diagnostics.
  • Value: โญโญโญโญโญ Provides an actionable, highly cost-effective workflow for auditing, cleaning, and desaturating LLM evaluation benchmarks.