When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Fair Comparison of LLMs¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/chaochaoxx/ACE
Area: LLM (Other)
Keywords: Large Language Model Calibration, Accuracy Confounder, Accuracy-Controlled Evaluation, Ranking Reversal, Uncertainty Estimation
TL;DR¶
Reveals that standard global calibration metrics like ECE and Brier Score are severely confounded by task accuracy, introduces the tripartite ACE (Accuracy-Controlled Evaluation) framework with instance, distribution, and candidate alignment, and demonstrates that apparent calibration gains of larger or thinking models frequently reverse once accuracy is controlled.
Background & Motivation¶
As Large Language Models (LLMs) are progressively deployed in safety-critical domains such as healthcare diagnosis, legal jurisprudence, and autonomous programming, measuring whether a model knows what it knowsโtermed confidence calibrationโhas become a fundamental pillar of trustworthy AI. Researchers and practitioners routinely employ global summary metrics such as the Expected Calibration Error (ECE) and Brier Score (BS) to evaluate the degree to which a model's assigned confidence matches its empirical accuracy. Across literature, cross-model comparisons have rested upon an intuitive yet largely unverified assumption: that raw global calibration scores are directly comparable across disparate systems, and that a lower ECE or Brier Score reliably signifies superior predictive uncertainty estimation.
However, this conventional paradigm suffers from a profound methodological and statistical flaw: global calibration metrics are intrinsically confounded by underlying task accuracy. Rather than evaluating confidence distribution quality in isolation, global metrics inherently aggregate confidence-accuracy discrepancies over mixtures of correct and incorrect outcomes. When comparing models that differ substantially in capabilities, a higher-performing model can mechanically achieve a superior global calibration score simply because it answers more questions correctly, even when its underlying confidence behaviors remain completely identical or actively pathologically overconfident. An empirical diagnosis across multiple mainstream open-weight model families and benchmark datasets confirms a strong negative correlation (\(\rho = -0.670, p < 10^{-5}\)) between task accuracy gaps and raw ECE differentials, demonstrating that reported "calibration advantages" of larger architectures are largely artifacts of outcome composition rather than genuine metacognitive acumen.
Critically, this accuracy confound induces a pathological failure mode termed "ranking reversal": a model that appears significantly better calibrated under raw global metrics can become demonstrably less calibrated once task accuracy is held constant. The existence of frequent ranking reversals directly invalidates naรฏve cross-model calibration comparisons. To untangle task accuracy from confidence estimation quality, the core idea of this work is to establish ACE (Accuracy-Controlled Evaluation), a rigorous framework featuring three complementary evaluation viewsโInstance-Aligned (IA), Distribution-Aligned (DA), and Candidate-Aligned (CA)โthat explicitly equalize predictive accuracy conditions to enable fair, unconfounded cross-model calibration comparison.
Method¶
Overall Architecture¶
The ACE framework resolves cross-model calibration bias by evaluating confidence estimates under matched correctness distributions. Given an evaluation benchmark \(\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N\) and two competing models \(M_A\) and \(M_B\) (with empirical accuracies \(a_A = \text{Acc}(M_A)\) and \(a_B = \text{Acc}(M_B)\) where \(a_A > a_B\)), ACE abandons naive global score comparison over \(\mathcal{D}\). Instead, it formulates two generative calibration views that isolate self-scoring confidences, alongside an evaluative calibration view over a shared candidate answer space: Instance-Aligned calibration (IA) filters the evaluation strictly to shared outcomes (\(\mathcal{D}_{\text{both-right}} \cup \mathcal{D}_{\text{both-wrong}}\)); Distribution-Aligned calibration (DA) leverages propensity score importance weighting from causal inference to adjust the stronger model's outcome proportions to the reference accuracy level; and Candidate-Aligned calibration (CA) constructs a pooled answer space wherein both models score the identical set of candidate solutions to eliminate generation discrepancies and self-preference biases.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Benchmark Set & Model Predictions<br/>(x_i, y_i) -> Predictions & Confidences"] --> B{"Select ACE Alignment View"}
B --> C["1. Instance-Aligned Calibration (IA)<br/>Intersect shared outcomes: both-right & both-wrong"]
B --> D["2. Distribution-Aligned Calibration (DA)<br/>Inverse propensity weighting to match reference accuracy"]
B --> E["3. Candidate-Aligned Calibration (CA)<br/>Score shared candidate pool to remove self-preference"]
C --> F["Fair Calibration Assessment<br/>Compute aligned ECE / Brier Score & detect ranking reversals"]
D --> F
E --> F
Key Designs¶
1. Instance-Aligned Calibration (IA): Enforcing Strict Instance-Level Correctness and Exposing Shared Blindspots
To tackle the instance-level asymmetry where competing models succeed and fail on divergent subsets of questions, IA restricts the evaluation strictly to instances where both models achieve identical outcomes: \(\mathcal{D}_{\text{IA}} = \mathcal{D}_{\text{both-right}} \cup \mathcal{D}_{\text{both-wrong}}\). By construction, the empirical accuracy of both models on \(\mathcal{D}_{\text{IA}}\) is identically equal:
Beyond establishing strict equality of predictive success on an instance-by-instance basis, this design yields a clean orthogonal decomposition: \(\mathcal{D}_{\text{both-right}}\) quantifies shared competence, while \(\mathcal{D}_{\text{both-wrong}}\) inspects shared cognitive blindspots. If a nominally superior model continues to assign higher confidence on \(\mathcal{D}_{\text{both-wrong}}\), its ostensible raw global calibration lead merely masks an acute, perilous form of overconfidence on tasks neither system understands. Across all tested model-pair-dataset cells, IA maintains high data retention (median 78.0%, IQR [66.3%, 82.2%], minimum 55.2%), guaranteeing that findings are not artifacts of statistical power degradation.
2. Distribution-Aligned Calibration (DA): Causal Importance Weighting for Full-Sample Outcome Balance
While IA offers immaculate instance-level control, it necessarily discards asymmetric samples where only one model is correct (roughly 22% of observations). To evaluate models across the entire dataset without introducing downsampling variance, DA frames cross-model comparison through the lens of causal observational studies (Rosenbaum & Rubin, 1983). Viewing model identity \(m \in \{A, B\}\) as a treatment variable and instance correctness \(z_i^{(m)} \in \{0, 1\}\) as an observable confounder, DA adopts Horvitz-Thompson density-ratio importance weighting to rebalance model \(M_A\)'s predictions against the target accuracy \(a_B\):
This continuous reweighting satisfies \(\sum_i w_i^{(A)} z_i^{(A)} / \sum_i w_i^{(A)} = a_B\), perfectly matching aggregate outcome composition. Using the normalized weights, weighted calibration metrics \(\text{DA-BS}_{a_A \to a_B}\) and \(\text{DA-ECE}_{a_A \to a_B}\) are computed over all \(N\) original examples. DA retains full statistical coverage while demonstrating outstanding effective precision (median Kish effective sample size ratio \(\text{ESS}/N = 0.897\), with 115 out of 120 experimental cells exceeding the \(\text{ESS}/N \ge 0.5\) diagnostic threshold).
3. Candidate-Aligned Calibration (CA): Symmetrical Candidate Pooling to Disentangle Generation Capability and Self-Preference
Both IA and DA evaluate generative calibration, where models self-assess their own generated answers. However, stronger models inherently generate more coherent and persuasive surface forms, potentially confounding confidence estimation with answer articulateness. CA resolves this by moving to an evaluative paradigm: for each query \(x_i\), answers generated by both systems are pooled into a unified candidate set \(\mathcal{U}_i = \{s_i^{(A)}, s_i^{(B)}\}\). Each model then independently evaluates every candidate \(s \in \mathcal{U}_i\), outputting confidence \(e_i^{(m)}(s) \in [0, 1]\) regarding whether candidate \(s\) is correct according to ground-truth label \(y_i(s) \in \{0, 1\}\). The Candidate-Aligned Brier Score is formulated as:
By evaluating an identical candidate distribution, CA isolates pure evaluative calibration, completely neutralizing generative divergence while explicitly measuring "self-preference bias"โthe systemic tendency of models to inflate confidence for self-generated answers over objectively superior competitor solutions.
Benchmark & Evaluation Protocol¶
The empirical investigation encompasses 14 locally deployed model configurations across four major open-source families: Qwen2.5 (7B and 72B), LLaMA 3.1 (8B and 70B), Qwen3 (8B, 14B, 32B, with enable_thinking toggle), and the open GPT-OSS suite (20B and 120B, with reasoning_effort low/high control), served in bfloat16 via vLLM. Experiments span five diverse benchmarks across two broad cognitive categories: open-ended factual knowledge (FreshQA with 600 questions, TriviaQA with 1,500 questions) and challenging multi-step reasoning (GPQA Diamond with 198 questions, LiveBench Reasoning with 200 questions, MMLU-Pro with 2,841 questions). Three canonical confidence elicitation protocols are systematically scrutinized: Verbalized Confidence (numerical 0โ100 prompt, normalized to \([0, 1]\)), \(P(\text{True})\) (extracting linear probability from binary verification log-probabilities), and Self-Consistency (fraction of agreeing answers across 10 independent sampling runs). Furthermore, Appendix E.7 extends the framework to continuous graded correctness tasks (IF-multi constraint compliance and APPS coding test-case pass rates).
Experimental Results¶
Main Results: Raw Global Calibration vs. ACE-Aligned Metrics Across Benchmarks¶
The table below displays raw task accuracy, raw global calibration metrics, and accuracy-controlled metrics under the ACE framework for representative model pairs (scale comparison and thinking vs. non-thinking modes). All calibration metrics (Brier Score and ECE) are multiplied by 100; lower values indicate superior calibration (\(\downarrow\)). The difference row \(\Delta = \text{Model}_1 - \text{Model}_2\) highlights reversals where the raw winner loses its advantage:
| Dataset | Evaluated Pair (Model 1 vs Model 2) | Elicitation Method | Accuracy (%) | Raw Brier | Raw ECE | DA Brier | DA ECE | IA Brier | IA ECE |
|---|---|---|---|---|---|---|---|---|---|
| TriviaQA | Qwen2.5-7B vs Qwen2.5-72B | Verbalize | 41.4 vs 71.7 | 31.2 vs 20.5 | 32.8 vs 20.6 | 31.2 vs 38.7 | 32.8 vs 40.0 | 25.7 vs 26.7 | 27.0 vs 27.6 |
| TriviaQA | Difference \(\Delta\) (Reversal Detected) | Verbalize | -30.3 | +10.7 (72B wins) | +12.2 (72B wins) | -7.5 (7B wins!) | -7.2 (7B wins!) | -1.0 (7B wins!) | -0.6 (7B wins!) |
| GPQA | Qwen3-8B-nothink vs Qwen3-8B-think | Verbalize | 46.0 vs 61.6 | 41.9 vs 26.8 | 42.5 vs 25.9 | 45.5 vs 45.1 | 45.5 vs 46.4 | 38.0 vs 38.3 | 36.7 vs 37.8 |
| GPQA | Difference \(\Delta\) (Reversal Detected) | Verbalize | -15.6 | +15.2 (think wins) | +16.6 (think wins) | +0.4 (think wins) | -0.9 (nothink!) | -0.3 (nothink!) | -1.1 (nothink!) |
| LiveBench | Qwen3-32B-nothink vs Qwen3-32B-think | Verbalize | 47.0 vs 85.5 | 43.7 vs 12.0 | 44.5 vs 11.7 | 43.7 vs 40.7 | 44.5 vs 43.1 | 18.9 vs 17.3 | 17.3 vs 18.1 |
| LiveBench | Difference \(\Delta\) (Reversal Detected) | Verbalize | -38.5 | +31.7 (think wins) | +32.8 (think wins) | +3.0 (think wins) | +1.4 (think wins) | +1.6 (think wins) | -0.7 (nothink!) |
| FreshQA | LLaMA3.1-8B vs LLaMA3.1-70B | Verbalize | 32.1 vs 49.3 | 48.5 vs 32.1 | 50.7 vs 32.9 | 45.1 vs 50.5 | 47.2 vs 51.9 | 43.7 vs 49.0 | 46.6 vs 50.5 |
| FreshQA | Difference \(\Delta\) (Reversal Detected) | Verbalize | -17.2 | +16.3 (70B wins) | +17.9 (70B wins) | -5.4 (8B wins!) | -4.7 (8B wins!) | -5.3 (8B wins!) | -3.9 (8B wins!) |
| MMLU-Pro | GPT-OSS-20B-high vs GPT-OSS-120B-high | P(True) | 86.7 vs 88.2 | 16.1 vs 21.7 | 16.2 vs 21.8 | 25.3 vs 23.4 | 25.3 vs 23.5 | 15.6 vs 20.2 | 15.7 vs 20.2 |
| MMLU-Pro | Difference \(\Delta\) (Reversal Detected) | P(True) | -1.5 | -5.6 (20B wins) | -5.6 (20B wins) | +1.9 (120B wins!) | +1.8 (120B wins!) | -4.5 (20B wins) | -4.5 (20B wins) |
Analysis: Method-Level and Dataset-Level Ranking Reversal Dynamics¶
Across 120 evaluated comparative configurations, ranking reversals occur with remarkable frequency. The table below delineates the distribution of exclusive and joint ranking reversals across confidence estimation methods relative to raw ECE winners:
| Confidence Method | Exclusive IA RR (%) | Exclusive DA RR (%) | Exclusive CA RR (%) | Joint (DA + IA) RR (%) | Overall No-Reversal Rate NRR (%) |
|---|---|---|---|---|---|
| Verbalized | 7.5 | 2.5 | 7.5 | 35.0 | 27.5 |
| P(True) | 2.5 | 22.5 | 20.0 | 15.0 | 30.0 |
| Self-Consistency | 10.0 | 2.5 | 35.0 | 12.5 | 27.5 |
| Aggregate Overview | 6.7 | 9.2 | 20.8 | 20.0 (DA & IA agree in 79.2%) | 28.3 |
At the dataset level, MMLU-Pro demonstrates the highest stability with a no-reversal rate of 50.0%. In contrast, open-domain knowledge QA tasks FreshQA and TriviaQA are heavily dominated by candidate-aligned reversals (exclusive CA reversal rates of 45.8% and 33.3%, respectively). Paired bootstrap resampling with 2,000 iterations demonstrates that among the 265 original reversals, the median probability of retaining opposite signs is 91.9%, and 122 reversals hold in over 95% of bootstrap resamples, proving that reversals reflect systemic confounded shifts rather than sampling noise.
Key Findings¶
- The Scale Advantage in Calibration is Substantially Illusory: Across model scaling axes (e.g., 7B vs. 72B, 8B vs. 70B), raw global calibration metrics suggest that massive models consistently exhibit superior uncertainty estimation. Under ACE control (specifically DA and IA), these ostensible gaps shrink dramatically by 60%โ90%, and frequently overturn entirely (e.g., Qwen2.5-7B and LLaMA3.1-8B outperform their larger siblings on TriviaQA and FreshQA under controlled evaluation).
- Thinking Modes Improve Accuracy, Not Pure Metacognition: Comparisons between thinking and non-thinking variants (Qwen3 and GPT-OSS) reveal that CoT-driven reasoning gains in raw ECE are mediated primarily by converting incorrect answers to correct ones. On instances where thinking models still fail (\(\mathcal{D}_{\text{both-wrong}}\)), they manifest severe confidence inflation, leading to widespread calibration degradation once accuracy is normalized.
- Accuracy-Gap Threshold Effect: The occurrence of ranking reversals exhibits a non-linear threshold dynamic rather than monotonic growth. When the task accuracy difference between compared models is below 0.1, reversal rates remain low (\(<20\%\)). Once the accuracy gap surpasses 0.1, reversal rates jump sharply to the 40%โ60% range and plateau, indicating that any substantial capability divergence renders raw calibration comparisons inherently suspect.
Highlights & Insights¶
- Rigorous Mathematical Formalization of Accuracy Bias: Proposition 1 proves that under identical conditional confidence distributions, the Brier Score is strictly linear with respect to accuracy \(a\), while Proposition 2 proves that under overconfidence, ECE is monotonically decreasing with \(a\). This theoretical foundation definitively explains why naive global metrics systematically favor higher-accuracy systems.
- Tripartite Alignment Protocol: Rather than proposing an isolated metric, ACE provides a structured diagnostic battery: DA serves as the robust primary reporting standard preserving full sample size; IA serves as an instance-level corroborating baseline exposing blindspot overconfidence; and CA acts as an evaluative stress-test isolating self-preference bias.
- Standard-Setting Practical Recommendations: The findings offer actionable guidelines for the community: standard calibration benchmarks must cease reporting raw cross-model ECE as an authoritative leaderboard metric and instead adopt accuracy-controlled reporting as an indispensable companion.
Limitations & Future Work¶
- Computational Overhead of Candidate Evaluation: The CA view requires quadratic cross-evaluation over merged answer pools, which introduces non-trivial inference latency and API cost in long-context or open-ended generation tasks.
- Empirical Boundary on Proprietary Frontiers: Experiments were conducted rigorously across 14 open-weight configurations; validating these dynamics on proprietary APIs (e.g., Claude 3.5 Sonnet, GPT-4o) remains constrained by the absence of log-probability outputs and unexposed internal reasoning tokens.
- Bridge to Downstream Policy Deployment: ACE functions as a post-hoc diagnostic and comparative framework. Translating accuracy-controlled calibration scores into operational downstream policiesโsuch as confidence-gated rejection, automated routing, and selective hallucination abstentionโremains an open research direction.
Related Work & Insights¶
- vs. Standard Global Calibration (Guo et al., 2017; Nixon et al., 2019): Traditional studies treat raw test-set ECE and Brier Score as directly comparable across different models. ACE reveals that this conflates task capability with calibration quality, formalizing the accuracy confound and proposing principled corrections.
- vs. Marginal Predictive Corrections (Si et al., 2022; Moskvichev & Sejdinovic, 2025): While recent theoretical frameworks like CKCE note distortions caused by marginal predictive distributions, they focus on single-model recalibration or kernel operators. ACE directly targets comparative fairness between models, introducing importance-reweighted and instance-matched operational metrics.
- vs. Studies on Reasoning Model Uncertainty (Zeng et al., 2025; Yoon et al., 2025): Prior work argued that extended reasoning naturally sharpens confidence calibration. ACE reveals that this apparent advantage is overwhelmingly driven by reduced error rates; once accuracy is equalized, thinking models frequently show worse miscalibration on their remaining errors.
Rating¶
- Novelty: โญโญโญโญโญ Formulates the accuracy confound in cross-model LLM calibration through causal principles and identifies the critical ranking reversal phenomenon.
- Soundness & Rigor: โญโญโญโญโญ Comprehensive theoretical limit-case proofs accompanied by 14 model configurations, 5 diverse benchmarks, 3 elicitation protocols, and 2,000-sample bootstrap validation.
- Writing Quality: โญโญโญโญโญ Exceptionally articulate prose, elegant mathematical exposition, and meticulous diagnostic reporting.
- Practical Impact: โญโญโญโญโญ Fundamentally shifts the paradigm of LLM confidence benchmarking and establishes an essential evaluation protocol for trustworthy NLP.