When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Faithful Uncertainty¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/chaochaoxx/ACE
Area: LLM Reasoning
Keywords: LLM Calibration, Accuracy Confounding, Ranking Reversal, Uncertainty Estimation, Evaluation Protocols
TL;DR¶
This paper demonstrates theoretically and empirically that standard global calibration metrics (ECE and Brier Score) are fundamentally confounded by task accuracy, frequently producing ranking reversals where models favored by raw metrics cease to be favored under controlled accuracy; to resolve this, it establishes ACE (Accuracy-Controlled Evaluation) across Instance-, Distribution-, and Candidate-Aligned views for fair cross-model comparisons.
Background & Motivation¶
As Large Language Models (LLMs) are progressively deployed into high-stakes decision-making domains such as medicine, law, and autonomous operations, achieving high empirical task accuracy is no longer sufficient; models must faithfully quantify their predictive certainty so downstream systems know when to trust or verify their outputs. Existing literature routinely evaluates uncertainty calibration using global scalar metrics such as Expected Calibration Error (ECE) and Brier Score (BS), employing them directly for comparative benchmarks across model scales (e.g., comparing 7B to 70B parameter models) or reasoning paradigms (e.g., comparing thinking or chain-of-thought models against non-thinking counterparts). An entrenched yet largely unexamined assumption underpins this standard evaluation practice: that lower global ECE or Brier scores signify fundamentally superior metacognitive confidence estimation.
However, this conventional comparison across heterogeneous models is deeply confounded by variations in underlying task accuracy. Global calibration metrics aggregate over prediction sets that intermingle correct and incorrect outcomes; consequently, their numerical scores are dictated not only by the intrinsic fidelity of the assigned confidence values, but also mechanically by the proportion of correct versus incorrect predictions. In the typical high-confidence and overconfident regime, two models that output identical extreme confidence assignments will diverge in global score simply because the higher-accuracy model makes fewer errors. Across mainstream model families, empirical differences in accuracy exhibit a powerful negative correlation with raw ECE differences (\(\rho = -0.670, p < 10^{-5}\)), raising a fundamental concern that reported calibration advantages of larger and thinking models may merely mirror task accuracy rather than genuine uncertainty calibration.
Critically, this accuracy confounding frequently induces what the authors term "ranking reversal" (RR)—a phenomenon where a model deemed better calibrated under raw global metrics turns out to be less calibrated once accuracy is appropriately controlled. Naive evaluations that report raw global scores systematically favor the stronger task performer, masking severe overconfidence in shared blind spots and skewing comparative claims regarding the calibration benefits of scale and test-time reasoning compute. Core idea: explicit control of task accuracy is an essential prerequisite for cross-model calibration comparisons, realized through ACE via instance-level outcome matching, causal distribution reweighting, and shared candidate evaluation.
Method¶
Overall Architecture¶
ACE (Accuracy-Controlled Evaluation) establishes a rigorous framework consisting of three complementary alignment views designed to disentangle calibration quality from task competence. For generative calibration, where models assign confidence to their own outputs, ACE provides Instance-Aligned (IA) calibration by filtering comparisons strictly to instances where both models share the same outcome, alongside Distribution-Aligned (DA) calibration which applies causal importance reweighting across the entire dataset to equalize aggregate outcome proportions. For evaluative calibration, Candidate-Aligned (CA) calibration neutralizes generative variance and self-preference bias by having models judge an identical, shared pool of candidate responses.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Input Evaluation Set D={(xi, yi)}<br/>Models MA and MB with Acc: aA > aB"] --> B["Theoretical Confounder Diagnosis<br/>Analyze BS linear drift & ECE overconfidence"]
B --> C["Instance-Aligned Calibration (IA)<br/>Filter to shared outcome subset DIA"]
B --> D["Distribution-Aligned Calibration (DA)<br/>Causal importance reweighting to baseline aB"]
B --> E["Candidate-Aligned Calibration (CA)<br/>Evaluate shared candidate pool Ui"]
C --> F["Dual-Perspective Cross-Verification<br/>Audit intrinsic metacognition & shared blind spots"]
D --> F
E --> F
F --> G["Faithful Calibration Ranking<br/>Diagnose and report Ranking Reversal"]
Key Designs¶
1. Instance-Aligned Calibration (IA): Decoupling task success via shared-outcome filtering
When comparing models with unequal cognitive capacity, global aggregation allows the stronger model's higher accuracy to dilute its miscalibration. The Instance-Aligned view addresses this issue by restricting calibration measurement strictly to instances where both models achieve identical correctness: \(D_{\text{both-right}} = \{i: z_i^{(A)}=1 \land z_i^{(B)}=1\}\) and \(D_{\text{both-wrong}} = \{i: z_i^{(A)}=0 \land z_i^{(B)}=0\}\), yielding the unified aligned subset \(D_{\text{IA}} = D_{\text{both-right}} \cup D_{\text{both-wrong}}\). On this subset, both models possess identical empirical accuracy:
Holding both individual instances and outcomes fixed enables direct inspection of confidence fidelity. In particular, isolating \(D_{\text{both-wrong}}\) uncovers whether a nominally stronger model retains high confidence on shared blind spots—questions neither model can solve. If an advanced model assigns aggressively elevated confidence on \(D_{\text{both-wrong}}\), its global calibration lead is an artifact of accuracy masking acute overconfidence. Across 40 evaluated model-pair cells, IA maintains high data retention (median 78.0%, IQR [66.3%, 82.2%], minimum 55.2%), ensuring statistical reliability without data sparsity.
2. Distribution-Aligned Calibration (DA): Causal propensity reweighting over full evaluation samples
While IA delivers immaculate instance matching, it inevitably discards discordant predictions where one model succeeds while the other fails. To preserve 100% sample coverage while eliminating aggregate outcome confounding, the Distribution-Aligned view imports the principle of inverse probability weighting from causal inference (Horvitz-Thompson estimation). In this formulation, model identity acts as the treatment variable \(T \in \{A, B\}\), correctness status \(z_i^{(m)} \in \{0, 1\}\) acts as the confounder \(X\), and calibration loss acts as the observed outcome \(Y\). To adjust model \(M_A\) (\(a_A > a_B\)) to match the target accuracy level \(a_B\), DA computes density ratio importance weights:
Using these weights, the total weighted outcome composition strictly satisfies \(\sum_i w_i^{(A)} z_i^{(A)} / \sum_i w_i^{(A)} = a_B\). Weighted Brier scores (\(\text{DA-BS}_{a_A \to a_B}\)) and weighted ECE (\(\text{DA-ECE}\)) are subsequently calculated. Unlike naive rejection sampling, DA avoids sampling variance and sample reduction. Across 120 experimental cells, the Kish Effective Sample Size ratio (ESS/N) attains a median of 0.897, with 115 out of 120 cells exceeding 0.5, establishing DA as an optimal default metric.
3. Candidate-Aligned Calibration (CA): Standardizing evaluation spaces to isolate self-preference bias
Generative calibration inherently couples a model's confidence with its idiosyncratic generation style, phrasing length, and response distribution. The Candidate-Aligned view eliminates this asymmetry by transforming generative evaluation into candidate verification over a shared hypothesis space. For each prompt \(x_i\), candidate solutions generated by both models are combined into pool \(U_i = \{s_i^{(A)}, s_i^{(B)}\}\). Each model independently scores the correctness probability \(e_i^{(m)}(s) \in [0, 1]\) of every candidate \(s \in U_i\) against verified ground-truth correctness labels \(y_i(s) \in \{0, 1\}\):
By anchoring both models to the exact same candidate distribution, CA isolates pure evaluative calibration. Crucially, CA exposes self-preference bias: cases where a model inflates confidence for its own erroneous derivation while underestimating a superior competing candidate generated by the rival model.
Benchmark & Evaluation Protocol¶
The empirical investigation explores 14 locally deployed model configurations spanning four prominent open-weight architectures across two axes:
1. Model Scale Axis: Qwen2.5 (7B vs. 72B), LLaMA 3.1 (8B vs. 70B), GPT-OSS (20B vs. 120B);
2. Reasoning Mode Axis: Qwen3 (8B/32B under enable_thinking=True/False) and GPT-OSS (20B/120B under reasoning_effort=high/low).
Evaluations span five canonical datasets across knowledge tasks (FreshQA with 600 questions, TriviaQA with 1,500 questions) and advanced reasoning challenges (GPQA Diamond with 198 questions, LiveBench Reasoning subset with 200 questions, and MMLU-Pro with 2,841 questions). Three representative confidence elicitation protocols are systematically compared: Verbalized confidence (0–100 prompt extraction), P(True) verification token probabilities, and Self-Consistency agreement over 10 independent sampling paths. Furthermore, Section E.7 demonstrates the mathematical generalization of ACE to graded correctness metrics evaluated on APPS and IF-multi.
Key Experimental Results¶
Main Results: Pervasive Ranking Reversals Under ACE¶
Table 1 presents the key quantitative comparisons on the challenging reasoning benchmarks GPQA Diamond and LiveBench. The raw baseline figures demonstrate that superior task accuracy consistently yields lower raw ECE, whereas accuracy control via DA and IA systematically diminishes these gaps and induces acute ranking reversals (\(\Delta\) sign inversion).
| Dataset | Model Comparison Pair (M1 vs. M2) | Confidence Method | Accuracy (M1 vs. M2) | Raw ECE (M1 / M2 / \(\Delta\)) | DA ECE (M1 / M2 / \(\Delta\)) | IA ECE (M1 / M2 / \(\Delta\)) | Reversal Status |
|---|---|---|---|---|---|---|---|
| GPQA | Qwen2.5-7B vs. 72B | Verbalize | 33.3% vs. 50.0% | 54.8 / 41.2 / +13.6 | 57.1 / 57.1 / 0.0 | 52.1 / 54.6 / -2.5 | IA Reversal (7B favored) |
| GPQA | LLaMA3.1-8B vs. 70B | P(True) | 18.7% vs. 39.9% | 40.4 / 37.3 / +3.1 | 40.4 / 52.0 / -11.6 | 43.0 / 55.9 / -12.9 | Strong DA+IA Reversal |
| GPQA | Qwen3-8B (NoThink vs. Think) | P(True) | 46.0% vs. 61.6% | 39.3 / 25.9 / +13.4 | 39.3 / 47.9 / -8.6 | 40.2 / 43.6 / -3.4 | Strong DA+IA Reversal |
| GPQA | Qwen3-32B (NoThink vs. Think) | Verbalize | 51.5% vs. 67.7% | 39.2 / 31.9 / +7.3 | 39.2 / 46.2 / -7.1 | 28.9 / 36.0 / -7.1 | Strong DA+IA Reversal |
| LiveBench | Qwen2.5-7B vs. 72B | P(True) | 17.0% vs. 36.5% | 61.4 / 54.3 / +7.1 | 61.4 / 71.6 / -10.2 | 59.6 / 73.5 / -13.9 | Strong DA+IA Reversal |
| LiveBench | Qwen3-8B (NoThink vs. Think) | Verbalize | 38.5% vs. 77.0% | 55.4 / 16.0 / +39.4 | 55.1 / 53.4 / +1.8 | 30.8 / 30.7 / +0.0 | Raw +39.4 advantage erased to parity |
| LiveBench | Qwen3-32B (NoThink vs. Think) | Self-Cons. | 47.0% vs. 85.5% | 8.8 / 17.4 / -8.7 | 8.8 / 8.0 / +0.8 | 27.0 / 9.3 / +17.7 | Extreme shifts across views |
| LiveBench | GPT-OSS-20B (Low vs. High) | Verbalize | 59.5% vs. 89.5% | 8.9 / 6.3 / +2.6 | 8.9 / 14.8 / -5.9 | 16.3 / 7.4 / +8.8 | DA Reversal |
(Note: ECE reported in percentage points; lower values indicate better calibration. \(\Delta = \text{Model}_1 - \text{Model}_2\), meaning \(\Delta > 0\) favors Model 2 and \(\Delta < 0\) favors Model 1.)
Ablation Study & Empirical Analysis¶
Table 2 highlights the structural composition of ranking reversal regimes across all 120 testing conditions (Figure 4) and provides quantitative metrics from the extension to graded correctness (APPS and IF-multi, Table 11).
| Diagnostic Dimension | Breakdown Category | Value / Rate | Concrete Empirical Findings & Implications |
|---|---|---|---|
| Reversal Profile (Figure 4) | No Reversal (NRR) | 28.3% | In fewer than one-third of cases do raw winners remain robustly superior |
| Exclusive CA Reversal | 20.8% | Candidate alignment captures independent evaluative/self-preference dynamics | |
| Joint DA + IA Reversal | 20.0% | DA and IA co-occur frequently, confirming mutual generative consistency | |
| All-Three Reversal | 10.0% | Raw global superiority is completely spurious across all controlled views | |
| DA & IA Agreement Rate | 79.2% | Proves that causal reweighting closely mirrors instance-level matched outcomes | |
| Bootstrap Validation | Paired Bootstrap Resampling | 91.9% Median | 265 original reversals retain sign reversal across 2,000 paired bootstrap resamples |
| Graded Correctness (Table 11) | IF-multi (Constraint Following) | Raw: -5.8 ECE | Qwen2.5-72B initially dominates Qwen2.5-7B under raw metrics |
| IF-multi (IA / DA) | IA: +3.2 / DA: +3.1 | Accuracy control completely reverses ranking to favor Qwen2.5-7B | |
| APPS (Unit Test Pass Rate) | Raw: -8.0 ECE | Raw metrics suggest strong calibration advantage for 72B | |
| APPS (IA / DA) | IA: +5.9 / DA: +8.5 | Controlling test-pass distributions demonstrates 7B is substantially better calibrated |
Key Findings¶
- Calibration Advantages Are Substantially Overstated: Controlling for accuracy reveals reversal rates between 0% and 53% in scale-based comparisons, reaching up to 65% when comparing thinking models against non-thinking models under self-consistency. Thinking regimes primarily enhance raw task correctness rather than genuine introspective self-awareness.
- Accuracy Gaps Exhibit a Distinct Threshold Effect: Reversal rates remain relatively muted (< 20%) when the accuracy gap between compared models is below 0.1. Once the accuracy delta exceeds 0.1, reversal rates surge abruptly to a sustained high plateau of 40%–80%, demonstrating that large capability gaps inevitably corrupt raw calibration signals.
- Shared Blind Spots Expose Dangerous Overconfidence: An analysis of confidence distributions on \(D_{\text{both-wrong}}\) reveals that Verbalized and P(True) methods maintain alarmingly elevated mean confidences (0.72–0.81 and 0.60–0.69, respectively) on questions that neither model can solve. Conversely, Self-Consistency displays noticeably flatter distributions with lower blind-spot means (0.22–0.59), showing far healthier conservatism.
Highlights & Insights¶
- Mathematical Proof of Mechanical Drift: The paper analytically proves that the Brier Score varies linearly with task accuracy (\(BS(a) = L_0 + a \cdot \Delta L\)) under identical conditional confidence distributions, and establishes that ECE shrinks linearly with accuracy in overconfident regimes, formalizing the exact mathematical mechanism behind accuracy confounding.
- Principled Adaptation of Causal Propensity Reweighting: Adapting Horvitz-Thompson causal inverse weighting to calibration benchmarks allows Distribution-Aligned calibration to retain 100% of the evaluation sample while achieving effective outcome balance, providing a reusable, variance-controlled metric for LLM evaluation.
- Divergence of Generative vs. Evaluative Calibration: Revealing that CA reversals frequently diverge from IA and DA highlights an essential cognitive asymmetry in LLMs: how a model values its own reasoning path differs sharply from how it evaluates competing alternatives, driven by deep-seated self-preference bias.
Limitations & Future Work¶
- Restricted to Open-Weight Architectures: The experimental validation centers on 14 open-weight configurations; extending ACE across leading proprietary frontier systems (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) will further clarify whether frontier commercial post-training alters these confounding dynamics.
- Decoupled from Downstream Operational Routing: ACE currently acts as an analytical evaluation framework; translating accuracy-controlled calibration scores into operational gating thresholds for selective prediction, deferral policies, or multi-agent routing remains an open engineering avenue.
- Discretization Dependency in Continuous Calibration: The graded correctness formulation currently relies on discrete mappings or equal-width stratification; developing kernelized or non-parametric continuous propensity estimators for free-form reasoning represents an important methodological next step.
Related Work & Insights¶
- vs. Post-Hoc Calibration Techniques (Guo et al., 2017; Platt, 1999): Prior methods focus on optimizing post-hoc mapping functions (such as temperature scaling) to rectify miscalibration for a single static model. ACE reveals that even perfectly scaled models cannot be directly compared across different performance tiers without explicit accuracy conditioning.
- vs. Marginal and Conditional Calibration Metrics (Minderer et al., 2021; Moskvichev & Sejdinovic, 2025): While theoretical constructs like CKCE recognize that marginal distribution shifts distort cross-model evaluations, ACE establishes an intuitive, causally grounded suite of protocols (IA, DA, CA) that make accuracy-controlled benchmarking practical and interpretable for the NLP community.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the accuracy confounding mechanism and ranking reversal phenomenon with exemplary conceptual clarity.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tests 14 models across 5 benchmarks, 3 elicitation modes, bootstrap stability, and continuous task extensions.
- Writing Quality: ⭐⭐⭐⭐⭐ Beautifully structured prose with pristine mathematical derivations, cohesive narrative flow, and informative visuals.
- Value: ⭐⭐⭐⭐⭐ Delivers an essential paradigm shift for trustworthy AI evaluation, correcting widespread benchmarking misconceptions regarding model scale and reasoning calibration.