Skip to content

When Calibration Rankings Reverse: Accuracy-Controlled Evaluation for Faithful Uncertainty

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/chaochaoxx/ACE
Area: LLM Reasoning
Keywords: LLM Calibration, Accuracy Confounding, Ranking Reversal, Uncertainty Estimation, Evaluation Protocols

TL;DR

This paper demonstrates theoretically and empirically that standard global calibration metrics (ECE and Brier Score) are fundamentally confounded by task accuracy, frequently producing ranking reversals where models favored by raw metrics cease to be favored under controlled accuracy; to resolve this, it establishes ACE (Accuracy-Controlled Evaluation) across Instance-, Distribution-, and Candidate-Aligned views for fair cross-model comparisons.

Background & Motivation

As Large Language Models (LLMs) are progressively deployed into high-stakes decision-making domains such as medicine, law, and autonomous operations, achieving high empirical task accuracy is no longer sufficient; models must faithfully quantify their predictive certainty so downstream systems know when to trust or verify their outputs. Existing literature routinely evaluates uncertainty calibration using global scalar metrics such as Expected Calibration Error (ECE) and Brier Score (BS), employing them directly for comparative benchmarks across model scales (e.g., comparing 7B to 70B parameter models) or reasoning paradigms (e.g., comparing thinking or chain-of-thought models against non-thinking counterparts). An entrenched yet largely unexamined assumption underpins this standard evaluation practice: that lower global ECE or Brier scores signify fundamentally superior metacognitive confidence estimation.

However, this conventional comparison across heterogeneous models is deeply confounded by variations in underlying task accuracy. Global calibration metrics aggregate over prediction sets that intermingle correct and incorrect outcomes; consequently, their numerical scores are dictated not only by the intrinsic fidelity of the assigned confidence values, but also mechanically by the proportion of correct versus incorrect predictions. In the typical high-confidence and overconfident regime, two models that output identical extreme confidence assignments will diverge in global score simply because the higher-accuracy model makes fewer errors. Across mainstream model families, empirical differences in accuracy exhibit a powerful negative correlation with raw ECE differences (\(\rho = -0.670, p < 10^{-5}\)), raising a fundamental concern that reported calibration advantages of larger and thinking models may merely mirror task accuracy rather than genuine uncertainty calibration.

Critically, this accuracy confounding frequently induces what the authors term "ranking reversal" (RR)—a phenomenon where a model deemed better calibrated under raw global metrics turns out to be less calibrated once accuracy is appropriately controlled. Naive evaluations that report raw global scores systematically favor the stronger task performer, masking severe overconfidence in shared blind spots and skewing comparative claims regarding the calibration benefits of scale and test-time reasoning compute. Core idea: explicit control of task accuracy is an essential prerequisite for cross-model calibration comparisons, realized through ACE via instance-level outcome matching, causal distribution reweighting, and shared candidate evaluation.

Method

Overall Architecture

ACE (Accuracy-Controlled Evaluation) establishes a rigorous framework consisting of three complementary alignment views designed to disentangle calibration quality from task competence. For generative calibration, where models assign confidence to their own outputs, ACE provides Instance-Aligned (IA) calibration by filtering comparisons strictly to instances where both models share the same outcome, alongside Distribution-Aligned (DA) calibration which applies causal importance reweighting across the entire dataset to equalize aggregate outcome proportions. For evaluative calibration, Candidate-Aligned (CA) calibration neutralizes generative variance and self-preference bias by having models judge an identical, shared pool of candidate responses.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
    A["Input Evaluation Set D={(xi, yi)}<br/>Models MA and MB with Acc: aA > aB"] --> B["Theoretical Confounder Diagnosis<br/>Analyze BS linear drift & ECE overconfidence"]
    B --> C["Instance-Aligned Calibration (IA)<br/>Filter to shared outcome subset DIA"]
    B --> D["Distribution-Aligned Calibration (DA)<br/>Causal importance reweighting to baseline aB"]
    B --> E["Candidate-Aligned Calibration (CA)<br/>Evaluate shared candidate pool Ui"]
    C --> F["Dual-Perspective Cross-Verification<br/>Audit intrinsic metacognition & shared blind spots"]
    D --> F
    E --> F
    F --> G["Faithful Calibration Ranking<br/>Diagnose and report Ranking Reversal"]

Key Designs

1. Instance-Aligned Calibration (IA): Decoupling task success via shared-outcome filtering

When comparing models with unequal cognitive capacity, global aggregation allows the stronger model's higher accuracy to dilute its miscalibration. The Instance-Aligned view addresses this issue by restricting calibration measurement strictly to instances where both models achieve identical correctness: \(D_{\text{both-right}} = \{i: z_i^{(A)}=1 \land z_i^{(B)}=1\}\) and \(D_{\text{both-wrong}} = \{i: z_i^{(A)}=0 \land z_i^{(B)}=0\}\), yielding the unified aligned subset \(D_{\text{IA}} = D_{\text{both-right}} \cup D_{\text{both-wrong}}\). On this subset, both models possess identical empirical accuracy:

\[\text{Acc}_{D_{\text{IA}}}(M_A) = \text{Acc}_{D_{\text{IA}}}(M_B) = \frac{|D_{\text{both-right}}|}{|D_{\text{IA}}|}\]

Holding both individual instances and outcomes fixed enables direct inspection of confidence fidelity. In particular, isolating \(D_{\text{both-wrong}}\) uncovers whether a nominally stronger model retains high confidence on shared blind spots—questions neither model can solve. If an advanced model assigns aggressively elevated confidence on \(D_{\text{both-wrong}}\), its global calibration lead is an artifact of accuracy masking acute overconfidence. Across 40 evaluated model-pair cells, IA maintains high data retention (median 78.0%, IQR [66.3%, 82.2%], minimum 55.2%), ensuring statistical reliability without data sparsity.

2. Distribution-Aligned Calibration (DA): Causal propensity reweighting over full evaluation samples

While IA delivers immaculate instance matching, it inevitably discards discordant predictions where one model succeeds while the other fails. To preserve 100% sample coverage while eliminating aggregate outcome confounding, the Distribution-Aligned view imports the principle of inverse probability weighting from causal inference (Horvitz-Thompson estimation). In this formulation, model identity acts as the treatment variable \(T \in \{A, B\}\), correctness status \(z_i^{(m)} \in \{0, 1\}\) acts as the confounder \(X\), and calibration loss acts as the observed outcome \(Y\). To adjust model \(M_A\) (\(a_A > a_B\)) to match the target accuracy level \(a_B\), DA computes density ratio importance weights:

\[w_i^{(A)} = \begin{cases} \frac{a_B}{a_A}, & \text{if } z_i^{(A)} = 1 \\ \frac{1 - a_B}{1 - a_A}, & \text{if } z_i^{(A)} = 0 \end{cases}\]

Using these weights, the total weighted outcome composition strictly satisfies \(\sum_i w_i^{(A)} z_i^{(A)} / \sum_i w_i^{(A)} = a_B\). Weighted Brier scores (\(\text{DA-BS}_{a_A \to a_B}\)) and weighted ECE (\(\text{DA-ECE}\)) are subsequently calculated. Unlike naive rejection sampling, DA avoids sampling variance and sample reduction. Across 120 experimental cells, the Kish Effective Sample Size ratio (ESS/N) attains a median of 0.897, with 115 out of 120 cells exceeding 0.5, establishing DA as an optimal default metric.

3. Candidate-Aligned Calibration (CA): Standardizing evaluation spaces to isolate self-preference bias

Generative calibration inherently couples a model's confidence with its idiosyncratic generation style, phrasing length, and response distribution. The Candidate-Aligned view eliminates this asymmetry by transforming generative evaluation into candidate verification over a shared hypothesis space. For each prompt \(x_i\), candidate solutions generated by both models are combined into pool \(U_i = \{s_i^{(A)}, s_i^{(B)}\}\). Each model independently scores the correctness probability \(e_i^{(m)}(s) \in [0, 1]\) of every candidate \(s \in U_i\) against verified ground-truth correctness labels \(y_i(s) \in \{0, 1\}\):

\[\text{CA-BS}(M_m) = \frac{\sum_{i} \sum_{s \in U_i} \left(e_i^{(m)}(s) - y_i(s)\right)^2}{\sum_i |U_i|}\]

By anchoring both models to the exact same candidate distribution, CA isolates pure evaluative calibration. Crucially, CA exposes self-preference bias: cases where a model inflates confidence for its own erroneous derivation while underestimating a superior competing candidate generated by the rival model.

Benchmark & Evaluation Protocol

The empirical investigation explores 14 locally deployed model configurations spanning four prominent open-weight architectures across two axes: 1. Model Scale Axis: Qwen2.5 (7B vs. 72B), LLaMA 3.1 (8B vs. 70B), GPT-OSS (20B vs. 120B); 2. Reasoning Mode Axis: Qwen3 (8B/32B under enable_thinking=True/False) and GPT-OSS (20B/120B under reasoning_effort=high/low).

Evaluations span five canonical datasets across knowledge tasks (FreshQA with 600 questions, TriviaQA with 1,500 questions) and advanced reasoning challenges (GPQA Diamond with 198 questions, LiveBench Reasoning subset with 200 questions, and MMLU-Pro with 2,841 questions). Three representative confidence elicitation protocols are systematically compared: Verbalized confidence (0–100 prompt extraction), P(True) verification token probabilities, and Self-Consistency agreement over 10 independent sampling paths. Furthermore, Section E.7 demonstrates the mathematical generalization of ACE to graded correctness metrics evaluated on APPS and IF-multi.

Key Experimental Results

Main Results: Pervasive Ranking Reversals Under ACE

Table 1 presents the key quantitative comparisons on the challenging reasoning benchmarks GPQA Diamond and LiveBench. The raw baseline figures demonstrate that superior task accuracy consistently yields lower raw ECE, whereas accuracy control via DA and IA systematically diminishes these gaps and induces acute ranking reversals (\(\Delta\) sign inversion).

Dataset Model Comparison Pair (M1 vs. M2) Confidence Method Accuracy (M1 vs. M2) Raw ECE (M1 / M2 / \(\Delta\)) DA ECE (M1 / M2 / \(\Delta\)) IA ECE (M1 / M2 / \(\Delta\)) Reversal Status
GPQA Qwen2.5-7B vs. 72B Verbalize 33.3% vs. 50.0% 54.8 / 41.2 / +13.6 57.1 / 57.1 / 0.0 52.1 / 54.6 / -2.5 IA Reversal (7B favored)
GPQA LLaMA3.1-8B vs. 70B P(True) 18.7% vs. 39.9% 40.4 / 37.3 / +3.1 40.4 / 52.0 / -11.6 43.0 / 55.9 / -12.9 Strong DA+IA Reversal
GPQA Qwen3-8B (NoThink vs. Think) P(True) 46.0% vs. 61.6% 39.3 / 25.9 / +13.4 39.3 / 47.9 / -8.6 40.2 / 43.6 / -3.4 Strong DA+IA Reversal
GPQA Qwen3-32B (NoThink vs. Think) Verbalize 51.5% vs. 67.7% 39.2 / 31.9 / +7.3 39.2 / 46.2 / -7.1 28.9 / 36.0 / -7.1 Strong DA+IA Reversal
LiveBench Qwen2.5-7B vs. 72B P(True) 17.0% vs. 36.5% 61.4 / 54.3 / +7.1 61.4 / 71.6 / -10.2 59.6 / 73.5 / -13.9 Strong DA+IA Reversal
LiveBench Qwen3-8B (NoThink vs. Think) Verbalize 38.5% vs. 77.0% 55.4 / 16.0 / +39.4 55.1 / 53.4 / +1.8 30.8 / 30.7 / +0.0 Raw +39.4 advantage erased to parity
LiveBench Qwen3-32B (NoThink vs. Think) Self-Cons. 47.0% vs. 85.5% 8.8 / 17.4 / -8.7 8.8 / 8.0 / +0.8 27.0 / 9.3 / +17.7 Extreme shifts across views
LiveBench GPT-OSS-20B (Low vs. High) Verbalize 59.5% vs. 89.5% 8.9 / 6.3 / +2.6 8.9 / 14.8 / -5.9 16.3 / 7.4 / +8.8 DA Reversal

(Note: ECE reported in percentage points; lower values indicate better calibration. \(\Delta = \text{Model}_1 - \text{Model}_2\), meaning \(\Delta > 0\) favors Model 2 and \(\Delta < 0\) favors Model 1.)

Ablation Study & Empirical Analysis

Table 2 highlights the structural composition of ranking reversal regimes across all 120 testing conditions (Figure 4) and provides quantitative metrics from the extension to graded correctness (APPS and IF-multi, Table 11).

Diagnostic Dimension Breakdown Category Value / Rate Concrete Empirical Findings & Implications
Reversal Profile (Figure 4) No Reversal (NRR) 28.3% In fewer than one-third of cases do raw winners remain robustly superior
Exclusive CA Reversal 20.8% Candidate alignment captures independent evaluative/self-preference dynamics
Joint DA + IA Reversal 20.0% DA and IA co-occur frequently, confirming mutual generative consistency
All-Three Reversal 10.0% Raw global superiority is completely spurious across all controlled views
DA & IA Agreement Rate 79.2% Proves that causal reweighting closely mirrors instance-level matched outcomes
Bootstrap Validation Paired Bootstrap Resampling 91.9% Median 265 original reversals retain sign reversal across 2,000 paired bootstrap resamples
Graded Correctness (Table 11) IF-multi (Constraint Following) Raw: -5.8 ECE Qwen2.5-72B initially dominates Qwen2.5-7B under raw metrics
IF-multi (IA / DA) IA: +3.2 / DA: +3.1 Accuracy control completely reverses ranking to favor Qwen2.5-7B
APPS (Unit Test Pass Rate) Raw: -8.0 ECE Raw metrics suggest strong calibration advantage for 72B
APPS (IA / DA) IA: +5.9 / DA: +8.5 Controlling test-pass distributions demonstrates 7B is substantially better calibrated

Key Findings

  • Calibration Advantages Are Substantially Overstated: Controlling for accuracy reveals reversal rates between 0% and 53% in scale-based comparisons, reaching up to 65% when comparing thinking models against non-thinking models under self-consistency. Thinking regimes primarily enhance raw task correctness rather than genuine introspective self-awareness.
  • Accuracy Gaps Exhibit a Distinct Threshold Effect: Reversal rates remain relatively muted (< 20%) when the accuracy gap between compared models is below 0.1. Once the accuracy delta exceeds 0.1, reversal rates surge abruptly to a sustained high plateau of 40%–80%, demonstrating that large capability gaps inevitably corrupt raw calibration signals.
  • Shared Blind Spots Expose Dangerous Overconfidence: An analysis of confidence distributions on \(D_{\text{both-wrong}}\) reveals that Verbalized and P(True) methods maintain alarmingly elevated mean confidences (0.72–0.81 and 0.60–0.69, respectively) on questions that neither model can solve. Conversely, Self-Consistency displays noticeably flatter distributions with lower blind-spot means (0.22–0.59), showing far healthier conservatism.

Highlights & Insights

  • Mathematical Proof of Mechanical Drift: The paper analytically proves that the Brier Score varies linearly with task accuracy (\(BS(a) = L_0 + a \cdot \Delta L\)) under identical conditional confidence distributions, and establishes that ECE shrinks linearly with accuracy in overconfident regimes, formalizing the exact mathematical mechanism behind accuracy confounding.
  • Principled Adaptation of Causal Propensity Reweighting: Adapting Horvitz-Thompson causal inverse weighting to calibration benchmarks allows Distribution-Aligned calibration to retain 100% of the evaluation sample while achieving effective outcome balance, providing a reusable, variance-controlled metric for LLM evaluation.
  • Divergence of Generative vs. Evaluative Calibration: Revealing that CA reversals frequently diverge from IA and DA highlights an essential cognitive asymmetry in LLMs: how a model values its own reasoning path differs sharply from how it evaluates competing alternatives, driven by deep-seated self-preference bias.

Limitations & Future Work

  • Restricted to Open-Weight Architectures: The experimental validation centers on 14 open-weight configurations; extending ACE across leading proprietary frontier systems (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) will further clarify whether frontier commercial post-training alters these confounding dynamics.
  • Decoupled from Downstream Operational Routing: ACE currently acts as an analytical evaluation framework; translating accuracy-controlled calibration scores into operational gating thresholds for selective prediction, deferral policies, or multi-agent routing remains an open engineering avenue.
  • Discretization Dependency in Continuous Calibration: The graded correctness formulation currently relies on discrete mappings or equal-width stratification; developing kernelized or non-parametric continuous propensity estimators for free-form reasoning represents an important methodological next step.
  • vs. Post-Hoc Calibration Techniques (Guo et al., 2017; Platt, 1999): Prior methods focus on optimizing post-hoc mapping functions (such as temperature scaling) to rectify miscalibration for a single static model. ACE reveals that even perfectly scaled models cannot be directly compared across different performance tiers without explicit accuracy conditioning.
  • vs. Marginal and Conditional Calibration Metrics (Minderer et al., 2021; Moskvichev & Sejdinovic, 2025): While theoretical constructs like CKCE recognize that marginal distribution shifts distort cross-model evaluations, ACE establishes an intuitive, causally grounded suite of protocols (IA, DA, CA) that make accuracy-controlled benchmarking practical and interpretable for the NLP community.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the accuracy confounding mechanism and ranking reversal phenomenon with exemplary conceptual clarity.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tests 14 models across 5 benchmarks, 3 elicitation modes, bootstrap stability, and continuous task extensions.
  • Writing Quality: ⭐⭐⭐⭐⭐ Beautifully structured prose with pristine mathematical derivations, cohesive narrative flow, and informative visuals.
  • Value: ⭐⭐⭐⭐⭐ Delivers an essential paradigm shift for trustworthy AI evaluation, correcting widespread benchmarking misconceptions regarding model scale and reasoning calibration.