Quantile‑Adaptive Temperature Scaling for Confidence Calibration

Conference: ECCV2026
Paper: ECCV Paper
Area: Others / Confidence Calibration
Keywords: post-hoc calibration, empirical quantiles, temperature scaling, distribution shift, long-tailed learning

TL;DR

QaTS chooses a sample's temperature from its confidence quantile in the calibration set, fitting only two positive parameters to improve calibration while preserving class predictions; on CIFAR-100 with RN-50, for example, ECE falls from 5.32 with standard TS to 2.56.

Background & Motivation

Correct classification does not imply reliable confidence: a network that frequently outputs high probabilities may be correct substantially less often than those probabilities suggest. Calibration asks whether average correctness among predictions with similar confidence matches their reported probabilities, rather than simply increasing Top-1 accuracy. This distinction matters in medical decisions and risk estimation because downstream systems may use probabilities to decide whether to defer to a human or proceed. Training-time strategies such as Focal Loss and Label Smoothing can discourage overconfidence, but require intervention in classifier training. Post-hoc calibration instead freezes an existing model and adjusts its outputs using a labeled calibration set, making it suitable for deployed networks. Temperature Scaling (TS), which divides every logit by a shared positive scalar, is a simple baseline that preserves class predictions.

The limitation of TS is that one temperature must accommodate samples with different behavior across the confidence distribution. Figure 1 finds that standard classification often has its largest correctness-confidence discrepancies at low quantiles, whereas the dominant discrepancies move toward intermediate quantiles in long-tailed classification. Here, low confidence is relative to other predictions from the same model and does not imply that this group is free from overconfidence. A group of predictions can be less certain than the highest-confidence samples while still exceeding its own empirical accuracy, so probability softening remains appropriate. Building a temperature directly from absolute confidence may also make it sensitive to score-scale changes associated with architectures, class counts, and distribution shifts. The authors therefore consider a sample's relative position within the calibration confidence distribution instead of adding another feature-conditioned temperature network.

QaTS does not split the data into many groups and fit a separate temperature for each; it uses an empirical cumulative distribution function to obtain a quantile coordinate and then learns a simple temperature curve. This allows different correction strengths for low- and high-quantile samples while adding only two learnable parameters. The simple curve also imposes a clear restriction: temperature must decrease monotonically with quantile, so it cannot freely target an error peak confined to the middle of the distribution. Core Idea: replace absolute scores as the temperature input with empirical calibration-distribution quantiles, using a low-capacity monotone linear function for sample-wise calibration while leaving class decisions unchanged.

Method

Overall Architecture

Inputs are the complete logit vectors from a frozen classifier and a separate labeled calibration dataset. Preparation computes the maximum uncalibrated softmax probability for each calibration sample and stores its empirical CDF; fitting updates only the temperature function's two parameters. At test time, the original logits provide confidence, the same calibration CDF supplies its quantile, and the resulting temperature scales all logits for that sample. The output remains a complete class-probability distribution, not merely a substituted maximum-confidence value. The method comprises empirical quantile mapping, monotone linear temperature, and order-preserving calibration, without retraining the classifier.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Frozen-model logits"] --> B["Empirical quantile<br/>mapping"]
    F["Calibration confidence<br/>Stored empirical CDF"] --> B
    B --> C["Monotone linear<br/>temperature"]
    G["Calibration labels"] -.->|NLL fits a,b only| C
    C --> D["Order-preserving<br/>calibration"]
    A -->|Original logits| D
    D --> E["Calibrated probabilities<br/>Unchanged class predictions"]

Label supervision in the diagram is used only during calibration; test samples do not require ground-truth labels. Testing does not require collecting labeled samples from every new target domain before fitting a temperature. This deployment setup differs both from target-domain recalibration and from generating multiple augmented test views and averaging predictions.

Key Designs

1. Empirical quantile mapping: describe confidence by its position in the calibration distribution

For each calibration sample, take the maximum original softmax probability as confidence and sort these values. The empirical CDF counts the fraction of calibration samples with confidence no greater than the current value, so an identical absolute probability can occupy different quantile positions for different models. The mechanism in Equations (5)–(6), page 6, can be expressed using the following notation, where \(N\) is the number of calibration samples:

\[ q(x)=F_{\mathrm{conf}}(\operatorname{conf}(x)),\qquad F_{\mathrm{conf}}(u)=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\operatorname{conf}(x_i)\leq u\}. \]

For example, \(q(x)=0.2\) means that approximately 20% of calibration samples have confidence no greater than the current sample, not that its prediction has a 20% probability of being correct. This coordinate replaces absolute probability intervals with relative distributional position, making calibration discrepancies easier to inspect along a quantile axis. The paper also rewrites population ECE in quantile coordinates; under suitable regularity conditions, this is an equivalent change of variables that leaves ECE unchanged. Consequently, a uniform quantile axis does not automatically increase the total metric weight of rare samples, nor does it turn the metric into a macro-average over classes.

The empirical CDF is a step function: tied confidence values receive identical quantiles, and strict uniformity is not guaranteed with finite samples. Although the temperature function is linear in quantile coordinates, its composition with raw confidence remains a sample-dependent mapping determined by the calibration distribution. Testing uses the stored calibration reference distribution; the paper's definition does not rerank every test batch to assign fresh quantiles. If only test scores change while the reference CDF remains fixed, quantile values can change even when the ordering among test samples is preserved. The authors' rank-stability argument should therefore be read as motivation for empirical robustness, not as a strict invariance theorem for arbitrary domain shifts.

2. Monotone linear temperature: constrain correction strength with two positive parameters

After obtaining a quantile, QaTS uses one shared monotonically decreasing linear function rather than storing independent temperatures for separate quantile intervals. Equation (8), page 7, defines:

\[ T(x)=a\bigl(1-q(x)\bigr)+b,\qquad a>0,\quad b>0. \]

Here, \(b\) sets the base temperature near the highest quantile, while \(a\) controls the additional temperature when moving from high to low quantiles. Temperature therefore lies in \([b,a+b]\), and low-quantile samples never receive a smaller temperature than high-quantile samples. A higher temperature pulls class probabilities toward a uniform distribution, reducing maximum confidence; this suits the low-quantile overconfidence observed in the paper. Compared with a high-capacity auxiliary network, two parameters restrict how freely the calibrator can fit limited validation data.

A smaller temperature at high quantiles must be distinguished from guaranteed sharpening of high-quantile predictions. Sharpening relative to the original probabilities occurs only below temperature 1; when \(b>1\), even the highest quantile is softened, just less strongly. All reported \(b\) values in Table 6 exceed 1, so the abstract's description of sharpening high-confidence predictions should not be treated as the operation performed in those settings. As \(a\) approaches zero, QaTS approaches TS with a shared temperature \(b\), but the strictly positive constraint does not imply that it attains \(a=0\). The line also cannot independently raise middle-quantile temperatures while lowering both extremes, which is an expressivity limit of its low-capacity design.

3. Order-preserving calibration: fit probabilities with labels without changing the winning class

All class logits for a sample are divided by the same positive temperature before applying softmax again, rather than adjusting only the predicted class probability. Equation (9), page 7, gives the probability transformation:

\[ \widetilde p_k(x)=\frac{\exp(\ell_k(x)/T(x))}{\sum_{j=1}^{K}\exp(\ell_j(x)/T(x))}. \]

Positive scaling leaves the ordering of logits within a sample unchanged, so the maximum-probability class is preserved. This explains why ECE improvements need not sacrifice classification accuracy and why applying the same principle to individual segmentation predictions can preserve the class mask. The guarantee concerns within-sample class ordering, not confidence ordering across different samples. Downstream rejection curves and cross-sample risk rankings therefore still require separate evaluation.

Calibration freezes the original logits and their quantiles, using ground-truth labels to supervise the entire scaled probability distribution. Because binned ECE is inconvenient to differentiate, the authors optimize differentiable negative log-likelihood (NLL), rather than directly minimizing the ECE reported in the tables. Using the ground-truth class index, Equation (10), page 7, has the equivalent objective:

\[ \min_{a,b>0}\; -\frac{1}{N}\sum_{i=1}^{N}\log\widetilde p_{y_i}(x_i;a,b). \]

NLL is a strictly proper scoring rule, but optimizing it on a finite calibration set within a restricted temperature family does not guarantee lower test ECE. The experiments support practical improvements on these tasks, not a universal equivalence between the optimization objective and stepwise monotonic ECE reduction. The available main text specifies positivity constraints and the loss but does not fully specify the optimizer, learning rate, positive-parameter implementation, or calibration-set sizes. Training epochs, a softplus parameterization, and runtime measurements are therefore not invented here.

A Worked Example

Use the fitted parameters from Table 6 for CIFAR-100-LT, R-100, RN-50: \(a=1.6389\) and \(b=2.3315\). Suppose querying the calibration CDF gives \(q=0.2\) and \(q=0.8\) for two test samples; these quantiles illustrate the mechanism and are not additional experimental samples reported by the paper. Substitution into the temperature formula gives approximately \(3.6426\) and \(2.6593\), so the first probability distribution is softened more strongly. Both temperatures exceed 1, meaning that neither sample is sharpened; only the correction strength differs. Each complete logit vector is then divided by its corresponding temperature before softmax; specific calibrated probabilities cannot be calculated without the actual logits. The process neither needs to know whether these test predictions are correct nor changes either sample's original maximum-logit class.

Key Experimental Results

Main Results

The paper reports 162 settings spanning standard and long-tailed visual classification, synthetic distribution shifts, medical classification, text classification, and semantic segmentation. All architectures and datasets use 15 bins for ECE; the tables below retain the paper's numerical ECE scale, with lower values better. ECE sums the absolute difference between average accuracy and average confidence in each confidence bin, weighted by that bin's fraction of samples. FeatClip modifies class predictions, so ECE comparisons must be interpreted together with accuracy changes.

Selected results from Table 1, page 10, all reporting standard-classification ECE:

Dataset	Backbone	Uncalibrated	TS	IR	FeatClip	QaTS
CIFAR-10	RN-50	1.71	1.23	0.64	0.55	0.31
CIFAR-10	ViT-S	1.80	1.01	0.45	0.62	0.51
CIFAR-100	RN-50	8.62	5.32	3.08	3.01	2.56
CIFAR-100	ViT-B	10.79	9.43	4.67	2.57	2.26
ImageNet	RN-50	3.71	2.26	2.72	1.67	1.58
ImageNet	ViT-B	5.77	3.28	2.80	2.93	2.49

QaTS is not first in every standard setting: IR achieves 0.45 on CIFAR-10 with ViT-S, compared with 0.51 for QaTS. On CIFAR-100 with RN-50, QaTS reduces ECE by 2.76 points relative to TS and by 0.45 points relative to FeatClip. On ImageNet with ViT-B, the strongest competing entry is IR at 2.80, giving a gap of 0.31; the original table annotates 0.44, which matches the difference from FeatClip and should not be repeated as the gap to the runner-up.

Selected results from Table 3, page 12: ViT-B/16 is calibrated on clean CIFAR-100 and directly tested on CIFAR-100-C, averaging over 15 corruption types at each severity. QaTS does not re-estimate parameters, whereas HFE requires OOD samples from each target domain, so their information access differs.

Corruption severity	Uncalibrated	TS	FeatClip	QaTS	HFE
1	15.66	13.85	3.69	3.35	5.44
2	19.00	16.79	4.69	4.24	7.34
3	22.55	19.92	5.78	5.42	9.42
4	26.69	23.64	7.70	7.20	12.12
5	32.96	29.27	10.65	9.89	15.62

Ablation Study

Table 7, page 15, retains the quantile mechanism and compares linear temperature with monotone piecewise-linear temperatures using different segment counts, all with ViT-B/16. This is therefore an ablation of temperature-function capacity, not removal of the empirical CDF, and cannot independently quantify the contribution of quantiles versus raw probabilities. Table 7 abbreviates its columns as C100-LT and C100-C; its uncalibrated and linear rows match the main text's R-100 and severity-5 values, so the original column names are retained without inventing additional protocol details.

Temperature function	C100 ECE	C100-LT ECE	C100-C ECE
Uncalibrated	10.79	52.51	32.96
Linear QaTS	2.26	8.37	9.89
Piecewise linear, K=2	2.25	8.44	10.14
Piecewise linear, K=4	2.24	8.68	10.28
Piecewise linear, K=10	2.25	8.76	10.64

Increasing capacity gives only a small clean-data gain: K=4 moves from 2.26 to 2.24, but deteriorates to 8.68 and 10.28 in the other columns. The corruption ECE for K=10 exceeds the linear form by 0.75, supporting the authors' interpretation that simple parameterization generalizes better without directly proving the overfitting mechanism. Table 6 also shows RN-50's \(a\) increasing from 0.0438 on balanced data to 1.6389 at R-100, while \(b\) rises from 1.5960 to 2.3315, indicating stronger learned quantile-dependent correction under long-tailed conditions.

Key Findings

• Long-tailed results also have exceptions: Table 2 reports 5.06 for QaTS and 4.78 for FeatClip on R-100 with RN-50, so QaTS is not best in every long-tailed setting.
• In Table 5, VOC-2012 with DLV3-R50 achieves ECE 2.94 versus 3.69 for TS; NewsGroup-20 with BERT-B achieves 3.80 versus 5.42 for TS, showing that the mechanism is not limited to image-level classification.
• The training-loss analysis on page 14 reports CIFAR-100, RN-50 ECE of 8.62 for the CE baseline, 5.54 after Brier training, and 2.72 after adding QaTS, supporting complementarity between post-hoc and training-time calibration.
• Across the 161 classification settings in Figure 2, the authors report higher accuracy for QaTS than FeatClip in 94% of settings; this primarily reflects preservation of the original class decisions, not improved accuracy over the uncalibrated model.

Highlights & Insights

• Quantiles are the calibrator's input representation, not an additional classifier. The empirical CDF captures the reference distribution while the learnable component remains restricted to two parameters.
• Strength and shape are interpretable separately: \(b\) represents base softening and \(a\) the additional difference across quantiles. This makes parameter analysis easier to interpret than a black-box temperature network.
• The piecewise-linear ablation shows that lower clean-validation calibration error need not imply greater robustness on long-tailed or corrupted data. Calibrator selection must also consider generalization.

Limitations & Future Work

• A fixed reference CDF does not guarantee unchanged quantile values when only test scores undergo a monotone transformation, and arbitrary logit transformations need not preserve cross-sample confidence ordering; distribution-shift results are evidence for specific settings.
• The decreasing-temperature prior matches the observed trend but may not suit low-quantile underconfidence or nonmonotonic error patterns. More flexible alternatives should be studied while controlling calibration sample size and generalization risk.
• The paper mainly reports binned ECE, and the available main text does not provide multi-seed error bars or confidence intervals; small numerical gaps cannot automatically be interpreted as statistically significant improvements.
• The supplied text includes the main paper and references but not the cited appendix, preventing verification of piecewise-function constraints, extended experiments, and finer implementation protocols; the main method and Tables 1–7 remain readable, but reproducibility is incomplete.
• The medical Table 4 caption says an average over 10 datasets, while the accompanying paragraph says 11 and the earlier description separately lists HAM10000 and 10 MedMNIST2D sources. This inconsistency is retained rather than resolving the aggregation count without evidence.

Related Work & Insights

• vs TS: TS learns only a global temperature; QaTS adds a confidence-distribution reference and a quantile slope while retaining the decision-preserving property of positive temperature scaling.
• vs AdaTS and GC: AdaTS uses test-time augmentation, while GC uses semantic grouping information; QaTS constructs its conditioning variable directly from confidence without requiring additional views or semantic grouping modules.
• vs FeatClip: Feature clipping can change predictions, whereas QaTS explicitly separates probability calibration from class decisions, emphasizing why evaluation should not optimize ECE in isolation.
• Research direction: compare fixed-CDF and updated-reference variants under controlled monotone score transformations, real domain shifts, and varying calibration-set sizes; this is a reader proposal, not an experiment established by the paper.

Rating

• Novelty: 4/5. Combining empirical quantiles with a two-parameter temperature is clear and economical, though it remains a structured extension of post-hoc temperature scaling.
• Experimental Thoroughness: 4/5. Broad task, long-tail, and corruption coverage and useful parameterization ablations; significance and some implementation details remain unclear.
• Writing Quality: 3/5. The central mechanism is easy to explain, but rank-stability language, medical dataset counts, and some table annotations require caution.
• Value: 4/5. A useful low-capacity calibration baseline for frozen classifiers, particularly when class predictions must remain unchanged.