Self-Calibrating Language Models via Test-Time Discriminative Distillation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/rissalhedna/SECL
Area: LLM Evaluation
Keywords: Model Calibration, Test-Time Training, Generation-Discrimination Gap, Self-Supervised Distillation, Expected Calibration Error
TL;DR¶
This paper introduces SECL, the first test-time training pipeline for language model confidence calibration that leverages the model's inherently better-calibrated discriminative probabilities as a self-supervision signal, triggering bounded directional LoRA updates via adaptive entropy gating to reduce Expected Calibration Error by 56%โ78% without ground-truth labels while preserving task accuracy.
Background & Motivation¶
Large language models (LLMs) are systematically overconfident, routinely reporting high certainty on queries they answer incorrectly. Alignment procedures such as Reinforcement Learning from Human Feedback (RLHF) and preference tuning exacerbate this phenomenon by prioritizing assertive delivery aligned with human preferences over actual truthfulness. In high-stakes domains like medical triage, legal analysis, and clinical diagnostics, uncalibrated overconfidence poses severe safety risks. Existing calibration solutions face critical operational bottlenecks: sampling-based approaches like self-consistency or semantic entropy incur massive computational overhead through multi-pass decoding and fail on consistent hallucinations; static white-box probing methods require offline supervised annotations and degrade sharply under distribution shifts; and reward-driven fine-tuning methods either rely on ground-truth correctness labels or suffer severe out-of-distribution (OOD) degradation under standard reinforcement learning.
Theoretical and empirical investigations reveal a structural asymmetry: LLMs harbor an internal discriminative signal that is significantly better calibrated than their verbalized confidence. When prompted to evaluate whether an answer is correct (\(P(\text{True})\)), models produce probability judgments with far lower calibration error than their generation-time confidence. Theoretically, Kalai et al. established that generative error is lower-bounded by roughly twice the corresponding discriminative binary classification error, confirming that models can frequently recognize mistakes they cannot avoid producing. Although prior studies documented this gap diagnostically or exploited it via static inference-time heuristics, distilling this internal capability into generative confidence dynamically during test-time distribution shifts remained unexplored.
Translating this insight into test-time adaptation presents two foundational dilemmas: continuous adaptation across an incoming question stream incurs prohibitive compute if evaluated on every instance, and unconstrained online gradient descent on noisy pseudo-labels risks catastrophic forgetting or confidence collapse. Core idea: exploit the intrinsic generation-discrimination gap as label-free self-supervision, triggering lightweight LoRA bursts exclusively upon entropy-detected distribution shifts and distilling distractor-normalized discriminative confidence into verbalized confidence via bounded directional updates.
Method¶
Overall Architecture¶
SECL (SElf-Calibrating Language Models) operates through three interconnected stages: test-time streaming change detection via entropy tracking, label-free self-supervision synthesis via distractor normalization, and conservative test-time adaptation via low-rank parameter updates.
During streaming inference, an adaptive entropy gate monitors the exponential moving average of output token entropy using the Page-Hinkley cumulative sum test. If the input distribution remains stationary, the frozen model and its accumulated LoRA weights process queries directly. Once a distribution shift is detected, the system triggers a calibration burst over a window of \(B=50\) consecutive questions. Within this burst, the unadapted base model generates candidate answers and distractors to compute normalized discriminative confidence (\(\text{NormPTrue}\)). If verbalized confidence and the discriminative signal disagree by more than one confidence bin, lightweight LoRA parameters on intermediate-to-late Transformer layers are updated using a clipped directional mean squared error objective. Across the question stream, adaptation parameters compound continuously without resetting.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Incoming Question Stream"] --> B["Adaptive Entropy Gating<br/>EMA entropy tracking & Page-Hinkley test"]
B -->|No distribution shift| C["Direct Inference<br/>Frozen LoRA outputs calibrated confidence"]
B -->|Shift detected: Burst B=50| D["NormPTrue Self-Supervision<br/>Frozen base model + distractor softmax"]
D --> E["Bin-Gate Filter<br/>Select samples with disagreement > 1 bin"]
E --> F["Conservative Directional TTT<br/>Bounded step clipping & weight accumulation"]
F --> C
Key Designs¶
1. Adaptive Entropy Gating: Page-Hinkley Change Detection for Burst Adaptation
To prevent redundant computation on stable distributions and shield model representations from unnecessary gradient perturbations, SECL introduces adaptive entropy gating. The framework tracks the output token distribution entropy \(H_t\) with an exponential moving average (\(\alpha_{\text{ema}}=0.05\)) and maintains the Page-Hinkley cumulative sum statistic: $\(m_t = \sum_{s=1}^t (H_s - \bar{H}_t - \epsilon)\)$ where \(\bar{H}_t\) represents the running mean entropy and \(\epsilon=0.05\) suppresses minor stochastic fluctuations. An adaptation alert fires when \(m_t - \min_{s\le t} m_s > \lambda\) (with detection threshold \(\lambda=3.0\)). Upon firing, the cumulative statistics reset and initiate a calibration burst of \(B=50\) consecutive queries. Restricting adaptation to post-shift transition windows reduces the proportion of processed training questions to just 6%โ26% of the stream, rendering SECL cheaper than the discriminative baseline it distills from.
2. NormPTrue Self-Supervision: Mitigating Suggestibility Bias via Distractor Normalization
Raw token probabilities for \(P(\text{True}\mid q, a)\) suffer from suggestibility bias, as autoregressive LLMs systematically exhibit an affirmation bias toward whatever candidate text is presented. SECL neutralizes this systematic bias by normalizing \(P(\text{True})\) across candidate distractors. For multiple-choice questions, the given alternative options serve as distractors; for open-ended queries, the model generates \(K=4\) plausible alternative answers \(\{d_1, \dots, d_K\}\). A temperature-scaled softmax produces the continuous supervision target: $\(\text{NormPTrue}(a) = \frac{\exp(P_{\text{True}}(a)/\tau)}{\exp(P_{\text{True}}(a)/\tau) + \sum_{k=1}^K \exp(P_{\text{True}}(d_k)/\tau)}\)$ where \(\tau\) is a model-specific normalization temperature. Computing this score on the frozen base model (without active LoRA adapters) guarantees that running parameter updates do not contaminate the supervision source.
3. Conservative Directional TTT: Bounded Step Clipping with Persistent Weight Accumulation
While distractor normalization removes population-level affirmation bias, per-instance discriminative estimates retain high variance. Directly regressing verbalized confidence onto \(\text{NormPTrue}\) leads to overshooting and instability. SECL resolves this via a conservative directional update rule. Given verbalized soft confidence \(c_i\) (computed as the expected value over 10 digit token bins \(0\sim9\)) and discriminative target \(c_i^*\), the training target is constructed as: $\(\hat{c}_i = c_i + \alpha_{\text{step}} \cdot \text{clip}(c_i^* - c_i, -\delta, \delta)\)$ with step size \(\alpha_{\text{step}}=0.5\) and clipping bound \(\delta=0.15\). The mechanism trusts the direction of the discriminative vector while bounding individual step magnitudes to \([-\alpha_{\text{step}}\delta, +\alpha_{\text{step}}\delta]\), allowing zero-mean instance noise to cancel out across the burst. A bin-gate filter bypasses gradient calculation whenever \(|\text{bin}(c_i) - \text{bin}(c_i^*)| \le 1\). Crucially, LoRA weights accumulate across domains without resetting, enabling calibration capability to compound over time.
Loss & Training¶
The optimization objective minimizes the mean squared error between verbalized confidence and the clipped directional target: $\(\mathcal{L}_i = (c_i - \hat{c}_i)^2\)$ Updates are applied via LoRA (rank \(r=8\), scaling factor \(\alpha=16\)) targeting the query and value projection matrices (\(q\text{\_proj}, v\text{\_proj}\) or fused \(qkv\text{\_proj}\)) within the last 4 to 8 Transformer layers, modifying only 0.01%โ0.02% of total parameters. Optimization employs AdamW with learning rate \(5\times 10^{-5}\) for 3 epochs per selected question, operating entirely without ground-truth correctness labels.
Key Experimental Results¶
Main Results¶
Evaluations span a sequential stream of 2,000 questions across four distinct domains: GSM8K (mathematical reasoning), MMLU (multitask knowledge), ARC-Challenge (scientific reasoning), and TruthfulQA (adversarial misconceptions). SECL is evaluated against the uncalibrated verbalized baseline, the multi-pass discriminative benchmark P(True) Norm, and DINCO.
| Model | Method | Amortized Cost (FWD-eq)โ | ECEโ | Brier Scoreโ | AUROCโ | Task Accuracy (%) |
|---|---|---|---|---|---|---|
| Llama 3.2-3B | Verbalized | 1.0 | 0.170 | 0.292 | 0.510 | 57.6 |
| P(True) Norm | 6.0 | 0.065 | 0.223 | 0.694 | 57.6 | |
| DINCO | ~10.0 | 0.101 | 0.207 | 0.762 | 46.6 | |
| SECL (Ours) | 1.8โ4.6 (4.6) | 0.050 | 0.241 | 0.587 | 57.7 | |
| Llama 3.1-8B | Verbalized | 1.0 | 0.225 | 0.258 | 0.684 | 64.4 |
| P(True) Norm | 6.0 | 0.120 | 0.211 | 0.718 | 64.6 | |
| DINCO | ~10.0 | 0.117 | 0.210 | 0.756 | 52.2 | |
| SECL (Ours) | 2.8 | 0.083 | 0.222 | 0.643 | 64.6 | |
| Gemma 2-2B | Verbalized | 1.0 | 0.256 | 0.314 | 0.558 | 51.6 |
| P(True) Norm | 6.0 | 0.141 | 0.259 | 0.650 | 51.6 | |
| DINCO | ~10.0 | 0.408 | 0.410 | 0.566 | 32.7 | |
| SECL (Ours) | 1.8 | 0.056 | 0.254 | 0.548 | 51.5 | |
| Phi 3.5-Mini | Verbalized | 1.0 | 0.251 | 0.275 | 0.600 | 66.7 |
| P(True) Norm | 6.0 | 0.154 | 0.227 | 0.675 | 66.7 | |
| DINCO | ~10.0 | 0.110 | 0.212 | 0.749 | 56.0 | |
| SECL (Ours) | 2.1 | 0.110 | 0.251 | 0.521 | 66.5 |
Ablation Study¶
Ablation experiments conducted on Llama 3.2-3B isolate the impact of gating, objective formulation, weight persistence, and supervision targets across the full 2,000-question stream:
| Configuration / Variant | ECEโ | Brier Scoreโ | AUROCโ | Questions Trained (%) | Note |
|---|---|---|---|---|---|
| Full SECL (Entropy-Gated B=50) | 0.050 | 0.241 | 0.587 | 25.6% | Canonical setting balancing cost and calibration |
| Always-On Adaptation | 0.047 | 0.240 | 0.593 | 100.0% | 4ร compute with marginal ECE difference |
| Bin-Gate Only (\(\le 1\) bin) | 0.052 | 0.242 | 0.585 | 55.8% | Without entropy detection, updates double |
| Plain MSE Loss (No Directional Clip) | 0.085 | - | - | 100.0% | High instance variance corrupts adaptation |
| Reset LoRA Weights Per Question | 0.237 | 0.327 | 0.484 | 100.0% | Isolated updates degrade calibration below chance |
| Supervised via Self-Consistency (\(N=10\)) | 0.432 | 0.470 | 0.443 | 25.6% | Low-entropy modal bias induces severe overconfidence |
Key Findings¶
- Distillation Surpasses the Teacher Signal: On all four primary models, SECL's adapted verbalized ECE outperforms the supervision target P(True) Norm itself (e.g., Llama 3.2-3B achieves ECE 0.050 vs. 0.065 for P(True) Norm), demonstrating that bounded gradient updates effectively filter out per-instance noise while capturing the systematic calibration signal.
- Persistent Weight Accumulation is Mandatory: Resetting LoRA weights between questions collapses ECE to 0.237 and drops AUROC to 0.484 (below chance), proving that calibration adaptation fundamentally requires cumulative parameter consolidation.
- Supervision Target Quality Sets the Performance Ceiling: Replacing NormPTrue with Self-Consistency (\(N=10\)) degrades ECE from 0.050 to 0.432 (2.5ร worse than unadapted baseline), confirming that distilling an overconfident modal distribution severely miscalibrates the model.
- Preservation of Core Task Accuracy: Task performance shifts by less than 1 percentage point across all models (e.g., Llama 3.2-3B accuracy is 57.7% vs. 57.6%), confirming that adapting a fraction of late-layer projection parameters adjusts confidence expressions without corrupting underlying reasoning.
Highlights & Insights¶
- Pioneering TTT for Uncertainty Calibration: Extends test-time training from task-accuracy enhancement to confidence calibration, offering an unsupervised alternative to static post-hoc recalibration.
- Exploiting the Theoretical Asymmetry of Language Models: Leverages the mathematical principle that generative error is lower-bounded by twice the discriminative error, converting an inherent internal gap into an actionable self-supervision signal.
- Directional Loss for Robust Pseudo-Labeling: Introduces a bounded update formulation that trusts the expected gradient direction across populations while bounding single-step displacement, effectively insulating test-time training from label noise.
Limitations & Future Work¶
- Dependence on a Pre-existing Generation-Discrimination Gap: SECL requires the discriminative signal to outperform generative confidence. On models where this gap is absent (such as Qwen 2.5-3B, where NormPTrue is inferior to verbalized confidence across all temperatures), SECL yields no calibration gain.
- Calibration-Discrimination Trade-off: Because LoRA updates optimize probability mass across discrete confidence bins to minimize ECE, fine-grained instance ranking quality (AUROC) can degrade slightly on certain architectures (e.g., Phi 3.5-Mini drops from 0.600 to 0.521).
- Vulnerability to Stream-Level Adversarial Attacks: The Page-Hinkley entropy monitor and unsupervised distractor generation assume benign distribution shifts; adversarial inputs could theoretically suppress burst adaptation or inject poisoned pseudo-labels into the accumulated weights.
Related Work & Insights¶
- vs DINCO (Wang & Stengel-Eskin, 2025): DINCO relies on static inference-time NLI reweighting and multi-sample self-consistency (~10 forward passes per query) and fails on Gemma (ECE 0.408); SECL bakes calibration directly into model weights via lightweight test-time training, achieving lower ECE at 2โ5ร lower compute without external verifiers.
- vs Post-Hoc Temperature / Platt Scaling: Supervised post-hoc scaling fits parameters on labeled validation data and achieves low ECE primarily by collapsing confidence into narrow ranges (e.g., [0.44, 0.56] on Llama); SECL operates strictly without ground-truth labels and preserves a wide confidence span ([0.05, 0.85]).
Rating¶
- Novelty: โญโญโญโญโญ [First framework applying test-time training to confidence calibration via the generation-discrimination gap]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 5 models from 2B to 12B, 4 diverse domains, and 7 rigorous ablations]
- Writing Quality: โญโญโญโญโญ [Clear theoretical motivation, mathematically precise formulation, and honest trade-off characterization]
- Value: โญโญโญโญโญ [Provides a practical, label-free blueprint for deploying well-calibrated LLMs in distribution-shifting production environments]