Skip to content

PRICE: Preemptive Confidence Estimation in LLMs via Internal Representations

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Preemptive Confidence Estimation, Internal Representations, Multi-head Attention Consensus, Hidden State Salience, Model Calibration

TL;DR

PRICE is a training-free, pre-generation confidence estimation framework for LLMs that extracts attention-head consensus and hidden-state semantic salience in a single forward pass, delivering reliable uncertainty quantification before decoding begins without training external probes or sampling multiple outputs.

Background & Motivation

Large language models (LLMs) have achieved remarkable success across diverse reasoning, comprehension, and decision-making benchmarks, leading to widespread deployment in critical domains such as clinical diagnosis and legal analysis. However, state-of-the-art LLMs suffer from severe overconfidence, routinely generating factually inaccurate claims or hallucinatory reasoning chains with high verbalized certainty. In safety-critical contexts, reliable deployment requires assessing model confidence prior to generation. Preemptive confidence estimation enables intelligent computation routing, such as triggering answer abstention, alerting end-users, or dynamically selecting cost-intensive strategies like Chain-of-Thought (CoT) prompting and Retrieval-Augmented Generation (RAG).

Existing confidence and uncertainty quantification frameworks are fundamentally constrained by either latency or generalization bottlenecks. Post-hoc methods inspect generated tokens, computing output entropy, perplexity, or consistency across multiple stochastically sampled reasoning paths (e.g., SelfCheckGPT, EigenScore). While semantic consistency can be effective, evaluation is only possible after full answer generation, and repeated autoregressive sampling incurs unacceptable computational overhead that prohibits real-time intervention. Conversely, internal probing techniques train supervised linear probes or multi-layer perceptrons on intermediate activation layers. Although accurate on in-distribution data, supervised probes demand extensive labeled datasets, incur separate training costs, and frequently suffer performance degradation under out-of-distribution domain shifts.

This creates a fundamental tension: practical systems need confidence scores before text generation without the latency of sampling, yet supervised classifiers built on internal features generalize poorly across domains. Empirical examination reveals that when an LLM is genuinely confident in its input understanding, multi-head attention heads display strong pattern consensus, and focus-span hidden states exhibit tight directional alignment alongside pronounced activation sparsity. The core idea is to couple attention-head consensus measured by normalized Jensen-Shannon Divergence with hidden-state semantic salience derived from directional coherence and Gini sparsity, forming an unsupervised, single-forward-pass preemptive confidence estimator whose optimal weighting minimizes calibration error in closed form.

Method

Overall Architecture

PRICE (PReemptive Internal Confidence Estimator) operates during the single forward pass of the input query through the LLM, extracting internal representations from the final Transformer layer. The pipeline extracts two orthogonal and complementary signals: a multi-head attention consensus signal (\(S_{\text{con}}\)) that measures structural agreement across attention heads, and a hidden-state semantic salience signal (\(S_{\text{sal}}\)) that evaluates the directional alignment and activation sparsity of hidden states across a trailing focus token span. These signals are fused via a weighted sum \(S_{\text{PRICE}} = \alpha \cdot S_{\text{con}} + (1 - \alpha) \cdot S_{\text{sal}}\), where \(\alpha\) can be fixed to a neutral 0.5 in completely unsupervised settings or tuned in closed form (\(\alpha^*\)) to minimize the empirical Brier score and tighten the theoretical upper bound on smooth expected calibration error (smECE).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query Prompt (T Tokens)"] --> B["Single Forward Pass (Extract Final Layer Tensors)"]
    B --> C["Multi-Head Attention Consensus Scon<br/>JSD / Mutual Information Across Heads"]
    B --> D["Hidden-State Semantic Salience Ssal<br/>Directional Coherence ร— Gini Sparsity"]
    C --> E["Closed-Form Optimal Fusion SPRICE<br/>Minimizes Brier Score & Upper Bounds smECE"]
    D --> E
    E --> F["Preemptive Confidence Decision<br/>Selective Routing / RAG / Abstention / Early Warning"]

Key Designs

1. Multi-Head Attention Consensus: Resolving Head Divergence via Mutual Information

To quantify how consistently attention heads focus on relevant input evidence, PRICE evaluates head agreement across all sequence positions. Given attention tensor components \(a_{h,t} \in \mathbb{R}^T\) for head \(h \in \{1,\dots,H\}\) at position \(t\), the mean attention distribution across heads is \(m_t = \frac{1}{H}\sum_{h=1}^H a_{h,t}\). The Jensen-Shannon Divergence at position \(t\) is computed as \(JSD_t = H(m_t) - \frac{1}{H}\sum_{h=1}^H H(a_{h,t})\), where \(H(\cdot)\) denotes Shannon entropy. PRICE formally proves (Proposition 2) that \(JSD_t\) is strictly equivalent to the mutual information \(I(Z; K)\) between a uniformly distributed head index \(Z\) and the selected key token index \(K\). Averaging across the sequence of length \(T\) and normalizing by its theoretical maximum \(\log H\) yields the consensus score:

\[S_{\text{con}} = 1 - \frac{\frac{1}{T}\sum_{t=1}^T JSD_t}{\log H}\]

When heads attend to identical evidence tokens, knowing the attended key reveals zero information about which head produced it, driving mutual information to zero and \(S_{\text{con}}\) to 1. When heads disperse erratically across disparate tokens, \(S_{\text{con}}\) drops sharply, capturing internal reasoning instability before any output token is emitted.

2. Hidden-State Semantic Salience: Countering Degenerate Consensus with Dual Geometric Constraints

Because attention consensus can produce high scores under uninformative, near-uniform attention distributions where all heads trivially match, PRICE introduces hidden-state semantic salience as a vital counterbalance. Leveraging causal masking dynamics where contextual information aggregates toward the prompt boundary, PRICE selects the last 30% of the sequence as the focus token span \(I_f\) (avoiding noise from the single terminal token). Salience is defined as the product of directional coherence and activation sparsity:

\[S_{\text{sal}} = \text{Coherence} \times \text{Sparsity}\]

Directional coherence measures the \(L_2\) norm of the mean hidden state \(\bar{h} = \frac{1}{k}\sum_{t \in I_f} h_t\), normalized via temperature scaling \(\tau = 30\):

\[\text{Coherence} = \phi\left(\frac{\|\bar{h}\|_2}{\tau}\right) = \tanh\left(\frac{\|\sum_{t \in I_f} h_t\|_2}{k \cdot \tau}\right)\]

Activation sparsity evaluates feature concentration by normalizing vector components into an absolute-value probability distribution \(p_t = \frac{|h_t|}{\|h_t\|_1}\) and computing its deviation from the uniform distribution via a normalized Gini coefficient:

\[G(p_t) = \frac{1}{2(1 - 1/d)} \sum_{j=1}^d \left| p_{t,j} - \frac{1}{d} \right|\]

Averaging \(G(p_t)\) over \(I_f\) yields Sparsity. A high \(S_{\text{sal}}\) requires both consistent semantic trajectory and sharp, sparse activation over critical feature dimensions, effectively penalizing diffuse or non-informative activations.

3. Closed-Form Optimal Weighting: Minimizing Calibration Bounds Analytically

To integrate \(S_{\text{con}}\) and \(S_{\text{sal}}\) without expensive iterative grid searches, PRICE formulates confidence as \(S_{\text{PRICE}} = \alpha \cdot S_{\text{con}} + (1 - \alpha) \cdot S_{\text{sal}}\). Under the model where both estimators are unbiased relative to true confidence \(p\) with error variances \(\sigma_1^2, \sigma_2^2\) and covariance \(C\), the population-optimal coefficient minimizing the expected Brier score \(\mathbb{E}[(S_\alpha - Y)^2]\) is \(\alpha^* = \frac{\sigma_2^2 - C}{\sigma_1^2 + \sigma_2^2 - 2C}\). For a finite verification dataset of \(n\) samples with binary correctness labels \(y_i \in \{0, 1\}\), PRICE derives the finite-sample closed-form solution:

\[\alpha^* = \frac{\sum_{i=1}^n (S_{\text{con},i} - S_{\text{sal},i})(y_i - S_{\text{sal},i})}{\sum_{i=1}^n (S_{\text{con},i} - S_{\text{sal},i})^2}\]

Crucially, the authors prove via Tonelli's theorem and Cauchy-Schwarz that smooth expected calibration error is upper-bounded by the root expected Brier score: \(\text{smECE}_\sigma(S_\alpha) \le C_K \sqrt{\mathbb{E}[(S_\alpha - Y)^2]}\). Thus, minimizing the Brier score in closed form simultaneously minimizes the theoretical upper bound on smECE. Furthermore, empirical evaluations demonstrate that calculating \(\alpha^*\) on as little as 20% of a validation split yields optimal convergence across datasets.

Loss & Training

PRICE is completely training-free and executes deterministically during standard inference forward passes. The underlying language models (Mistral-7B-v0.3-Instruct, Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct) are evaluated off-the-shelf without fine-tuning or parameter updates. Ground truth correctness labels \(y_i \in \{0, 1\}\) are obtained via 5-shot greedy decoding across benchmarks.

Two deployment modes are evaluated: 1. Zero-shot unsupervised setting: The weighting coefficient is fixed to \(\alpha = 0.5\) without accessing any labeled verification data, establishing a fair comparison against training-free pre-generation baselines (Prompt, Perplexity, Hidden Score, Attention Score). 2. Optimal calibration setting: The closed-form analytical equation for \(\alpha^*\) is computed on an 80% (or 20%) training split and evaluated on the held-out test split, benchmarking directly against supervised probing classifiers (Probe1 MLP and Probe2 Linear probe).

Key Experimental Results

Main Results

Evaluations were conducted across five diverse benchmark datasets spanning general knowledge, symbolic reasoning, and mathematics: MMLU, BBH, GPQA, ARC-C, and GSM8K. Performance was quantified via smooth expected calibration error (\(\text{smECE}\), lower is better) using reflected Gaussian kernel smoothing, and Brier score (lower is better).

The following table presents the comparative results from Table 1 with fixed \(\alpha = 0.5\) against training-free pre-generation baselines:

Model Method MMLU smECE / Brier BBH smECE / Brier GPQA smECE / Brier ARC-C smECE / Brier GSM8K smECE / Brier
Mistral-7B Prompt (Self-reported) 0.3689 / 0.3255 0.3088 / 0.3357 0.4111 / 0.3988 0.2989 / 0.3041 0.3201 / 0.3556
Perplexity 0.2655 / 0.3303 0.5579 / 0.5520 0.5303 / 0.5052 0.0846 / 0.1796 0.4115 / 0.5163
Hidden Score 0.1001 / 0.2478 0.1028 / 0.2389 0.3460 / 0.3547 0.0988 / 0.1829 0.1315 / 0.2972
Attention Score 0.1017 / 0.2608 0.1829 / 0.2720 0.4018 / 0.3959 0.1767 / 0.2097 0.0764 / 0.2511
PRICE (\(\alpha = 0.5\)) 0.0592 / 0.2295 0.0402 / 0.1989 0.1566 / 0.2555 0.1399 / 0.1901 0.0633 / 0.2348
Qwen2.5-7B Prompt (Self-reported) 0.1958 / 0.2296 0.1913 / 0.2196 0.3628 / 0.3508 0.2115 / 0.2443 0.4189 / 0.4336
Perplexity 0.1784 / 0.2434 0.4717 / 0.4722 0.5315 / 0.5111 0.1259 / 0.1114 0.4255 / 0.5092
Hidden Score 0.0864 / 0.2165 0.0875 / 0.2523 0.3122 / 0.3391 0.2226 / 0.1510 0.1515 / 0.2709
Attention Score 0.0983 / 0.2126 0.0542 / 0.2452 0.2674 / 0.3114 0.2293 / 0.1571 0.1043 / 0.2161
PRICE (\(\alpha = 0.5\)) 0.0531 / 0.1653 0.0588 / 0.2188 0.1292 / 0.2696 0.1203 / 0.1659 0.1038 / 0.2296
Llama-3.1-8B Prompt (Self-reported) 0.1798 / 0.1969 0.2548 / 0.3099 0.4518 / 0.4885 0.1295 / 0.1573 0.4259 / 0.3933
Perplexity 0.2272 / 0.2914 0.5211 / 0.5199 0.5620 / 0.5310 0.1329 / 0.1665 0.2902 / 0.3119
Hidden Score 0.0610 / 0.2256 0.1658 / 0.2642 0.3719 / 0.3746 0.0887 / 0.1574 0.1077 / 0.2987
Attention Score 0.0792 / 0.2405 0.1743 / 0.2697 0.2819 / 0.3248 0.0731 / 0.1558 0.0823 / 0.2621
PRICE (\(\alpha = 0.5\)) 0.0603 / 0.1953 0.0591 / 0.2139 0.1596 / 0.2563 0.0709 / 0.1399 0.0783 / 0.2452

Ablation & Architectural Analysis

The paper examined representation depth, head weighting schemes, and focus span lengths to validate design choices. The table below excerpts layer-wise calibration performance from Table 5 across BBH and GSM8K:

Model Layer Extracted (Layer Index) BBH smECE / Brier GSM8K smECE / Brier Note
Mistral-7B (32 Layers) 0.5 (Layer 16) 0.0611 / 0.2365 0.0449 / 0.2428 Incomplete confidence signal formation
0.7 (Layer 23) 0.0395 / 0.2281 0.0388 / 0.2201 Consistent error decline as depth increases
Last (Layer 32) 0.0234 / 0.2238 0.0261 / 0.2020 Deepest layer achieves superior calibration
Qwen2.5-7B (28 Layers) 0.5 (Layer 14) 0.0430 / 0.2460 0.1331 / 0.2253 High calibration error in intermediate layers
0.8 (Layer 23) 0.0719 / 0.2365 0.0879 / 0.1905 Features converge toward semantic clarity
Last (Layer 28) 0.0316 / 0.2326 0.0755 / 0.1888 Lowest smECE and Brier score
Llama-3.1-8B (32 Layers) 0.5 (Layer 16) 0.0616 / 0.2354 0.1259 / 0.2959 Weaker predictive representation
0.8 (Layer 26) 0.0532 / 0.2352 0.0695 / 0.2631 Marked calibration improvement
Last (Layer 32) 0.0339 / 0.2341 0.0538 / 0.2510 Final layer demonstrates peak calibration

Additionally, analyzing attention head weighting (Table 10) confirmed that uniform head averaging outperforms dynamic output-norm weighting. On MMLU, unweighted vs. weighted smECE was 0.0371 vs. 0.0788 for Mistral-7B, 0.0531 vs. 0.0937 for Qwen2.5-7B, and 0.0257 vs. 0.0699 for Llama-3.1-8B. Low-norm heads encode subtle evidence fluctuations, and weighting suppresses informative residual signals.

Key Findings

  • Orthogonal Signal Complementarity: The Pearson correlation between \(S_{\text{con}}\) and \(S_{\text{sal}}\) is only 0.13, indicating near-independence. In 30ร—30 bin heatmap analysis, samples with high consensus but low salience (Quadrant IV) exhibit high error rates due to diffuse pseudo-consensus, whereas samples with low consensus but high salience (Quadrant II) retain moderate accuracy due to localized feature activation. The joint score successfully rectifies both failure modes.
  • Dramatically Lower Inference Latency: Computing PRICE requires only 0.9ร— to 1.1ร— the runtime of generating a single output token. In contrast, sampling-based post-hoc estimators like EigenScore require approximately 9.5ร— more compute time while achieving comparable or higher smECE.
  • Parity with Supervised Probes: Armed with the closed-form weight \(\alpha^*\), PRICE matches or exceeds the calibration accuracy of supervised MLP probes (Probe1) and linear probes (Probe2) across most tasks without fitting any neural parameters, requiring only 20% data to estimate \(\alpha^*\).

Highlights & Insights

  • Information-Theoretic Foundation for Attention Entropy: The authors prove that Jensen-Shannon Divergence across multi-head attention rigorously quantifies the mutual information \(I(Z; K)\) between head identity and attended keys. This grounds attention pattern stability in classical information theory rather than heuristic distance measures.
  • Gini-Normalized Directional Salience: Combining vector sum magnitude with the Gini inequality coefficient establishes an unsupervised metric for semantic sharpness, ensuring that high confidence is assigned only when hidden representations exhibit coherent orientation and concentrated feature sparsity.
  • Brier-Minimizing Closed-Form Solution: Proving that smECE is mathematically upper-bounded by the square root of the expected Brier score justifies closed-form optimization, eliminating brute-force hyperparameter sweeps while guaranteeing provably bounded calibration error.

Limitations & Future Work

  • Requirement for White-Box Activation Access: PRICE requires direct access to model weights, attention matrices, and hidden states at the final Transformer layer, making it inapplicable to commercial black-box APIs (e.g., GPT-4o, Claude 3.5 Sonnet) that expose only generation logits or text outputs.
  • Inability to Diagnose Uncertainty Attribution: While PRICE effectively flags unstable internal representations, it cannot determine whether low confidence stems from epistemic knowledge gaps in training data, linguistic ambiguity in user prompts, or adversarial manipulation.
  • Mild Sensitivity in Cross-Domain Coefficient Transfer: Although \(\alpha^*\) transfers well between similar tasks (e.g., MMLU to BBH), cross-transferring between radically different reasoning types (e.g., GSM8K mathematical reasoning to BBH symbolic logic) leads to noticeable calibration degradation, requiring a small domain calibration split for peak performance.
  • vs Prompt-Based Self-Reported Confidence & P(True): Verbalized confidence is severely distorted by RLHF-induced sycophancy and overconfidence; PRICE inspects pre-linguistic internal activations, slashing calibration error by over 50%.
  • vs Post-Generation Sampling (EigenScore, SelfCheckGPT): Sampling-based methods demand multi-path autoregressive decoding, imposing high computational costs and preventing pre-generation compute allocation; PRICE operates on a single forward pass before decoding begins.
  • vs Supervised Representation Probes (Azaria & Mitchell 2023, Ji et al. 2024b): Standard probes require collecting labeled datasets and training dedicated classifiers that degrade under out-of-domain distribution shifts; PRICE provides an unsupervised, closed-form estimation framework with superior cross-domain robustness.

Rating

  • Novelty: โญโญโญโญโ˜† [Unifies multi-head mutual information and hidden-state Gini sparsity for single-pass preemptive confidence estimation]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 3 model families, 5 benchmark datasets, layer ablations, head weighting, and transfer studies]
  • Writing Quality: โญโญโญโญโญ [Rigorous mathematical proofs for Propositions 1โ€“4 and Lemmas 1โ€“2, paired with clear empirical visualizations]
  • Value: โญโญโญโญโญ [Provides a practical, lightweight foundation for adaptive test-time compute routing and safety-critical abstention]