Skip to content

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/shiyuni2000/PopCalib
Area: Model Compression
Keywords: Knowledge Popularity, Probabilistic Confidence Calibration, LLM Overconfidence, Hallucination Mitigation, Entity Co-occurrence

TL;DR

This paper investigates the root cause of LLM overconfidence in factual question answering through the lens of knowledge popularity, revealing that hallucinated answers systematically drift toward globally popular or frequently co-occurring entities, and introduces a lightweight popularity-aware MLP calibrator that slashes average confidence on incorrect answers from 0.765 to 0.254 while reducing Expected Calibration Error (ECE) from 0.356 to 0.050.

Background & Motivation

Building trustworthy and deployable large language model (LLM) systems requires calibrated confidence estimation—where the generation probability assigned to an answer faithfully reflects the empirical likelihood that the answer is factually correct. However, modern LLMs persistently suffer from severe overconfidence, routinely assigning near-certain generation probabilities to factually incorrect assertions. Such miscalibrated overconfidence poses grave risks in safety-critical applications such as clinical decision support, legal analysis, and automated financial intelligence. Conventional calibration methods—including post-hoc temperature scaling, consistency sampling across multiple decoded responses, and verbalized self-assessment via prompting—typically treat overconfidence as an unexplainable heuristic distortion of output logits, leaving the mechanistic origin of why models assign high confidence to incorrect outputs largely unresolved.

Fundamentally, overconfidence reflects an acute mismatch between the objective correctness of a generated prediction and the subjective probabilistic certainty assigned by the model. When examining pretraining dynamics, a natural question arises: what computational mechanisms drive these inflated probability assignments? Prior work has firmly established that corpus exposure strongly modulates factual recall—entities and facts occurring more frequently in pretraining data are substantially more likely to be retrieved correctly. However, these earlier investigations focused almost entirely on the ground-truth knowledge side, asking when and why models are able to answer correctly. In sharp contrast, overconfidence specifically concerns the probability mass allocated to the generated answer, especially when that generated output represents a factual failure.

This critical blind spot motivates the paper to shift attention from ground-truth-side popularity to generation-side knowledge popularity. Focusing on entity-centric factual QA formulated from Wikidata triplets, the authors systematically characterize knowledge popularity across three complementary dimensions: question entity popularity, generated answer popularity, and question-answer co-occurrence frequency within raw pretraining corpora. The empirical findings reveal a profound and systematic structural bias: LLM hallucinations are far from uniform random noise; rather, incorrect predictions consistently gravitate toward entities that are either globally more prominent or more densely co-occurring with the question entity than the ground-truth answer. Even among wrong predictions, confidence scales directly with knowledge popularity, and scaling model parameters fails to alleviate this pathological correlation. Core idea: LLM overconfidence is mechanistically driven by pretraining knowledge popularity priors hijacking the generation distribution; integrating generation-side popularity and co-occurrence features into a lightweight calibrator strips away ungrounded familiarity-induced confidence, effectively mitigating overconfidence and achieving faithful knowledge boundary calibration.

Method

Overall Architecture

The framework establishes an analytical and calibrating pipeline that operates in two sequential stages: a multi-factor popularity characterization stage that measures the familiarity and relational association of the query and output tokens, followed by a lightweight popularity-aware calibration network that refines the raw probabilistic confidence into a calibrated correctness probability.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Factual Question q<br/>Identify subject entity and target relation"] --> B["Three-Factor Knowledge Popularity Profiling<br/>Compute PopQ, PopGe, and RPopGe counts"]
    B --> C["Partial Correlation Analysis & Error Attribution<br/>OLS residualization isolates pure popularity effects"]
    C --> D["Lightweight Popularity-Aware Calibrator<br/>Three-layer MLP optimized via binary cross-entropy"]
    D --> E["Self-Generated Popularity Closed-Loop<br/>Model self-estimates popularity without corpora lookups"]

During inference, given a question \(q\), the model generates an answer sequence along with its greedy mean token probability \(c = \frac{1}{T}\sum_{i=1}^T P(g_i \mid g_{<i})\). The system simultaneously extracts generation-side popularity metrics \(x = [c, \text{PopQ}, \text{PopGe}, \text{RPopGe}]^\top\), which are fed into a three-layer multi-layer perceptron (MLP). The network outputs a well-calibrated confidence score \(\hat{y} = \sigma(\text{MLP}(x)) \in [0, 1]\) that aligns with the factual correctness of the answer.

Key Designs

1. Three-Factor Knowledge Popularity Profiling: Decoupling Entity Prominence from Relational Density

Addressing the limitation of previous studies that relied on a single crude entity frequency proxy, the authors introduce a three-factor popularity measurement protocol. For each entity-centric triplet query, three complementary signals are extracted: question entity popularity \(\text{PopQ}\), quantified by the number of Wikipedia sitelinks across languages linking to the Wikidata entity; generated answer entity popularity \(\text{PopGe}\), similarly measured by the sitelink count of the generated entity; and question-answer co-occurrence \(\text{RPopGe}\), computed as the number of independent Wikipedia articles in which both the question entity and the generated entity co-occur. Unlike prior work that only measured ground-truth properties (\(\text{PopGT}\) and \(\text{RPopGT}\)), profiling the generated entities directly captures the model's ungrounded generative priors and explains why a model places unwarranted trust in a specific false answer.

2. Partial Correlation Analysis & Error Attribution: Rigorous OLS Residualization

To eliminate collinearity confounds among popularity signals—such as the inherent tendency of famous question entities to co-occur with famous target entities—the methodology employs rank-space ordinary least squares (OLS) residualization to compute partial Spearman correlation coefficients. For any predictor \(x\) and target \(y\) under a set of control variables \(Z\): $\(\hat{\beta}_x = \arg\min_{\beta} \|\text{rank}(x) - Z\beta\|_2^2, \quad r_x = \text{rank}(x) - Z\hat{\beta}_x\)$ The residualized correlation \(r = \text{Pearson}(r_x, r_y)\) cleanly isolates the unique marginal contribution of each popularity feature. The analysis reveals that across incorrect samples, model confidence maintains a strong, statistically significant positive partial correlation with \(\text{RPopGe}\), confirming that the model's overconfidence on false claims is directly driven by strong relational co-occurrence in the pretraining memory.

3. Lightweight Popularity-Aware Calibrator: Multi-Feature Nonlinear Probability Mapping

To correct overconfidence without touching the underlying language model weights, the authors propose a modular parametric calibration network. The calibrator consists of a lightweight three-layer MLP with hidden layer dimensions of 128, 64, and 32 units, each accompanied by batch normalization, a ReLU non-linearity, and a dropout rate of 0.3. Given the input feature vector \(x\) combining raw token confidence \(c\) and popularity signals, the network computes the calibrated correctness probability via: $\(P(\hat{y} = 1) = \sigma\big(\text{MLP}(x)\big)\)$ Supervised by empirical binary answer correctness \(y_i \in \{0, 1\}\), the network is trained with the binary cross-entropy loss: $\(\mathcal{L}_{\text{CE}} = -\sum_{i=1}^N \big[y_i \log P_i + (1 - y_i) \log (1 - P_i)\big]\)$ Because the input integrates co-occurrence and popularity evidence, the calibrator can distinguish whether high confidence is backed by factual precision or merely inflated by corpus familiarity, pulling down the confidence of popular wrong entities toward 0.

4. Self-Generated Popularity Closed-Loop: Eliminating External Corpus Lookups

To bypass the requirement of real-time lookups into multi-gigabyte Wikipedia dumps during production deployment, the framework explores an internal self-generation paradigm. The LLM itself is prompted to estimate the familiarity of the question entity (\(\widehat{\text{PopQ}}\)), the generated entity (\(\widehat{\text{PopGe}}\)), and the strength of their relational co-occurrence (\(\widehat{\text{RPopGe}}\)). Feeding these model-introspected popularity scores into the calibration MLP yields substantial calibration improvements over standard baselines, proving that LLMs preserve implicit representations of knowledge popularity that can be extracted to police their own confidence boundaries.

Loss & Training

The calibration MLP is trained end-to-end using the Adam optimizer with an initial learning rate of \(5 \times 10^{-4}\) and a weight decay of \(10^{-4}\). To prevent class imbalance from distorting the Alignment metric on datasets with extreme accuracy regimes (e.g., accuracy below 25%), majority-class undersampling is applied on training splits whose natural accuracy falls outside \([0.25, 0.75]\), establishing a balanced 1:1 training prior. The data is partitioned using a stratified 50/50 train/test split with random seed 42. A 10% stratified holdout of the training split is designated as the validation set, and training terminates via early stopping after 10 consecutive epochs without validation BCE loss improvement.

Key Experimental Results

Main Results

The experiments are conducted across six diverse LLMs (Llama-3-8B-Instruct, Qwen2-7B-Instruct, Qwen2.5-7B/14B/32B-Instruct, and ChatGPT GPT-3.5-Turbo-1106) across three benchmarks of varying knowledge popularity: Movies (10,964 samples), Songs (2,157 samples), and Basketball (13,309 samples). The primary metrics are average confidence on incorrect answers \(\text{Conf}_w\) (direct measure of overconfidence), Expected Calibration Error ECE (lower is better), and binary decision Alignment (%) on held-out test splits. The macro-averaged results over the six models are presented below:

Method Movies \(\text{Conf}_w \downarrow\) Movies ECE \(\downarrow\) Movies Align. \(\uparrow\) Songs \(\text{Conf}_w \downarrow\) Songs ECE \(\downarrow\) Songs Align. \(\uparrow\) Basketball \(\text{Conf}_w \downarrow\) Basketball ECE \(\downarrow\) Basketball Align. \(\uparrow\) Avg. \(\text{Conf}_w \downarrow\) Avg. ECE \(\downarrow\) Avg. Align. \(\uparrow\)
Raw Prob 0.780 0.387 81.88 0.775 0.355 80.72 0.741 0.326 68.66 0.765 0.356 77.08
Self-Consistency 0.320 0.157 79.33 0.295 0.121 83.52 0.336 0.173 67.16 0.317 0.150 76.67
Verbalized Confidence 0.497 0.350 64.97 0.584 0.341 65.85 0.388 0.420 57.95 0.490 0.371 62.92
Temp Scaling 0.603 0.247 81.88 0.595 0.218 80.72 0.571 0.147 68.66 0.590 0.204 77.08
Platt Scaling 0.307 0.082 81.88 0.398 0.173 80.72 0.414 0.060 68.66 0.373 0.105 77.08
Isotonic Regression 0.258 0.024 81.88 0.269 0.055 80.72 0.388 0.035 68.66 0.305 0.038 77.08
Prob + \(\text{RPopGe}\) 0.180 0.024 89.54 0.219 0.084 88.42 0.395 0.045 69.87 0.265 0.051 82.61
Prob + All Pop. (Corpora) 0.145 0.021 91.40 0.236 0.095 87.97 0.382 0.035 71.79 0.254 0.050 83.72
Prob + All LLM (Self-Gen) 0.246 0.024 83.49 0.312 0.075 83.57 0.384 0.048 69.99 0.314 0.049 79.02

Ablation Study

To rigorously isolate the benefit of popularity features from the nonlinear expressive power of the MLP, the authors introduce a capacity-matched ablation baseline \(\text{MLP}(c)\), which takes raw confidence alone as input using the exact same network architecture, optimization, and data splits under both balanced and natural class distributions:

Distribution Setting Method Configuration ECE \(\downarrow\) Brier Score \(\downarrow\) AUROC \(\uparrow\) Alignment (%) \(\uparrow\)
Balanced Distribution Vanilla Prob 0.356 0.322 0.839 77.08
Prob + Temp 0.204 0.227 0.839 77.08
Prob + Platt 0.105 0.172 0.839 77.08
Prob + Isotonic 0.038 0.156 0.837 77.08
Capacity Ablation MLP(\(c\)) 0.052 0.158 0.840 77.08
Popularity Calibrator MLP(\(c + \text{RPopGe}\)) 0.050 0.123 0.877 82.62
Natural Distribution Vanilla Prob 0.513 0.421 0.836 86.29
Prob + Temp 0.345 0.258 0.836 86.29
Prob + Platt 0.054 0.108 0.836 86.29
Prob + Isotonic 0.017 0.101 0.835 86.23
Capacity Ablation MLP(\(c\)) 0.031 0.103 0.835 86.29
Popularity Calibrator MLP(\(c + \text{RPopGe}\)) 0.041 0.081 0.876 89.71

Key Findings

  • Standard Supervised Calibration Merely Rescales but Fails to Discriminate: While Platt scaling, isotonic regression, and \(\text{MLP}(c)\) significantly decrease ECE by rescaling probability scores, their monotonic formulation leaves AUROC completely unchanged (0.839 vs 0.840) and produces zero gain in Alignment (77.08%). In sharp contrast, injecting \(\text{RPopGe}\) elevates AUROC to 0.877 and Alignment to 82.62%, demonstrating that popularity features provide crucial discriminatory information to separate true knowledge from hallucinated claims.
  • Question-Answer Co-occurrence Dominates Over Individual Popularity: Among all individual features, \(\text{RPopGe}\) alone achieves the highest Alignment on Movies (87.64%) and Songs (87.26%). While general entity popularity (\(\text{PopGe}\)) offers minor gains, it is the co-occurrence density between the question and the false entity that predominantly tricks the model into assigning overconfident probabilities.
  • Model Scaling Fails to Dissolve Popularity-Induced Overconfidence: Within the Qwen2.5 family (7B, 14B, 32B), larger parameters do not weaken the partial correlation between confidence and \(\text{RPopGe}\) on incorrect answers (remaining around \(+0.20\) to \(+0.25\)). Increased capacity amplifies associative memory, meaning scaling alone cannot resolve popularity-driven overconfidence without explicit boundary calibration.

Highlights & Insights

  • Shifting the Popularity Lens from Ground Truth to Generated Outputs: Previous studies viewed popularity solely as a predictor of factual recall; this work redefines popularity as an inductive bias governing the structure of erroneous generations, proving that errors systematically fall into high-frequency attractor entities.
  • Formal Proof of Monotonic Calibration Failure on Binary Decision Tasks: By pairing ECE with AUROC and Alignment across capacity-matched ablations, the authors elegantly demonstrate that traditional single-scalar recalibration techniques only adjust margin slopes without altering rank ordering, explaining why prior calibration baselines consistently fail downstream decision-making.
  • Zero-Resource Transferability Across Models and Domains: The learned popularity-aware calibration mapping exhibits exceptional cross-model transferability (achieving 0.861 AUROC and 86.00% Alignment across disparate models) and cross-domain robustness, validating that the relationship between corpus frequency and confidence reflects a fundamental statistical property of autoregressive transformers.

Limitations & Future Work

  • External Corpora Serve as Imperfect Proxies for True Pretraining Data: Because commercial and frontier open-source models do not disclose their proprietary training corpora mixes, Wikipedia and Wikidata serve only as external approximations of real-world exposure frequencies.
  • Correlational Evidence Rather than Counterfactual Causal Intervention: The study establishes rigorous statistical associations between knowledge popularity and confidence inflation, but does not conduct counterfactual intervention experiments (e.g., injecting synthetic triplets at controlled frequencies during pretraining).
  • Evaluation Limited to Single-Hop Entity Factoid QA: The formalization assumes clean entity triplets (subject-relation-object); adapting this popularity framework to multi-hop chain-of-thought reasoning, long-form creative generation, or instruction-following remains an open frontier.
  • vs Platt Scaling / Isotonic Calibration (Guo et al., 2017; Zadrozny and Elkan, 2002): Conventional methods fit monotonic 1D transformations on predicted logits; this paper breaks the monotonic bottleneck by infusing relational popularity signals, achieving superior AUROC and discrimination.
  • vs Self-Consistency Sampling (Manakul et al., 2023; Kuhn et al., 2023): Consistency-based uncertainty estimation requires sampling 10+ responses per query and computing semantic clustering, incurring enormous inference latency and compute overhead; the proposed MLP operates on a single greedy decoding pass with negligible overhead.
  • vs Internal-State Activation Probing (Su et al., 2024; Ni et al., 2025): Hidden-state probing requires white-box access to intermediate layer representations; this method relies purely on output token probabilities and entity metadata, making it fully applicable to black-box APIs like ChatGPT.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering perspective connecting pretraining knowledge popularity and entity co-occurrence directly to generated-side overconfidence and calibration.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation covering 6 major LLMs, 3 diverse domains, rigorous OLS partial correlation controls, and capacity-matched ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined statistical rigor, and compelling prose structure without filler content.
  • Value: ⭐⭐⭐⭐⭐ Delivers both foundational scientific insights into LLM pretraining dynamics and an immediately deployable, low-cost calibration tool for reliable AI deployment.