Post-hoc Alignment of LLM-judges to Human Judgment Distribution¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Evaluation
Keywords: LLM-as-a-judge, Human Label Variation, Soft-Label Prediction, Post-hoc Alignment, Entropy-Aware Calibration
TL;DR¶
Addressing the severe misalignment of LLM-judges when predicting unaggregated soft-labels despite human-level hard-label accuracy, this paper proposes NAPHA (eNtropy-Aware Post-Hoc Alignment), which stratifies instances by prediction Shannon entropy and routes them to specialized lightweight MLPs to accurately match human judgment distributions with minimal training overhead.
Background & Motivation¶
In automatic evaluation powered by Large Language Models (LLM-as-a-judge, LLMaJ), standard practice benchmarks model judgments against a single, aggregated ground-truth hard-label. In conventional dataset construction pipelines, even when multiple domain experts or crowd-workers annotate generated text on criteria such as coherence, fluency, sentiment, or ethical violations, researchers routinely collapse annotator disagreement via majority voting, mean pooling, or adjudication. Low inter-annotator disagreement is widely treated as a proxy for high data quality, actively suppressing dissenting voices. However, subjective tasks in natural language processingโsuch as abstractive summarization evaluation, dialogue engagement assessment, adversarial sentiment classification, and moral deliberationโnaturally exhibit irreconcilable Human Label Variation (HLV). Far from being mere annotation noise, such disagreement reflects genuine diversity in cultural backgrounds, personal values, and intrinsic linguistic ambiguity. Suppressing this variation erases minority perspectives, obscures valid uncertainty, and undermines the fairness and generalization of automated evaluators in pluralistic environments.
The central tension in current LLMaJ systems lies in the stark disconnect between hard-label competence and soft-label incompetence: while contemporary LLMs achieve near human-level accuracy when predicting majority-aggregated hard-labels, they fail catastrophically at capturing unaggregated soft-labels, i.e., the empirical Human Judgment Distribution (HJD). Standard calibration typically assumes that a model should maximize confidence around a single true class, whereas HJD modeling requires the predicted probability vector to accurately mirror the proportion of human votes across all candidate categories. Empirical evidence reveals that whether LLMs are prompted directly with soft-label demonstrations (SLP) or queried repeatedly across multiple simulated annotators (SimAnn), their raw output distributions suffer from acute distortions. Specifically, LLMs exhibit a conflict-averse tendency to smooth probabilities toward non-committal intermediate labels, failing to reproduce polarized, multi-modal human disagreements. Crucially, this mis-alignment is highly non-uniform across ambiguity levels: models perform decently on low-disagreement samples but deteriorate drastically on high-entropy instancesโprecisely where capturing pluralistic human perspectives is most critical.
To overcome this bottleneck without requiring intrusive parameter fine-tuning, expensive gradient computation, or restricted logit access, this work argues that attempting to fit a single monolithic calibration function across heterogeneous disagreement strata is fundamentally suboptimal. Core idea: propose NAPHA (eNtropy-Aware Post-Hoc Alignment), a lightweight black-box post-hoc alignment framework that partitions initial model predictions into discrete low, medium, and high entropy strata based on Shannon entropy and routes each instance to a specialized MLP alignment model, achieving sample-efficient and robust alignment to human judgment distributions.
Method¶
Overall Architecture¶
NAPHA operates as an entirely post-hoc, modular pipeline on top of any black-box base LLM evaluator. It requires neither access to internal model representations nor modifications to the underlying LLM weights, relying solely on the generated output distribution over candidate evaluation classes. The overall inference pipeline proceeds through three sequential stages: First, a base LLM judge (such as Claude-4-Sonnet, GPT-OSS-120B, or Qwen3-32B) generates an initial soft-label prediction vector \(\hat{y} \in [0, 1]^n\) over \(n\) discrete rating classes, obtained via soft-label prompting (SLP) or simulated annotator sampling (SimAnn). Second, an entropy calculation module computes the Shannon entropy of \(\hat{y}\) and discretizes the sample into low, medium, or high entropy strata according to empirical dataset terciles. Third, a dynamic router dispatches the prediction vector to an entropy-specific, lightweight Multilayer Perceptron (MLP) alignment model, which applies a non-linear mapping to yield the final aligned soft-label prediction \(\hat{y}_{\mathrm{cal}}\).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Evaluation Instance<br/>Text / Instruction / Candidates"] --> B["Base LLM Judge<br/>SLP or SimAnn Initial Prediction"]
B --> C["Entropy Tercile Stratification<br/>Compute Shannon Entropy & Discretize"]
C --> D["Discrete Entropy-Stratum Dynamic Router<br/>Dispatch Features by Low / Med / High Strata"]
D -->|Low Entropy| E1["Specialized Lightweight Alignment Network (Low-H MLP)<br/>Preserve High-Confidence Modes"]
D -->|Medium Entropy| E2["Specialized Lightweight Alignment Network (Med-H MLP)<br/>Adjust Subset Vote Splitting"]
D -->|High Entropy| E3["Specialized Lightweight Alignment Network (High-H MLP)<br/>Reconstruct Polarized & High-Disagreement Tails"]
E1 --> F["Calibrated Human Judgment Distribution<br/>Final Aligned Soft-Labels"]
E2 --> F
E3 --> F
Key Designs¶
1. Entropy Tercile Stratification: Global uniform calibration ignores stratum heterogeneity -> Compute prediction Shannon entropy and partition via dataset terciles -> Accurately stratify human disagreement intensity
Standard post-hoc calibration techniques implicitly assume that model mis-calibration is homogeneous across the input domain. In subjective evaluation, however, human disagreement varies fundamentally between unambiguous consensus samples and culturally contentious or linguistically ambiguous dilemmas. A global mapping trained indiscriminately across all data points is inevitably dominated by majority patterns, failing to adjust disparate regions of the probability simplex. To resolve this, NAPHA first computes the discrete Shannon entropy over the base model's normalized prediction vector \(\hat{y} = (\hat{y}_1, \dots, \hat{y}_n)\):
$\(H(\hat{y}) = -\sum_{j=1}^n \hat{y}_j \log(\hat{y}_j)\)$
Rather than relying on arbitrary manual thresholds, the continuous entropy distribution across calibration instances is partitioned into three balanced strata using empirical terciles \(Q_1\) and \(Q_2\):
$\(c = \begin{cases} \text{low}, & \text{if } H(\hat{y}) \le Q_1 \\ \text{medium}, & \text{if } Q_1 < H(\hat{y}) \le Q_2 \\ \text{high}, & \text{if } H(\hat{y}) > Q_2 \end{cases}\)$
This tercile-based discretization cleanly segregates clear consensus instances, nuanced split-vote instances, and maximally polarized dilemmas. The authors confirm empirically that collapsing the strata into a binary split (low vs. high) degrades alignment, as it fails to isolate the delicate transition dynamics present in moderately ambiguous instances.
2. Discrete Entropy-Stratum Dynamic Router: Monolithic functions fail on conflicting mis-alignment trends -> Route predictions conditionally to dedicated stratum models -> Decouple non-linear alignment parameter spaces
When an alignment function attempts to map all instances simultaneously, it must reconcile opposing transformation requirements: low-entropy instances require dampening overconfidence without disturbing the dominant class, whereas high-entropy instances require redistributing probability mass across competing modes. NAPHA adopts a piece-wise Mixture-of-Experts (MoE) routing formulation:
$\(\hat{y}_{\mathrm{cal}} = \sum_{c \in \{\text{low}, \text{medium}, \text{high}\}} \mathbb{I}(c(\hat{y}) = c) \cdot M_c(\hat{y})\)$
where \(\mathbb{I}(\cdot)\) is the indicator routing function. By constraining the sub-problem assigned to each dedicated expert \(M_c\), NAPHA drastically reduces the functional complexity required of each mapping. Specialized networks can independently adjust their transformation curvature, ensuring that aggressive mass redistribution in high-disagreement scenarios never corrupts the sharp unimodal predictions of low-disagreement items.
3. Specialized Lightweight Alignment Network: Parameter-heavy fine-tuning is compute-intensive and risks forgetting -> Train compact single-hidden-layer MLPs per stratum -> Achieve highly efficient and flexible multi-label probability warping
Fine-tuning foundation models (via SFT or LoRA) to output calibrated soft-labels requires prohibitive computational resources, risks catastrophic degradation on other reasoning tasks, and remains inaccessible for proprietary APIs. Instead, NAPHA instantiates each alignment model \(M_c\) as a lightweight Multilayer Perceptron (MLP) consisting of a single hidden layer with a ReLU non-linearity. The input and output dimensions are both \(n\) (the number of task rating categories), while the hidden layer dimension is compactly set to \(2n\). A final Softmax activation guarantees valid output probabilities. Despite consisting of only a few hundred parameters, this parameterization provides sufficient expressive capacity to learn arbitrary non-linear probability redistributions, consistently outperforming traditional calibration baselines such as linear transformations, temperature scaling, and parameterized temperature scaling.
Post-Hoc Calibration and Optimization Objectives¶
During training, the ground-truth Human Judgment Distribution is denoted as \(y \in [0, 1]^n\), estimated via maximum likelihood from \(m\) human annotations (\(y_j = \frac{1}{m} \sum_{k=1}^m \mathbb{I}(r_k = a_j)\)). The objective of each specialized alignment network \(M_{\phi_c}\) is to minimize the Kullback-Leibler (KL) divergence between the true distribution \(y\) and the mapped distribution \(\hat{y}_{\mathrm{cal}} = M_{\phi_c}(\hat{y})\): $\(\mathcal{L}(\phi_c) = \mathbb{E}_{(\hat{y}, y) \sim \mathcal{D}_c} \left[ D_{\mathrm{KL}}(y \parallel \hat{y}_{\mathrm{cal}}) \right] = \mathbb{E}_{(\hat{y}, y) \sim \mathcal{D}_c} \left[ \sum_{j=1}^n y_j \log \frac{y_j}{\hat{y}_{\mathrm{cal}, j}} \right]\)$ Weight decay regularization is incorporated during optimization to prevent overfitting on small calibration splits. Theoretically, the global expected alignment error across the full data distribution is the stratum-probability weighted sum of conditional errors: $\(\mathbb{E}[M] = \sum_{c \in \{\text{low}, \text{medium}, \text{high}\}} P(x \in X_c) \cdot \mathbb{E}[M_{\phi_c} \mid x \in X_c]\)$ Because the input dimensionality is strictly restricted to \(n\) and each stratum model optimizes over a cohesive sub-manifold, NAPHA exhibits extreme sample efficiency: model performance stabilizes with as little as 10% of training data, requiring no extensive human label collection.
Key Experimental Results¶
Main Results¶
The framework is comprehensively evaluated across five diverse human-annotated datasets spanning varying subjective dimensions: news summarization multi-criteria scoring (SummEval, expert annotations), multi-dimensional dialogue next-utterance evaluation (TopicalChat, crowd annotations), high-disagreement natural language inference (ChaosNLI, 100 crowd annotations per sample), adversarial sentiment classification (DynaSent Round 2), and community ethical judgments (Anecdotes, Reddit AITA with \(\ge 15\) votes per post). The primary backbone is Claude-4-Sonnet evaluated at temperature \(t=0\), alongside GPT-OSS-120B and Qwen3-32B in appendix studies. Evaluation metrics comprise Distribution Calibration Error (DistCE, \(\downarrow\)), measuring the maximum absolute probability discrepancy across events, and Jensen-Shannon Distance (JSD, \(\downarrow\)).
The table below summarizes soft-label prediction performance under a realistic 20/80 train/test split across 20 independent runs. Baselines include raw base models (Base), NAPHA with predicted entropy routing (NAPHA Predicted), and the theoretical upper bound using oracle ground-truth entropy routing (NAPHA Oracle):
| Dataset | Metric | SLP-HE (Base) | SLP-HE (+NAPHA) | SLP-SE (Base) | SLP-SE (+NAPHA) | SimAnn (Base) | SimAnn (+NAPHA) | Oracle Bound (SLP-SE) |
|---|---|---|---|---|---|---|---|---|
| Anecdotes | DistCE \(\downarrow\) JSD \(\downarrow\) |
0.299 0.326 |
0.271 (0.001) 0.292 (0.001) |
0.299 0.326 |
0.272 (0.002) 0.294 (0.001) |
0.325 0.385 |
0.293 (0.001) 0.308 (0.001) |
0.172 (0.001) 0.224 (0.001) |
| ChaosNLI | DistCE \(\downarrow\) JSD \(\downarrow\) |
0.221 0.259 |
0.187 (0.002) 0.213 (0.001) |
0.200 0.230 |
0.174 (0.001) 0.198 (0.001) |
0.247 0.305 |
0.190 (0.002) 0.216 (0.001) |
0.153 (0.002) 0.186 (0.002) |
| DynaSent | DistCE \(\downarrow\) JSD \(\downarrow\) |
0.268 0.313 |
0.263 (0.003) 0.306 (0.002) |
0.272 0.330 |
0.265 (0.002) 0.305 (0.001) |
0.296 0.367 |
0.294 (0.002) 0.323 (0.001) |
0.172 (0.004) 0.249 (0.003) |
| SummEval | DistCE \(\downarrow\) JSD \(\downarrow\) |
0.433 0.465 |
0.303 0.356 |
0.371 0.453 |
0.303 0.358 |
0.562 0.560 |
0.305 0.360 |
0.263 0.335 |
| TopicalChat | DistCE \(\downarrow\) JSD \(\downarrow\) |
0.363 0.404 |
0.349 0.376 |
0.374 0.424 |
0.363 0.385 |
0.374 0.433 |
0.349 0.375 |
0.312 0.363 |
Note: Values in parentheses indicate standard errors (SE) across 20 runs. SummEval and TopicalChat present macro-averages across all evaluated dimensions.
Ablation Study & Stratified Analysis¶
To evaluate the necessity of entropy stratification and model architectures, the authors conduct granular ablations across entropy tiers and alternative calibration baselines. The table below displays per-stratum performance on the Anecdotes dataset using the SLP-SE backbone:
| Configuration / Stratum | Overall DistCE \(\downarrow\) | Overall JSD \(\downarrow\) | Low-H DistCE \(\downarrow\) | Low-H JSD \(\downarrow\) | Med-H DistCE \(\downarrow\) | Med-H JSD \(\downarrow\) | High-H DistCE \(\downarrow\) | High-H JSD \(\downarrow\) |
|---|---|---|---|---|---|---|---|---|
| Raw Base (SLP-SE Base) | 0.299 | 0.326 | 0.196 | 0.279 | 0.311 | 0.352 | 0.388 | 0.341 |
| Single Unstratified MLP | 0.268 | 0.288 | 0.216 | 0.285 | 0.294 | 0.313 | 0.294 | 0.262 |
| NAPHA (Predicted Routing) | 0.272 (0.002) | 0.294 (0.001) | 0.226 (0.004) | 0.295 (0.003) | 0.290 (0.002) | 0.316 (0.002) | 0.299 (0.003) | 0.269 (0.002) |
| NAPHA (Oracle Routing) | 0.172 (0.001) | 0.224 (0.001) | 0.034 (0.003) | 0.119 (0.003) | 0.274 (0.002) | 0.304 (0.001) | 0.209 (0.001) | 0.211 (0.001) |
In alignment architecture comparisons (Table 14 on Anecdotes): - Linear Transformation achieves DistCE 0.285 (Oracle 0.191). - Temperature Scaling degrades substantially to DistCE 0.365 (Oracle 0.265). - Parameterized Temperature Scaling (PTS) yields DistCE 0.310 (Oracle 0.320). - Dirichlet Calibration achieves DistCE 0.267 (Oracle 0.176), competitive with MLP. - NAPHA MLP achieves DistCE 0.272 (Oracle 0.172), offering optimal non-linear warping flexibility.
Key Findings¶
- Severe Disconnect Between Hard- and Soft-Label Performance: In hard-label prediction, Claude-4 achieves macro F1 scores of 0.99 (low H), 0.81 (medium H), and 0.61 (high H) on ChaosNLI, surpassing average human annotators (0.93, 0.74, 0.53). Conversely, raw soft-label DistCE sits between 0.20 and 0.43, demonstrating that superior discrete classification skill does not translate into faithful population distribution modeling.
- Maximal Gains on High-Entropy Disagreements: On Anecdotes, raw base models exhibit an extreme high-entropy DistCE of 0.388. NAPHA slashes this error to 0.299 (and 0.209 under Oracle routing). Because high-entropy instances encapsulate polarized societal conflicts, NAPHA delivers its largest improvements exactly where diversity representation is most urgent.
- Routing Misclassification Explains Low-Entropy Penalty: Under predicted entropy routing, low-entropy DistCE slightly increases (from 0.196 to 0.226 on Anecdotes), whereas under Oracle routing it plunges to 0.034. This proves the degradation stems entirely from base model misclassification of entropy tiers rather than sub-model failure. In practice, a hybrid rule can retain raw base predictions when entropy is flagged as low.
- Extreme Data Efficiency: Training size ablations indicate that scaling down training data from 50% to 25% and 10% causes virtually no degradation in alignment gains (ChaosNLI DistCE remains constant at 0.171, 0.170, and 0.170), confirming that NAPHA can be trained on minimal human-annotated budgets.
Highlights & Insights¶
- Decoupling Uncertainty from Population Pluralism: The paper formalizes a critical conceptual distinction: in HLV contexts, a soft-label vector does not represent epistemic uncertainty regarding one elusive true answer, but aleatoric variation where multiple answers are concurrently true across diverse human groups.
- Exposing the "Conflict-Aversion" Bias in Aligned LLMs: Qualitative Chain-of-Thought inspection demonstrates that safety-tuned LLMs gravitate toward mediator stances ("nobody is in the wrong"), whereas human annotators are deeply polarized (e.g., 2/3 blaming the author and 1/3 blaming the partner). This polite homogenization is a primary driver of soft-label miscalibration.
- Turnkey Black-Box Deployability: By requiring only output token probabilities and a tiny single-layer MLP, NAPHA introduces negligible compute overhead (<1ms per instance), providing a drop-in alignment layer suitable for closed-source APIs and production evaluation platforms alike.
Limitations & Future Work¶
- Bottleneck in LLM Entropy Self-Assessment: NAPHA's routing relies on the entropy of the base LLM's own predictions, yet LLMs achieve weighted F1 scores of only 0.35โ0.56 when classifying their own entropy tier. Developing dedicated, external entropy predictors represents a high-leverage path toward closing the gap to Oracle performance.
- Sparse Human Annotations in Benchmark Datasets: Three out of five evaluated benchmarks (SummEval, TopicalChat, DynaSent) possess fewer than 6 annotators per instance, introducing high empirical sampling variance into the ground-truth soft labels.
- Lack of White-Box Features and Sociodemographic Metadata: The study treats LLMs purely as black boxes and does not exploit internal hidden representations or output logit manifolds. Furthermore, annotator sociodemographics (such as age, gender, and nationality) are omitted, precluding personalized or demographic-targeted distribution alignment.
Related Work & Insights¶
- vs. Conventional LLM-as-a-judge (e.g., MT-Bench / AlpacaEval): Mainstream LLM evaluation benchmarks force consensus through scalar score averaging; this work pioneers distribution-level evaluation, revealing that high correlation with majority labels conceals massive errors in distribution matching.
- vs. Temperature Scaling and Standard Calibration (Guo et al., 2017): Classical calibration aligns scalar confidence with expected empirical accuracy; NAPHA targets the entire multi-dimensional categorical distribution via entropy stratification and KL minimization.
- vs. Simulated Annotators (SimAnn / Jung et al., 2025): SimAnn requires sampling multiple reasoning traces at elevated temperatures (10+ inference calls per instance), multiplying API costs while remaining biased; NAPHA achieves superior alignment via single-inference post-hoc warping.
Rating¶
- Novelty: โญโญโญโญโ [Pioneers the study of Human Label Variation in LLM judges and introduces an effective, lightweight entropy-aware post-hoc alignment mechanism.]
- Experimental Thoroughness: โญโญโญโญโญ [Spans five diverse subjective benchmarks, three model families, 20 independent runs, comprehensive ablations, and insightful qualitative analyses.]
- Writing Quality: โญโญโญโญโญ [Clear mathematical formulation, cohesive narrative, and lucid presentation of empirical trade-offs.]
- Value: โญโญโญโญโญ [Provides a practical, low-cost recipe for deploying pluralistic and fair automated evaluation in real-world systems.]