Skip to content

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

Conference: ACL2026 arXiv: 2606.05308 Code: To be confirmed Area: llm_evaluation Keywords: PPI, LLM-as-Judge, Bias Correction, Ranking Evaluation, Precision@K, Semi-supervised Estimation

TL;DR

PRECISE extends Prediction-Powered Inference (PPI) to ranking evaluation metrics. By combining a small number of human annotations with a large volume of LLM judgments, it corrects systemic biases in LLM systems while reducing estimation variance, enabling statistically reliable evaluation of ranking systems.

Background & Motivation

Background: Although LLM-as-a-Judge evaluation methods significantly reduce human annotation costs, they suffer from systemic biases. Directly replacing human annotations with LLM judgments distorts evaluation metrics. Limitations of Prior Work: Existing research primarily focuses on building better judges through prompt engineering, fine-tuning, or multi-agent debate, yet biases persist. Key Insight: This paper takes an orthogonal approach: it accepts that LLM judges are biased and uses statistical methods to correct them. Key Challenge: The core challenge lies in a granularity mismatch for hierarchical metrics (e.g., Precision@K): human annotation is per-document, whereas metrics are calculated per-query. Standard PPI cannot handle this problem because the naive output space is \(O(2^{|C|})\); when the corpus scale reaches millions, computation becomes infeasible.

Method

Overall Architecture

PRECISE is based on the PPI++ semi-supervised estimation framework: it utilizes a small gold-standard set \(\mathcal{D}_g\) (\(n\) human-annotated samples) and a large unlabeled set \(\mathcal{D}_u\) (\(N\) LLM-annotated samples, where \(N \gg n\)), achieving unbiased estimation through a bias correction term.

Key Designs

  1. PPI++ Bias-Corrected Estimator: The estimator is defined as \(\hat{\mu}_{PPI} = \frac{\lambda}{N}\sum_{i=1}^{N}\tilde{\mu}_u^{(i)} + \frac{1}{n}\sum_{i=1}^{n}[\phi_i - \lambda\tilde{\mu}_g^{(i)}]\), where the first term is the LLM-based estimate and the second term is the bias correction. The parameter \(\lambda \in [0,1]\) controls the weight of the LLM signal—when the LLM is well-calibrated, \(\lambda \approx 1\) to fully utilize unlabeled data for variance reduction; when LLM bias is high, \(\lambda \approx 0\) to fall back to a purely gold-standard estimation.
  2. Sparse Reconstruction of Hierarchical Metrics: Since Precision@K only depends on the top-K retrieved documents, the output space is reduced from \(O(2^{|C|})\) to \(O(2^K)\). The probability mass of non-retrieved documents is collapsed into an all-zero K-vector, allowing for exact enumeration when \(K \le 10\).
  3. Joint Distribution under Conditional Independence: For the K documents associated with each query, it is assumed that the LLM provides relevance probabilities \(\tilde{p}'(d_k)\) independently for each document, forming a joint distribution \(\tilde{p}(y) = \prod_{k=1}^{K} \tilde{p}'(d_k)^{y_k}(1-\tilde{p}'(d_k))^{(1-y_k)}\).

Loss & Training

There is no training process. \(\lambda\) is automatically tuned by minimizing the variance of \(\hat{\mu}_{PPI}\). The estimator remains unbiased for any value of \(\lambda > 0\).

Key Experimental Results

Main Results

Evaluated Precision@4 on the ESCI retrieval benchmark (\(n=30\) gold labels, \(N=60K\) LLM annotations):

Estimator Bias (↓) Std. Err. (↓) Inference Cost
Gold only (n=30) 1.04 4.45 —
+ Claude 3 Sonnet 0.70 3.50 $946
+ Claude 3 Haiku 0.29 3.86 $79

Ablation Study

  • Unlabeled/Gold Ratio: The framework saturates at a 100× ratio; \(N=3,000\) LLM queries provide nearly the same standard error as \(N=60,000\).
  • Production A/B Testing: Using \(n=100\) human annotations and \(N=8,400\) LLM judgments, three system variants were ranked within 2 hours (\(T1 \gg T2 \gg Control\)). T1 showed a Gain of +407 bps in daily sales and +571 bps in CTR. LLM-only estimation failed to distinguish variants due to systemic upward bias, whereas PPI correction restored discriminative capability.

Key Findings

  • The sampling distribution of PPI is narrower (lower variance) than gold-only estimation and consistently remains centered on the ground truth (unbiased).
  • Haiku achieved the lowest bias (0.29) at a 12× lower cost, making it the most cost-effective choice.

Highlights & Insights

  • Statistical vs. Engineering Approach: Rather than pursuing a "perfect" LLM judge, the method accepts bias and corrects it statistically—a small amount of gold data guarantees unbiasedness, and every additional LLM annotation reduces variance without introducing new bias.
  • Engineering Significance of Sparse Reconstruction: Reducing the output space of hierarchical metrics from exponential to enumerable makes PPI applicable to real-world ranking evaluation scenarios.
  • Production Validation: The evaluation was completed within 2 hours in a real search system and confirmed via A/B testing, proving its practical utility.

Limitations & Future Work

  • Hierarchical PPI was only validated on Precision@K; other hierarchical metrics (e.g., per-claim factuality, per-turn dialogue quality) were not tested.
  • The conditional independence assumption may not hold in diversity-sensitive ranking scenarios where document relevance is interdependent.
  • The gold set and unlabeled set must be identically distributed; temporal drift might weaken the bias correction effect.
  • PPI/PPI++ (Angelopoulos et al., 2023/2024): Theoretical foundation for this work, applying semi-supervised estimation to ranking evaluation.
  • LLM-as-a-Judge Bias Research (Chen et al., 2024): Confirms systemic bias in LLM judges, supporting the motivation for bias correction.
  • Doubly Robust Estimation (Oosterhuis, 2023): Shares the theoretical foundation and may provide a path for real-time online evaluation.

Rating

Dimension Score (1-10)
Novelty 7
Utility 9
Clarity 8
Experimental Thoroughness 6

Rating

  • Novelty: To be rated
  • Experimental Thoroughness: To be rated
  • Writing Quality: To be rated
  • Value: To be rated