LLM Judge Validation Under Sparse Overlap: From Inference to Design

Conference: NeurIPS2026
arXiv: 2609.31857v2
Area: LLM Evaluation
Keywords: LLM-as-a-judge, sparse annotation overlap, inter-annotator agreement, stratified sampling, sample-size planning

TL;DR

This paper treats the quantity and allocation of overlapping annotations for LLM-judge validation as a statistical design problem: variance analysis guides overlap planning, while stratified allocation improves representativeness; low overlap substantially changes deployment decisions and judge rankings on real benchmarks, but stratification can increase false approval even as it reduces false rejection.

Background & Motivation

Validating an LLM judge requires more than counting its ratings: its agreement with humans should reach a level comparable to agreement among humans. Annotation budgets can often support one initial pass across a corpus, but not multiple human labels for every item. Consequently, a judge can assess the entire corpus cheaply while human annotators share few items; the human-agreement reference needed for validation can itself become unstable.

Broad single-pass coverage and concentrated relabeling have different pitfalls. Independent small samples leave human pairs with substantially less co-coverage than each person's individual coverage. Having everyone label the same initial slice provides overlap, but data ordering may bias its difficulty or label composition. Switching agreement coefficients cannot recover missing shared observations, and some chance-corrected coefficients amplify noise under label skew. The paper therefore studies neither judge training nor inference of a uniquely correct label, but reliable measurement of judge agreement relative to humans under a limited relabeling budget.

The paper first analyzes sparse agreement-estimation noise, compares random, static stratified, and sequential allocation, and then converts error targets into sample-size plans. Core idea: secure enough shared observations for each validation rater, then allocate them using informative strata, rather than treating an unstable human-agreement estimate as a reliable deployment threshold.

Method

Overall Architecture

Inputs are evaluation items, complete candidate-judge labels, initial human labels, and a limited relabeling budget. The analysis path specifies the overlap structure and agreement estimand before using leave-one-out comparisons to decide whether a judge meets the threshold. The design path uses variance and decision margins to plan the budget and allocate subsequent labels through stratification. Outputs are acceptance/rejection decisions relative to human agreement or candidate rankings, not a trained model.

Solid arrows show statistical analysis of available data; the dashed arrow feeds the resulting plan into the next annotation round. This is not a neural network and has no training supervision or back-propagated loss.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Items and initial labels"] --> B["Overlap structure and<br/>agreement estimation"]
    B --> C["Leave-one-out validation"]
    C --> D["Overlap budget planning"]
    D --> E["Stratified annotation allocation"]
    E -.->|Re-estimate after relabeling| B
    C --> F["Accept, reject, or rank"]

Key Designs

1. Overlap structure and agreement estimation: distinguish individual coverage from actual shared observations

Let the corpus contain \(n\) items, \(K\) humans, and one judge. The judge labels every item; the simplified setting also has one primary human cover the full corpus while secondary humans label subsets. Multiple humans can collectively supply the initial layer, but their heterogeneity cannot consequently be ignored. Without a complete initial layer, the problem becomes a more general incomplete-block allocation problem, and the simplified derivations do not transfer unchanged.

Two overlap rates must be distinguished. The paper initially defines global \(\rho\) as the average overlap rate across all rater pairs, but subsequent designs and experiments also use each secondary rater's coverage rate. These are different quantities: when secondary masks are independent with equal coverage probability, expected overlap with the complete primary or judge is \(\rho n\), whereas overlap between two secondaries is only \(\rho^{2}n\). Thus \(m=\rho n\) converts a rate into a count only when the rate explicitly means each secondary's coverage; it cannot directly use the global pairwise average.

The default coefficient is observed agreement: the fraction of co-labeled items receiving the same label. Conditional on a fixed nonempty overlap set, with independent and identically distributed items and a label-independent mask, this sample mean is unbiased and its variance decreases inversely with shared sample count. Chance-corrected coefficients also estimate a denominator: Krippendorff's \(\alpha\) compares observed with expected disagreement, Cohen's \(\kappa\) corrects for chance agreement based on rater marginals, and Gwet's AC1 uses a different marginal-symmetric chance term.

\[ \mathrm{Var}(\hat p_{o,m})=\frac{p_o(1-p_o)}{m},\qquad \mathrm{Var}(\hat F_m)=\frac{\gamma_F(\pi)}{m}+O(m^{-2}). \]

The second relationship is the paper's asymptotic form, not a finite-sample guarantee for arbitrary sparse matrices. With fixed positive denominators, ratio estimators also retain inverse-sample-size variance and typically inverse-sample-size bias; observed agreement is exactly unbiased under the stated conditions. The simplified theory additionally uses fixed marginals, rater exchangeability, and the appendix's balanced overlap weights for multiply annotated items.

Under extreme label skew, the chance-correction denominators of \(\kappa\) and \(\alpha\) can become small, increasing local sensitivity. AC1's denominator satisfies \(1-p_e^{\mathrm{AC1}}\geq (L-1)/L\geq 1/2\) for any label distribution, preventing this denominator degeneration. Its interpretation of chance agreement differs, however, so it is not an unconditionally more correct coefficient.

One proof boundary is essential: the appendix's \(O((1-p_e^\kappa)^{-4})\) is a conservative upper bound, not a lower bound establishing necessary divergence at that rate. Comparing two big-O exponents alone does not prove that \(\kappa\) necessarily diverges faster than \(\alpha\). Numerator variances and covariances also change as marginals concentrate and may cancel; the amplification coefficient in the asymptotic relationship also cannot generally be determined solely by prevalence for an arbitrary joint distribution.

2. Leave-one-out validation: compare judge and human against the pool, not judge directly against the held-out human

Each comparison holds out one human and uses the remaining humans as the held-in pool. Eligible items require the held-out human, the judge, and at least one pool human to be observed. The judge score compares the judge against pool humans; the human reference compares the held-out human against the same pool. It is not direct agreement between judge and held-out human. Available item–rater pairs are pooled, so items with more pool labels contribute more pairs.

Denote the difference between these scores by \(d_k\). The paper allows a tolerance \(\varepsilon=0.05\), permitting the judge to fall slightly below the human reference. The judge is accepted when at least half of the leave-one-out comparisons pass.

\[ d_k=S(j;r_k)-S(r_{-k};r_k),\qquad T_k=\mathbf 1[d_k\geq-\varepsilon],\qquad \hat\omega=\frac{1}{K}\sum_{k=1}^{K}T_k,\qquad \text{accept iff }\hat\omega\geq0.5. \]

This rule transmits measurement uncertainty into deployment decisions: an inflated human reference can cause false rejection, while a low reference or an upward fluctuation in the judge difference can cause false approval. Real-benchmark experiments use the complete matrix's decision as a finite-sample reference. An “error” here means disagreement with that reference, not an error relative to objective truth, and does not certify AI safety or clinical suitability.

Leave-one-out comparisons share items and human pools, so they cannot automatically be treated as independent Bernoulli trials. Secondary-human overlap counts are usually the bottleneck. When the complete primary is held out, eligible items still require at least one secondary label; the entire corpus does not automatically enter that comparison.

3. Overlap budget planning: let the decision margin determine the shared sample count instead of prescribing one percentage

The appendix first applies the central limit theorem to per-item differences under observed-agreement scoring, then translates Gaussian tail probabilities into the overlap count needed for each human comparison. If the population mean difference plus tolerance is positive and per-item difference variance is \(\sigma_0^2\), the false-rejection probability of an individual leave-one-out comparison approximately depends on its standardized margin.

\[ \Pr(d_k<-\varepsilon)\approx \Phi\!\left(-\frac{(\mu_d+\varepsilon)\sqrt m}{\sigma_0}\right),\qquad m^*_{\mathrm{cert}}= \left\lceil\frac{z_{1-\alpha_{\mathrm{sig}}/2}^{\,2}\sigma_0^2}{(\mu_d+\varepsilon)^2}\right\rceil. \]

This target addresses an individual comparison's approximate false-rejection probability of at most \(\alpha_{\mathrm{sig}}/2\) and requires preliminary variance and margin estimates. It is neither a finite-sample error certificate for the full voting pipeline nor simultaneous control of false approval for arbitrary weak judges. Chance-corrected ratios can use the same planning form through the delta method, but are not the per-item averages in the original proof. Unbalanced real pair weights also require estimating the appropriate variance rather than treating the simple-mean proof as a complete derivation.

Selecting the best of several candidates requires distinguishing the best judge from its closest competitor and controlling multiple comparisons. The appendix assumes approximately equal single-judge variances, neglects covariance between scores, writes difference variance as twice the single-score variance, and applies Bonferroni control over competitors.

\[ m^*_{\mathrm{rank}}= \left\lceil\frac{2z_{1-\alpha_{\mathrm{sig}}/(J-1)}^{\,2}\sigma_0^2}{\Delta_{\min}^2}\right\rceil. \]

Here \(J\) is the candidate count and \(\Delta_{\min}\) the smallest population score gap between the best candidate and any competitor. The doubled variance requires zero or negligible covariance; approximately equal variances alone do not imply it. Real judges evaluated on shared items can be correlated. Typically small gaps and stricter multiplicity control make ranking harder than acceptance/rejection in this paper's data, but do not imply that every ranking task is necessarily harder.

4. Stratified annotation allocation: inspect representativeness and co-coverage separately

Static stratification, Strat, uses the complete primary human's labels as strata. Each secondary samples relabeling items in proportion to stratum sizes, with residual slots assigned to larger strata. When primary labels track the relevant marginals, subsets are less likely to miss rare categories, reducing marginal mismatch in the human-agreement reference. “Zero cost” means reallocating an unchanged label budget, not eliminating initial labels or organizational costs.

Seq-Refined updates the stratum signal using per-item modes of human labels already collected; Seq-Coverage additionally favors less-covered items. Tasks are fixed before the next human begins, without per-item adaptation during a person's session. Updating strata can help if existing consensus is more accurate, but coverage balancing can conflict with representativeness. Greater complexity is not necessarily better.

Representativeness and fragmentation are distinct: proportional sampling does not itself guarantee that secondaries label the same items. The introductory demonstration and Appendix B.1 explicitly use a shared stratified panel with judge-label strata, whereas the main design defines secondary-specific subsets using primary-human labels. This note does not merge these implementations into a theorem that every Strat design automatically maximizes secondary pairwise overlap. Implementation should record each secondary's coverage, secondary pairwise overlap, and whether a panel is actually shared.

Selecting items by observed labels makes the mask label-dependent, so it does not directly satisfy Theorem 1's independent-mask assumption. Stratification's advantage is primarily supported by mechanism and experiments; it requires within-stratum conditioning or a separate design-based derivation, rather than being directly proved by the independent-mask theorem. Static strata can degenerate under extreme skew, but whether sequential designs improve matters must be checked for each configuration, not inferred solely from a skew threshold.

A Worked Example

Consider \(n=500\) from the main synthetic setup and per-secondary coverage \(\rho=0.05\). Each secondary shares an expected 25 items with the complete primary, but two independently sampled secondaries share only 1.25 items on average. This explains why substantial individual annotation does not ensure a stable human reference. These counts are illustrative calculations from independent masks, not new experimental results.

To validate, hold out one secondary and use pool-human labels on its eligible shared items to compute the judge and held-out-human scores before applying the tolerance. Planning the next round does not train the judge: estimate difference variance and distance from the threshold, determine additional overlap, and allocate it through informative strata. Selecting the best judge additionally requires inspecting candidate gaps and dependence induced by shared items.

Key Experimental Results

Main Results

Main synthetic experiments use 500 items, 4 or 5 humans, 2 or 5 labels, human accuracy 0.85, and usually 300 repetitions per condition. Synthetic annotations follow latent categories and a symmetric noise channel with conditionally independent, homogeneous humans. These latent population parameters differ from the finite complete-matrix reference in real data and must not be interchanged.

Real evaluation covers 10 judges, 3 benchmark sources, and 4 matrices: skin-lesion visual-attribute ratings in WAX, star and aspect subtasks in CeBaB, and summarization ratings in SummEval. Newly collected judge labels total 28,822. Main WAX evaluation uses 246 items and 8 humans; CeBaB-aspects uses 1,008 aspect-level items and 10 humans; SummEval uses 6,400 dimension-level items and 3 humans. Visual data are used only to study agreement, not to provide a medical diagnosis procedure.

The following rows are selected from original Table 3. The count 40 means combinations of 10 judges and 4 matrices, not 40 distinct models. FR averages over 20 strong-pass cells, FA over 16 reject cells, and WDR over all 40 cells, including 4 borderline-pass cells. Therefore, WDR cannot be reconstructed by weighting only the FR and FA columns.

Original Table 3: per-secondary coverage	Design	Mean FR	Mean FA	Mean WDR
0.05	Random	.290	.112	.250
0.05	Strat	.142	.172	.184
0.05	Seq-Refined	.308	.098	.250
0.05	Seq-Coverage	.435	.066	.311
0.25	Random	.122	.071	.149
0.25	Strat	.027	.114	.092
0.25	Seq-Refined	.113	.070	.146
0.25	Seq-Coverage	.158	.075	.174
0.50	Random	.021	.079	.089
0.50	Strat	.007	.110	.077

Strat reduces false rejection at low overlap while increasing false approval; it does not dominate on both error types. At per-secondary coverage 0.25, its mean FR is .027, FA remains .114, and WDR is .092. This supports substantial improvement in false rejection for strong-pass judges, not reliable certification of all non-borderline judges on every task.

The next table selects ranking results from original Table 2. Rank err. is the fraction of pairwise orderings reversed relative to the complete reference; Top-1 is the probability of not selecting the reference-best judge. They measure different failures.

Original Table 2: per-secondary coverage	WAX Top-1	CeBaB-asp. Top-1	CeBaB-stars Top-1	SummEval Top-1	Mean Rank err.	Mean Top-1
0.05	.670	.887	.875	.177	.278	.652
0.25	.427	.507	.517	.003	.137	.363
0.50	.270	.213	.350	.000	.082	.208

Mean Top-1 error remains .208 even at coverage 0.50; crossing a deployment threshold does not imply reliable best-candidate selection. SummEval's larger sample makes its ranking more stable, but Appendix C.3.1 rejects every judge under the complete reference. Stable ranking does not mean that a judge is approved.

Ablation Study

The main analytical comparisons change annotation design rather than ablate network modules. The following configurations are selected from original Table 1 to examine bias, standard deviation, and tolerance reliability of human-pool Krippendorff agreement relative to the complete reference. Reliability is the fraction of trials with absolute estimation error at most 0.05, not a deployment pass rate.

Original Table 1: configuration	Coverage	Design	Absolute bias	Std	Reliability
cebab_stars (.26)	.05	Random	.087	.126	.22
cebab_stars (.26)	.05	Strat	.001	.074	.49
cebab_stars (.26)	.25	Random	.039	.040	.58
cebab_stars (.26)	.25	Strat	.001	.028	.93
lesion@t=2 (.93)	.25	Random	.035	.066	.51
lesion@t=2 (.93)	.25	Strat	.055	.080	.41
lesion@t=2 (.93)	.25	Seq-Coverage	.021	.045	.70

The star-rating configuration shows that informative static strata can reduce both bias and sampling spread; the extremely skewed lesion configuration provides a counterexample. High skew is constructed by class merging rather than occurring naturally in every original benchmark, so this analysis cannot directly characterize unmodified real tasks.

Key Findings

• Error types matter more than one aggregate score. Reduced false rejection accompanies increased false approval in Table 3. Design selection should reflect both costs, rather than minimize FR alone.
• Borderline cells remain difficult. Original Table 4 gives Strat borderline-pass WDR of .440, .323, and .295 at coverage 0.05, 0.25, and 0.50. The Gaussian intuition of approximately half errors at a threshold is not a universal assertion about every estimator.
• Absolute overlap counts transfer better than fixed rates. Appendix C.4's corpus-size experiments support planning shared sample counts; rate recommendations depend on variance, decision margins, prevalence, and corpus size.
• Clustering weakens independent-sample approximations. The strongly clustered configuration in Appendix D, Table 20 has reliability 10.0%, 72.3%, and 99.0% at coverage 0.05, 0.25, and 0.50. Increasing samples or stratifying by cluster is preferable to blindly reusing an independent-item formula.

Highlights & Insights

• Move missing-data concerns into experimental design. Instead of only changing estimators on an existing sparse matrix, the paper asks where the next annotation should go. Its transferable lesson is to report budgets, co-coverage structure, and representativeness, not merely annotator counts.
• Separate ranking from threshold validation. Ranking resolves small gaps between judges, whereas acceptance/rejection crosses a predefined human reference. Their annotation budgets should be configured separately rather than sharing one “sufficiently validated” label.
• Treat human agreement as an estimand. The human pool is not an intrinsically error-free ceiling; representativeness bias moves that ceiling. A more accurate reference can even increase approval relative to the previous threshold, so statistical improvement does not imply monotonic improvement in every risk measure.

Limitations & Future Work

• The theory has limited scope. Independent and identically distributed items, label-independent masks, fixed positive denominators, balanced overlap, and simplified exchangeability are strong assumptions. Heterogeneous humans, shared difficulty, and label-driven strata require more general weighted or design-based analyses.
• Proof sketches do not replace complete covariance calculations. Appendix A.2's \(\alpha\) ratio-variance expansion omits numerator–denominator covariance; the displayed \(\kappa\) cross-term sign also disagrees with its stated gradient. This note retains the variance order for fixed nondegenerate denominators, without using that expansion to prove a strict divergence ordering.
• Budget formulas are asymptotic plans, not full-pipeline certificates. Individual leave-one-out comparisons, candidate-score dependence, finite-corpus sampling corrections, and final-vote dependence need separate treatment. Inaccurate preliminary variance estimates also change sample targets.
• Some recommendations disagree with their tables. Appendix E.1 says Seq-Coverage beats Strat on both extremely skewed configurations, but Table 1 gives cebab_aspects@t=0 reliability .53 for Strat and .36 for Seq-Coverage at coverage .25. Its claim that high-overlap design differences are approximately .05 also conflicts with .41 versus .70 for lesion@t=2. Decisions should use configuration-specific evidence rather than a universal fallback rule.
• Different experimental results cannot be silently merged. Appendix C.2 reports nonzero random low-coverage WDR on SummEval, whereas C.3.1 states zero WDR for every judge at every design and overlap. The cached text does not resolve the metric/implementation conflict. Appendix B.1's shared-panel demonstration also differs from the main evaluation in item count, human count, and stratum signal.
• Some wording exceeds the evidence. Coverage 0.25 and 3–5 humans are empirical starting points under particular conditions; false approval, borderline decisions, and best-judge selection can remain unstable. More overlap does not guarantee monotonically decreasing errors in every finite-repetition configuration.
• Reproduction materials are incomplete. Several checklist Yes answers coexist with explanations that there is no unified code bundle and that proprietary-run prompts and versions are incompletely specified. The full text provides no verifiable code-repository link; this note does not invent one.

Related Work & Insights

• vs Alternative Annotator Test (Calderon et al.): The paper retains human leave-one-out validation but focuses on sparsity in its input annotation matrix and prospective budget allocation, rather than introducing a new judge model.
• vs Sparse Probability of Agreement: That work studies agreement estimation on already-collected sparse matrices; this paper additionally asks how to prevent instability before collecting labels. Better estimators and allocation designs are complementary.
• vs PPI / StratPPI: These methods use model predictions to improve inference for population performance statistics. This paper directly targets human agreement and judge thresholds, with stratification initially deciding who relabels which items. Different estimands prevent interchangeability merely because all use stratification.
• Research extensions: Develop explicit design-based variance estimation for label-driven strata, preserve cross-judge and cross-comparison covariance, and add an inconclusive outcome near thresholds. These are future directions, not guarantees already established by the paper.

Rating

• Novelty: 4/5 — Elevates sparse co-annotation quantity and allocation to primary design variables in judge validation.
• Experimental Thoroughness: 4/5 — Includes synthetic and real matrices, ranking, and design sensitivity, but inconsistent conventions and reproduction gaps constrain conclusions.
• Writing Quality: 3/5 — The practical problem is clear; theorem quantifiers, proof covariance terms, and appendix recommendations need more rigorous consistency checks.
• Value: 4/5 — Useful for evaluation design under limited annotation budgets, but not objective-truth validation or safe-deployment certification.