CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Evaluation
Keywords: LLM Evaluation, Judge Bias, Variance Decomposition, Round-Robin Scheduling, Generalizability Theory
TL;DR¶
Addressing systemic scoring biases in LLM-as-a-judge evaluation that cannot be averaged away by increasing test scenarios, this work establishes a generalizability theory variance decomposition and demonstrates that a round-robin judge assignment strategy (CyclicJudge) cancels panel-relative judge offsets and maximizes generation diversity at zero additional cost, cutting top-1 ranking flips by 61%.
Background & Motivation¶
Large language models serving as evaluators (LLM-as-a-judge) have emerged as the standard paradigm for open-ended model benchmarking, anchoring widely used suites like MT-Bench and Chatbot Arena. However, extensive empirical evidence demonstrates that LLM evaluators suffer from pervasive systematic biases: individual models exhibit marked leniency or harshness, alongside confounding position, length, and self-preference tendencies. Because competitive frontier models on modern leaderboards frequently differ by less than 0.5 score points, the idiosyncratic offsets of a single judge can easily invert relative model standings.
This vulnerability exposes a fundamental statistical tension in benchmark design. Systematic judge bias represents an additive shift rather than zero-mean random sampling noise; consequently, it cannot be averaged away simply by gathering more evaluation scenarios or stochastic generations. Mitigating judge bias fundamentally requires aggregating judgments across a diverse multi-judge panel. However, deploying an entire panel across every evaluated item scales compute costs linearly with panel size while severely constraining generation diversity under any fixed operational budget.
To resolve this bottleneck, the authors ground benchmark error analysis in classical generalizability theory from psychometrics, decomposing the variance of benchmark scores into scenario, generation, judge offset, and residual noise components. Core Idea: employ a round-robin assignment (CyclicJudge) that alternates judges from a panel across model generations, exactly canceling average panel-relative additive offsets at single-judge query cost while maximizing generation diversity.
Method¶
Overall Architecture¶
To formalize the trade-off between evaluation budget and benchmark reliability, CyclicJudge formulates a mixed-effects variance model for a benchmark consisting of \(n\) scenarios sampled from an evaluation pool of \(N\) items, with \(m\) stochastic generations per scenario evaluated by judges drawn from a total pool of \(K_{\mathrm{tot}}\). Given a per-scenario budget of \(B\) judge calls, the evaluation design challenge centers on how to allocate calls between generation diversity and judge panel coverage.
The pipeline of CyclicJudge operates with minimal computational overhead: rather than deploying all judges to identical responses or sampling judges uniformly at random, it globally shuffles the scenario collection and streams judges in continuous round-robin rotation across consecutive generations. Over complete cycles, additive judge offsets cancel out algebraically in the grand mean.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Evaluation Scenarios & Pool<br/>n scenarios ร panel of K_tot judges"] --> B["Variance Decomposition Modeling<br/>isolate scenario, generation, judge bias, and residual"]
B --> C["Continuous Round-Robin Allocation<br/>global scenario shuffling + cross-item cyclic pointer"]
C --> D["Offset Cancellation & Diversity Maximization<br/>exact algebraic cancellation over complete cycles"]
D --> E["Robust Benchmark Score & Leaderboard<br/>single-judge cost / minimal score variance"]
Key Designs¶
1. Generalizability Theory Variance Decomposition: Decoupling Systematic Offsets from Random Noise
To separate structural judge distortions from stochastic variability, the score \(X_{ij\ell}\) given by judge \(\ell\) to generation \(j\) of scenario \(i\) for evaluated model \(\theta\) is parameterized as a linear mixed-effects model: $\(X_{ij\ell} = \mu_{\theta} + \alpha_i + \beta_{ij} + \gamma_\ell + \epsilon_{ij\ell}\)$ where \(\mu_{\theta}\) is the panel-mean estimand; \(\alpha_i \sim (0, \sigma^2_\alpha)\) captures scenario difficulty variation; \(\beta_{ij} \sim (0, \sigma^2_\beta)\) represents generation variability from stochastic decoding (which vanishes for deterministic systems); \(\gamma_\ell\) denotes the fixed additive offset of judge \(\ell\) relative to the panel mean, subject to the centering constraint \(\sum_{\ell=1}^{K_{\mathrm{tot}}} \gamma_\ell = 0\) with population variance \(\sigma^2_\gamma = K_{\mathrm{tot}}^{-1} \sum \gamma_\ell^2\); and \(\epsilon_{ij\ell} \sim (0, \sigma^2_\epsilon)\) is the residual error. By the law of total variance and the finite population correction (FPC) for sampling \(K\) judges from \(K_{\mathrm{tot}}\), the variance of the benchmark grand mean \(\bar{X}\) decomposes as: $\(\mathrm{Var}(\bar{X}) = \underbrace{\frac{\sigma^2_\alpha}{n} + \frac{\sigma^2_\beta}{nm} + \frac{\sigma^2_\epsilon}{nmK}}_{\text{random noise}} + \underbrace{\frac{\sigma^2_\gamma}{K} \cdot \frac{K_{\mathrm{tot}} - K}{K_{\mathrm{tot}} - 1}}_{V_\gamma: \text{ judge bias}}\)$ This formulation formally proves that while random noise terms diminish with additional scenarios \(n\) or repeated generations \(m\), the bias term \(V_\gamma\) depends strictly on panel coverage \(K\), vanishing if and only if all judges in the reference panel are represented.
2. Round-Robin Allocation (CyclicJudge): Exact Offset Cancellation with Maximum Diversity
Under a fixed budget of \(B\) judge queries per scenario, the paper rigorously evaluates three allocation strategies: Strategy A (deploying all \(K_{\mathrm{tot}}\) judges to each generation, forcing \(m = B/K_{\mathrm{tot}}\)); Strategy B (sampling one judge uniformly at random per generation, with \(m = B\)); and Strategy C / CyclicJudge (allocating judge \(j \bmod K_{\mathrm{tot}}\) to generation \(j\), with \(m = B\)). Both Strategy A and CyclicJudge eliminate panel-relative bias (\(\bar{\gamma} = 0\)). However, Strategy B incurs an additional sampling variance of \(\sigma^2_\gamma / (nB)\). Comparing the remaining variance between Strategy A and CyclicJudge: $\(V_A(B) - V_C(B) = \frac{(K_{\mathrm{tot}} - 1)\sigma^2_\beta}{nB} \ge 0\)$ CyclicJudge strictly outperforms Strategy A by exploiting all \(B\) generations to suppress generation noise \(\sigma^2_\beta\). Furthermore, while Strategy A outperforms Strategy B only when \(\sigma^2_\gamma / \sigma^2_\beta > K_{\mathrm{tot}} - 1\), CyclicJudge achieves the mathematical lower bound everywhere: \(V_C(B) \le \min(V_A(B), V_B(B))\).
3. Cross-Scenario Continuous Cycling and Global Shuffling: Remainder Handling
When scenario budget \(B\) is not an exact multiple of panel size \(K_{\mathrm{tot}}\) (\(B = qK_{\mathrm{tot}} + r\) with \(0 \le r < K_{\mathrm{tot}}\)), each scenario leaves \(r\) uncancelled remainder calls. If cycles reset per scenario, these remainders accumulate systemic directional bias. CyclicJudge eliminates this by shuffling all \(n\) scenarios globally (decoupling dataset thematic order from judge assignment) and advancing a single continuous round-robin cycle across scenario boundaries. Across \(nB = QK_{\mathrm{tot}} + s\) total benchmark calls, at most \(s < K_{\mathrm{tot}}\) remainder calls remain uncancelled globally. This residual adds only \(\mathcal{O}(1 / (nB)^2)\) to the mean squared error, which is negligible relative to the \(\sigma^2_\epsilon / (nB)\) leading noise term. Even at the minimal budget \(B=1\) where each scenario receives a single generation scored by a single judge, continuous cycling across scenarios fully cancels judge offsets whenever \(K_{\mathrm{tot}} \mid n\).
Key Experimental Results¶
Main Results¶
Experiments are conducted on MT-Bench (80 general-domain two-turn questions) and MindEval (50 multi-turn clinical mental health support conversations, 10 turns each, 1โ5 scale). The study uses a fully crossed, symmetric design across five models: Qwen 2.5 7B Instruct, Llama 3.3 70B Instruct, GPT-5.2, Gemini 3 Flash, and Claude Sonnet 4.6 (each evaluated both as a subject and as a panel judge with \(K_{\mathrm{tot}} = 5\)). Two-way crossed ANOVA confirms that the judge main effect is highly statistically significant across both benchmarks (\(p < 0.001\)).
Table 1 presents the rank correlation and top-1 leaderboard flip frequencies across 5,000 bootstrap subsampling iterations at an operational budget of \(B=5\) judge calls per scenario.
| Allocation Strategy | Spearman \(\rho\) | Kendall \(\tau\)-b | Top-1 Flip Rate | Relative Flip Reduction vs. Single |
|---|---|---|---|---|
| Single judge | 0.716 | 0.661 | 51.3% | Baseline (0.0%) |
| Random 1 judge | 0.955 | 0.916 | 32.8% | -36.1% |
| All 5 judges | 0.969 | 0.938 | 27.2% | -47.0% |
| CyclicJudge (round-robin) | 0.980 | 0.960 | 20.0% | -61.0% |
Ablation Study¶
Table 2 breaks down the crossed ANOVA per-observation variance components across evaluated models on both benchmarks, providing the empirical foundation for strategy performance:
| Benchmark | Model | Scenario Var \(\hat{\sigma}^2_\alpha\) | Generation Var \(\hat{\sigma}^2_\beta\) | Judge Var \(\hat{\sigma}^2_\gamma\) | Residual Var \(\hat{\sigma}^2_\epsilon\) |
|---|---|---|---|---|---|
| MT-Bench | Qwen 2.5 7B | 1.530 | 0.266 | 0.947 | 1.486 |
| MT-Bench | Llama 3.3 70B | 0.882 | 0.238 | 0.503 | 1.130 |
| MT-Bench | GPT-5.2 | 0.634 | 0.076 | 0.339 | 0.564 |
| MT-Bench | Gemini 3 Flash | 0.408 | 0.000 | 0.435 | 0.661 |
| MT-Bench | Claude Sonnet 4.6 | 0.393 | 0.000 | 0.427 | 0.830 |
| MindEval | Qwen 2.5 7B | 0.034 | 0.100 | 0.523 | 0.159 |
| MindEval | Llama 3.3 70B | 0.014 | 0.019 | 0.280 | 0.069 |
| MindEval | GPT-5.2 | 0.021 | 0.047 | 0.132 | 0.207 |
| MindEval | Gemini 3 Flash | 0.015 | 0.016 | 0.150 | 0.103 |
| MindEval | Claude Sonnet 4.6 | 0.005 | 0.002 | 0.155 | 0.061 |
Key Findings¶
- Judge Bias Dominates Baseline Uncertainty: At the default operating point (\(m=1, K=1\)), judge offset variance \(\hat{\sigma}^2_\gamma\) accounts for over 94% of total benchmark variance across models. Single-judge evaluations frequently invert rankings: on MT-Bench, three distinct models claim top-1 depending on the chosen judge, and Qwen assigns itself a tie for first place due to self-preference bias.
- Generation Variance Shrinks with Capability: Advanced frontier models (Gemini 3 Flash, Claude Sonnet 4.6) exhibit negligible generation variance (\(\hat{\sigma}^2_\beta \approx 0\)), reflecting post-training stabilization. On MindEval, multi-turn clinical dialogue averages out token-level decoding randomness over 10 turns, compressing \(\hat{\sigma}^2_\beta\) by an order of magnitude and elevating judge bias to the overwhelming source of error.
- Empirical Regimes Match Theoretical Thresholds: The empirical data precisely validates the condition \(\sigma^2_\gamma / \sigma^2_\beta \gtrless K_{\mathrm{tot}} - 1\). On MT-Bench, where generation variance is relatively large, random judging beats all-judges for Qwen and Llama. On MindEval, where generation noise is suppressed, all-judges consistently outperforms random. In both regimes, CyclicJudge delivers the lowest variance everywhere, yielding a 27%โ40% variance reduction over random assignment at \(B=5\).
Highlights & Insights¶
- Algorithmic Cost Neutrality: Rather than multiplying compute expenditure by evaluating full panels on every instance, CyclicJudge demonstrates that simple round-robin permutation provides a mathematically proven "free lunch," eliminating additive panel offsets at zero marginal API cost.
- Optimal Budget Allocation Rule: Analysis of generation budget \(B_{\mathrm{gen}} = nm\) proves that total score variance is strictly minimized when setting \(m=1\) and maximizing scenario count \(n\). Practitioners should prioritize diverse prompt coverage over repeated generations, paired with round-robin judge rotation.
- Drop-in Infrastructure Compatibility: Because it operates purely at the level of task dispatch, CyclicJudge requires no model-specific retraining, prompt alterations, or Bayesian inference machinery, making it trivially deployable in automated CI/CD and leaderboard pipelines.
Limitations & Future Work¶
- Heterogeneous Judge Pricing: The theoretical framework assumes uniform latency and pricing across judges. When judge costs vary significantly (e.g., frontier API models vs. compact distilled evaluators), balanced cycling is no longer cost-optimal, necessitating weighted combinatorial designs.
- Residual Absorption of Interactions: The linear model incorporates only additive main effects, folding scenario-judge and generation-judge interactions into residual noise \(\epsilon_{ij\ell}\). CyclicJudge cancels average offsets, but interaction variance decays only at the standard \(\mathcal{O}(1/B)\) noise rate.
- Shared Panel Biases: CyclicJudge normalizes relative deviations within the chosen panel. Shared biases across all evaluators (e.g., universal verbosity or shared alignment blind spots) are absorbed into the panel-mean estimand \(\mu_{\theta}\) and require calibration against human gold standards.
Related Work & Insights¶
- vs. Multi-Judge Panels & Juries (Verga et al., 2024): Traditional panel approaches duplicate all judge queries across all outputs, scaling compute by \(K_{\mathrm{tot}}\). CyclicJudge achieves identical offset cancellation at \(1/K_{\mathrm{tot}}\) of the compute budget by spreading judges across the evaluation trajectory.
- vs. Hierarchical Bayesian Modeling (HiBayES, Luettgau et al., 2025): While Bayesian partial pooling requires iterative sampling and prior specifications, CyclicJudge provides closed-form, model-free variance minimization directly through experimental design.
Rating¶
- Novelty: โญโญโญโญโญ Elegant synthesis of generalizability theory and round-robin scheduling, establishing closed-form optimality for LLM judge allocation.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous symmetric crossed evaluation across five major frontier models with 5,000 bootstrap resamples on two distinct benchmarks.
- Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formulations, disciplined structural hierarchy, and seamless empirical validation.
- Value: โญโญโญโญโญ Offers immediate, cost-neutral practical utility for every organization running automated LLM benchmarks and leaderboards.