Skip to content

CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Evaluation
Keywords: LLM Evaluation, Judge Bias, Variance Decomposition, Round-Robin Scheduling, Generalizability Theory

TL;DR

Addressing systemic scoring biases in LLM-as-a-judge evaluation that cannot be averaged away by increasing test scenarios, this work establishes a generalizability theory variance decomposition and demonstrates that a round-robin judge assignment strategy (CyclicJudge) cancels panel-relative judge offsets and maximizes generation diversity at zero additional cost, cutting top-1 ranking flips by 61%.

Background & Motivation

Large language models serving as evaluators (LLM-as-a-judge) have emerged as the standard paradigm for open-ended model benchmarking, anchoring widely used suites like MT-Bench and Chatbot Arena. However, extensive empirical evidence demonstrates that LLM evaluators suffer from pervasive systematic biases: individual models exhibit marked leniency or harshness, alongside confounding position, length, and self-preference tendencies. Because competitive frontier models on modern leaderboards frequently differ by less than 0.5 score points, the idiosyncratic offsets of a single judge can easily invert relative model standings.

This vulnerability exposes a fundamental statistical tension in benchmark design. Systematic judge bias represents an additive shift rather than zero-mean random sampling noise; consequently, it cannot be averaged away simply by gathering more evaluation scenarios or stochastic generations. Mitigating judge bias fundamentally requires aggregating judgments across a diverse multi-judge panel. However, deploying an entire panel across every evaluated item scales compute costs linearly with panel size while severely constraining generation diversity under any fixed operational budget.

To resolve this bottleneck, the authors ground benchmark error analysis in classical generalizability theory from psychometrics, decomposing the variance of benchmark scores into scenario, generation, judge offset, and residual noise components. Core Idea: employ a round-robin assignment (CyclicJudge) that alternates judges from a panel across model generations, exactly canceling average panel-relative additive offsets at single-judge query cost while maximizing generation diversity.

Method

Overall Architecture

To formalize the trade-off between evaluation budget and benchmark reliability, CyclicJudge formulates a mixed-effects variance model for a benchmark consisting of \(n\) scenarios sampled from an evaluation pool of \(N\) items, with \(m\) stochastic generations per scenario evaluated by judges drawn from a total pool of \(K_{\mathrm{tot}}\). Given a per-scenario budget of \(B\) judge calls, the evaluation design challenge centers on how to allocate calls between generation diversity and judge panel coverage.

The pipeline of CyclicJudge operates with minimal computational overhead: rather than deploying all judges to identical responses or sampling judges uniformly at random, it globally shuffles the scenario collection and streams judges in continuous round-robin rotation across consecutive generations. Over complete cycles, additive judge offsets cancel out algebraically in the grand mean.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Evaluation Scenarios & Pool<br/>n scenarios ร— panel of K_tot judges"] --> B["Variance Decomposition Modeling<br/>isolate scenario, generation, judge bias, and residual"]
    B --> C["Continuous Round-Robin Allocation<br/>global scenario shuffling + cross-item cyclic pointer"]
    C --> D["Offset Cancellation & Diversity Maximization<br/>exact algebraic cancellation over complete cycles"]
    D --> E["Robust Benchmark Score & Leaderboard<br/>single-judge cost / minimal score variance"]

Key Designs

1. Generalizability Theory Variance Decomposition: Decoupling Systematic Offsets from Random Noise

To separate structural judge distortions from stochastic variability, the score \(X_{ij\ell}\) given by judge \(\ell\) to generation \(j\) of scenario \(i\) for evaluated model \(\theta\) is parameterized as a linear mixed-effects model: $\(X_{ij\ell} = \mu_{\theta} + \alpha_i + \beta_{ij} + \gamma_\ell + \epsilon_{ij\ell}\)$ where \(\mu_{\theta}\) is the panel-mean estimand; \(\alpha_i \sim (0, \sigma^2_\alpha)\) captures scenario difficulty variation; \(\beta_{ij} \sim (0, \sigma^2_\beta)\) represents generation variability from stochastic decoding (which vanishes for deterministic systems); \(\gamma_\ell\) denotes the fixed additive offset of judge \(\ell\) relative to the panel mean, subject to the centering constraint \(\sum_{\ell=1}^{K_{\mathrm{tot}}} \gamma_\ell = 0\) with population variance \(\sigma^2_\gamma = K_{\mathrm{tot}}^{-1} \sum \gamma_\ell^2\); and \(\epsilon_{ij\ell} \sim (0, \sigma^2_\epsilon)\) is the residual error. By the law of total variance and the finite population correction (FPC) for sampling \(K\) judges from \(K_{\mathrm{tot}}\), the variance of the benchmark grand mean \(\bar{X}\) decomposes as: $\(\mathrm{Var}(\bar{X}) = \underbrace{\frac{\sigma^2_\alpha}{n} + \frac{\sigma^2_\beta}{nm} + \frac{\sigma^2_\epsilon}{nmK}}_{\text{random noise}} + \underbrace{\frac{\sigma^2_\gamma}{K} \cdot \frac{K_{\mathrm{tot}} - K}{K_{\mathrm{tot}} - 1}}_{V_\gamma: \text{ judge bias}}\)$ This formulation formally proves that while random noise terms diminish with additional scenarios \(n\) or repeated generations \(m\), the bias term \(V_\gamma\) depends strictly on panel coverage \(K\), vanishing if and only if all judges in the reference panel are represented.

2. Round-Robin Allocation (CyclicJudge): Exact Offset Cancellation with Maximum Diversity

Under a fixed budget of \(B\) judge queries per scenario, the paper rigorously evaluates three allocation strategies: Strategy A (deploying all \(K_{\mathrm{tot}}\) judges to each generation, forcing \(m = B/K_{\mathrm{tot}}\)); Strategy B (sampling one judge uniformly at random per generation, with \(m = B\)); and Strategy C / CyclicJudge (allocating judge \(j \bmod K_{\mathrm{tot}}\) to generation \(j\), with \(m = B\)). Both Strategy A and CyclicJudge eliminate panel-relative bias (\(\bar{\gamma} = 0\)). However, Strategy B incurs an additional sampling variance of \(\sigma^2_\gamma / (nB)\). Comparing the remaining variance between Strategy A and CyclicJudge: $\(V_A(B) - V_C(B) = \frac{(K_{\mathrm{tot}} - 1)\sigma^2_\beta}{nB} \ge 0\)$ CyclicJudge strictly outperforms Strategy A by exploiting all \(B\) generations to suppress generation noise \(\sigma^2_\beta\). Furthermore, while Strategy A outperforms Strategy B only when \(\sigma^2_\gamma / \sigma^2_\beta > K_{\mathrm{tot}} - 1\), CyclicJudge achieves the mathematical lower bound everywhere: \(V_C(B) \le \min(V_A(B), V_B(B))\).

3. Cross-Scenario Continuous Cycling and Global Shuffling: Remainder Handling

When scenario budget \(B\) is not an exact multiple of panel size \(K_{\mathrm{tot}}\) (\(B = qK_{\mathrm{tot}} + r\) with \(0 \le r < K_{\mathrm{tot}}\)), each scenario leaves \(r\) uncancelled remainder calls. If cycles reset per scenario, these remainders accumulate systemic directional bias. CyclicJudge eliminates this by shuffling all \(n\) scenarios globally (decoupling dataset thematic order from judge assignment) and advancing a single continuous round-robin cycle across scenario boundaries. Across \(nB = QK_{\mathrm{tot}} + s\) total benchmark calls, at most \(s < K_{\mathrm{tot}}\) remainder calls remain uncancelled globally. This residual adds only \(\mathcal{O}(1 / (nB)^2)\) to the mean squared error, which is negligible relative to the \(\sigma^2_\epsilon / (nB)\) leading noise term. Even at the minimal budget \(B=1\) where each scenario receives a single generation scored by a single judge, continuous cycling across scenarios fully cancels judge offsets whenever \(K_{\mathrm{tot}} \mid n\).

Key Experimental Results

Main Results

Experiments are conducted on MT-Bench (80 general-domain two-turn questions) and MindEval (50 multi-turn clinical mental health support conversations, 10 turns each, 1โ€“5 scale). The study uses a fully crossed, symmetric design across five models: Qwen 2.5 7B Instruct, Llama 3.3 70B Instruct, GPT-5.2, Gemini 3 Flash, and Claude Sonnet 4.6 (each evaluated both as a subject and as a panel judge with \(K_{\mathrm{tot}} = 5\)). Two-way crossed ANOVA confirms that the judge main effect is highly statistically significant across both benchmarks (\(p < 0.001\)).

Table 1 presents the rank correlation and top-1 leaderboard flip frequencies across 5,000 bootstrap subsampling iterations at an operational budget of \(B=5\) judge calls per scenario.

Allocation Strategy Spearman \(\rho\) Kendall \(\tau\)-b Top-1 Flip Rate Relative Flip Reduction vs. Single
Single judge 0.716 0.661 51.3% Baseline (0.0%)
Random 1 judge 0.955 0.916 32.8% -36.1%
All 5 judges 0.969 0.938 27.2% -47.0%
CyclicJudge (round-robin) 0.980 0.960 20.0% -61.0%

Ablation Study

Table 2 breaks down the crossed ANOVA per-observation variance components across evaluated models on both benchmarks, providing the empirical foundation for strategy performance:

Benchmark Model Scenario Var \(\hat{\sigma}^2_\alpha\) Generation Var \(\hat{\sigma}^2_\beta\) Judge Var \(\hat{\sigma}^2_\gamma\) Residual Var \(\hat{\sigma}^2_\epsilon\)
MT-Bench Qwen 2.5 7B 1.530 0.266 0.947 1.486
MT-Bench Llama 3.3 70B 0.882 0.238 0.503 1.130
MT-Bench GPT-5.2 0.634 0.076 0.339 0.564
MT-Bench Gemini 3 Flash 0.408 0.000 0.435 0.661
MT-Bench Claude Sonnet 4.6 0.393 0.000 0.427 0.830
MindEval Qwen 2.5 7B 0.034 0.100 0.523 0.159
MindEval Llama 3.3 70B 0.014 0.019 0.280 0.069
MindEval GPT-5.2 0.021 0.047 0.132 0.207
MindEval Gemini 3 Flash 0.015 0.016 0.150 0.103
MindEval Claude Sonnet 4.6 0.005 0.002 0.155 0.061

Key Findings

  • Judge Bias Dominates Baseline Uncertainty: At the default operating point (\(m=1, K=1\)), judge offset variance \(\hat{\sigma}^2_\gamma\) accounts for over 94% of total benchmark variance across models. Single-judge evaluations frequently invert rankings: on MT-Bench, three distinct models claim top-1 depending on the chosen judge, and Qwen assigns itself a tie for first place due to self-preference bias.
  • Generation Variance Shrinks with Capability: Advanced frontier models (Gemini 3 Flash, Claude Sonnet 4.6) exhibit negligible generation variance (\(\hat{\sigma}^2_\beta \approx 0\)), reflecting post-training stabilization. On MindEval, multi-turn clinical dialogue averages out token-level decoding randomness over 10 turns, compressing \(\hat{\sigma}^2_\beta\) by an order of magnitude and elevating judge bias to the overwhelming source of error.
  • Empirical Regimes Match Theoretical Thresholds: The empirical data precisely validates the condition \(\sigma^2_\gamma / \sigma^2_\beta \gtrless K_{\mathrm{tot}} - 1\). On MT-Bench, where generation variance is relatively large, random judging beats all-judges for Qwen and Llama. On MindEval, where generation noise is suppressed, all-judges consistently outperforms random. In both regimes, CyclicJudge delivers the lowest variance everywhere, yielding a 27%โ€“40% variance reduction over random assignment at \(B=5\).

Highlights & Insights

  • Algorithmic Cost Neutrality: Rather than multiplying compute expenditure by evaluating full panels on every instance, CyclicJudge demonstrates that simple round-robin permutation provides a mathematically proven "free lunch," eliminating additive panel offsets at zero marginal API cost.
  • Optimal Budget Allocation Rule: Analysis of generation budget \(B_{\mathrm{gen}} = nm\) proves that total score variance is strictly minimized when setting \(m=1\) and maximizing scenario count \(n\). Practitioners should prioritize diverse prompt coverage over repeated generations, paired with round-robin judge rotation.
  • Drop-in Infrastructure Compatibility: Because it operates purely at the level of task dispatch, CyclicJudge requires no model-specific retraining, prompt alterations, or Bayesian inference machinery, making it trivially deployable in automated CI/CD and leaderboard pipelines.

Limitations & Future Work

  • Heterogeneous Judge Pricing: The theoretical framework assumes uniform latency and pricing across judges. When judge costs vary significantly (e.g., frontier API models vs. compact distilled evaluators), balanced cycling is no longer cost-optimal, necessitating weighted combinatorial designs.
  • Residual Absorption of Interactions: The linear model incorporates only additive main effects, folding scenario-judge and generation-judge interactions into residual noise \(\epsilon_{ij\ell}\). CyclicJudge cancels average offsets, but interaction variance decays only at the standard \(\mathcal{O}(1/B)\) noise rate.
  • Shared Panel Biases: CyclicJudge normalizes relative deviations within the chosen panel. Shared biases across all evaluators (e.g., universal verbosity or shared alignment blind spots) are absorbed into the panel-mean estimand \(\mu_{\theta}\) and require calibration against human gold standards.
  • vs. Multi-Judge Panels & Juries (Verga et al., 2024): Traditional panel approaches duplicate all judge queries across all outputs, scaling compute by \(K_{\mathrm{tot}}\). CyclicJudge achieves identical offset cancellation at \(1/K_{\mathrm{tot}}\) of the compute budget by spreading judges across the evaluation trajectory.
  • vs. Hierarchical Bayesian Modeling (HiBayES, Luettgau et al., 2025): While Bayesian partial pooling requires iterative sampling and prior specifications, CyclicJudge provides closed-form, model-free variance minimization directly through experimental design.

Rating

  • Novelty: โญโญโญโญโญ Elegant synthesis of generalizability theory and round-robin scheduling, establishing closed-form optimality for LLM judge allocation.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous symmetric crossed evaluation across five major frontier models with 5,000 bootstrap resamples on two distinct benchmarks.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear mathematical formulations, disciplined structural hierarchy, and seamless empirical validation.
  • Value: โญโญโญโญโญ Offers immediate, cost-neutral practical utility for every organization running automated LLM benchmarks and leaderboards.