Judge Reliability Harness: Stress Testing the Reliability of LLM Judges¶

Conference: ICLR 2026 arXiv: 2603.05399 Code: https://github.com/RANDCorporation/judge-reliability-harness Area: LLM Agent Keywords: LLM-as-judge, reliability testing, perturbation robustness, agentic evaluation, benchmark

TL;DR¶

This paper proposes Judge Reliability Harness (JRH), an open-source framework that systematically evaluates the reliability of LLM judges through synthetic tests including label flip, format invariance, semantic paraphrase, verbosity bias, and stochastic stability. The framework stress-tests four state-of-the-art judges across four benchmarks (FORTRESS, HarmBench, Persuade, AgentHarm), finding that no single judge is reliable across all scenarios.

Background & Motivation¶

Background: LLMs are widely used as autograders to score, rank, or classify AI outputs, serving as a cost-effective alternative to human evaluation. Works such as MT-Bench and Chatbot Arena have demonstrated that GPT-4-level judges can approach expert-level agreement.

Limitations of Prior Work: The reliability of LLM judges is rarely systematically evaluated or reported. Point estimates of agreement with human annotations on small validation sets do not guarantee robustness to input variations in format, wording, or length.

Key Challenge: Judges play a central role in the evaluation ecosystem, yet standardized reliability testing tools are lacking. Prior research has identified issues such as position bias and verbosity bias in LLM judges, but no practical, reproducible testing framework exists.

Goal: To construct a general, configurable validation suite that enables any LLM judge to undergo systematic reliability stress testing prior to deployment.

Key Insight: A synthetic data generation pipeline combined with optional human review to automatically produce multi-dimensional test cases.

Core Idea: A standardized testing framework driven by synthetic perturbations that systematically exposes reliability weaknesses of LLM judges across multiple dimensions.

Method¶

Overall Architecture¶

The JRH workflow proceeds as follows: (1) load seed datasets and normalize them to a unified schema; (2) run a synthetic data pipeline to generate perturbed test samples; (3) optionally conduct human-in-the-loop (HITL) review—accepting, editing, or rejecting generated test cases; (4) evaluate perturbed samples using the target judge; (5) aggregate reliability metrics into a report.

Key Designs¶

Base Perturbation Test Suite:
- Function: Tests judge responses to semantically preserving versus semantically inverting changes.
- Mechanism: Comprises two categories—discriminative tests (label flip: responses are rewritten to explicitly violate scoring criteria, such that the judge should reverse its judgment) and consistency tests (format invariance: only layout elements such as blank lines, indentation, and whitespace are altered; semantic paraphrase: wording is rewritten while preserving semantics; verbosity bias: content is expanded or compressed while preserving substance).
- Design Motivation: Discriminative tests verify whether a judge can distinguish quality differences, while consistency tests verify whether a judge remains stable under changes that do not affect quality.
Stochastic Stability Test:
- Function: Tests scoring consistency of a judge given identical inputs.
- Mechanism: Multiple copies of the same sample are created and submitted to the judge independently; the resulting scores are compared for consistency.
- Design Motivation: Stochastic sampling in LLMs may yield different scores for identical inputs, and such instability undermines the reproducibility of evaluations.
Synthetic Ordinal Test:
- Function: Generates synthetic samples covering each score level for multi-level scoring benchmarks.
- Mechanism: A score-bucket manager tracks already-generated levels; a temperature-increasing strategy combined with few-shot examples guides generation toward specific score levels; a validation LLM confirms that the target score has been achieved.
- Design Motivation: Tests the calibration ability of judges in ordinal scoring.
Agent Mode:
- Function: Perturbation testing targeting multi-turn agent conversation transcripts.
- Mechanism: agent_perturbation modifies transcripts to introduce violations; agent_positives modifies transcripts to satisfy criteria. The pipeline employs a multi-step editing chain: planning LLM → editing LLM → summarization LLM → validation LLM.
- Design Motivation: Agent evaluation is fundamentally different from single-turn text evaluation, requiring understanding of cumulative effects across multi-turn context.
Human-in-the-Loop (HITL) Review:
- Function: Ensures the quality of synthetic test data.
- Mechanism: A UI interface allows annotators to review, edit, or reject generated perturbation samples on a case-by-case basis.
- Design Motivation: Automatic generation may produce unreasonable perturbations—particularly for safety-sensitive content that triggers model safety guardrails—necessitating human quality control.

Key Experimental Results¶

Main Results¶

Benchmark	Most Reliable Judge	Least Reliable Judge	Key Finding
FORTRESS	Llama 4.1 Maverick	All models perform relatively well	Binary classification tasks yield high overall reliability
HarmBench	GPT-4o	Gemini 2.5 Pro (std=17.17%)	Claude achieves lowest std (11.13%)
Persuade	Gemini 2.5 Pro (std=11.10%)	Claude Sonnet 4.5 (std=17.18%)	Multi-level scoring significantly reduces reliability
AgentHarm	GPT-4o/Llama (0.906)	Gemini 2.5 Pro (75% positives)	Opus 4.5 achieves only 68.75% on perturbation

Ablation Study¶

Perturbation Type	General Performance	Notes
Semantic paraphrase	Highest robustness (minimum 40%)	Judges are relatively stable under semantic-level perturbations
Format invariance	Lowest reliability	Formatting changes have a greater impact than semantic changes
Label flip	Moderate	Discrimination accuracy varies by model and task
Verbosity bias	Moderate	Biases from longer/shorter versions exist but are not extreme
Stochastic stability	Model-dependent	Instability attributable to temperature sampling

Key Findings¶

No single judge is reliable across all benchmarks: An inverse relationship in variability is observed between Persuade and HarmBench—Claude is least stable on Persuade but most stable on HarmBench, while Gemini shows the opposite pattern.
Format perturbations > semantic perturbations: LLM judges are more sensitive to purely formatting changes (blank lines, indentation) than to semantic paraphrases, which is concerning given that different LLMs inherently produce outputs in different formats.
Binary vs. multi-level scoring: Reliability of all judges on Persuade (scored 1–6) is substantially lower than on binary classification tasks.
Asymmetric failure modes in agent evaluation: Certain judges exhibit high false-negative rates (missed violations), while others over-flag (high false-positive rates).
Llama 4.1 Maverick 17B offers the best cost-effectiveness: It matches top-tier judges on most benchmarks at substantially lower cost.

Highlights & Insights¶

Generalizability of the framework design: JRH can interface with any LLM judge and any benchmark dataset to produce standardized reliability reports. Its role as a meta-evaluation tool for "testing the judge" is highly valuable.
HITL is indispensable in agent mode: 14 out of 16 transcripts in agent_perturbation required human modification, indicating that current generative models are constrained by safety guardrails when editing harmful content, making full automation infeasible.
Implications of format sensitivity: If judges are unstable under formatting changes, cross-LLM rankings—where different models have different formatting conventions—may be driven by formatting differences rather than substantive capability differences.

Limitations & Future Work¶

Small sample sizes: Only 10–16 seed samples are used per benchmark, limiting statistical power.
Ecological validity of synthetic perturbations: Whether automatically generated perturbations faithfully reflect the variations encountered in production requires further validation.
Non-standardized judge prompts: Different judges are paired with different prompt templates, which itself introduces an additional confounding variable.
Open-source small models not tested: Only four large or medium-sized models are evaluated; coverage of a broader set of open-source evaluation models is absent.

vs. FBI benchmark (Doddapaneni et al. 2024): FBI uses targeted perturbation detection to assess whether LLMs can identify quality degradation; JRH is more general and configurable.
vs. CALM (Ye et al.): CALM quantifies the effects of position bias and verbosity bias; JRH standardizes such tests into a reusable framework.
vs. MT-Bench/Chatbot Arena: These works are "consumers" of judges; JRH serves as a quality assurance tool for the judges themselves.

Rating¶

Novelty: ⭐⭐⭐ The idea is intuitive, but systematic execution delivers substantial engineering value as a meta-evaluation framework.
Experimental Thoroughness: ⭐⭐⭐ Benchmark coverage is reasonable, though sample sizes are limited.
Writing Quality: ⭐⭐⭐⭐ The structure is complete and the methodology is described clearly.
Value: ⭐⭐⭐⭐ Highly valuable to the LLM evaluation community as a practical tool.