Skip to content

AR-Bench: Benchmarking Legal Reasoning with Judgment Prediction and Precedent Analysis

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Yifei6672/AR-Bench
Area: LLM Reasoning
Keywords: Legal Reasoning, Appellate Review, Error Detection, Statutory Penalties, Counterfactual Benchmark

TL;DR

This paper shifts legal AI from forward prediction to retrospective diagnostic reasoning (APPELLATE REVIEW), proposing a controlled benchmark of 8,700 criminal judgments across charge, prison term, and fine errors (AR-BENCH) and revealing that 16 state-of-the-art LLMs suffer from severe deficiencies in unguided anomaly detection and rule-based penalty calculations.

Background & Motivation

Equality before the law serves as an indispensable cornerstone of modern judicial fairness, mandating that criminal adjudication strictly adheres to proportional and balanced sentencing. However, even with elaborate and formalized court procedures, trial judges inevitably commit errors due to intricate real-world circumstances and the inherently abstract nature of statutory norms. To mitigate these risks, both civil and common law systems have instituted appellate review and post-conviction relief mechanisms to reexamine the accuracy of statutory applications and correct flawed judgments. In practice, however, these appellate mechanisms face tremendous pressure; for instance, judicial statistics in China demonstrate a 65.8% surge in appeal cases between 2015 and 2024, placing an overwhelming burden on judicial personnel and threatening the depth, consistency, and timeliness of manual review.

Against this pressing backdrop, current legal AI research remains predominantly confined to forward-looking, pretrial tasks—primarily legal judgment prediction (LJP) and automated document drafting. These conventional paradigms aim to infer final outcomes from raw fact descriptions or draft arguments from predefined outlines, formalizing legal reasoning as standard classification or open-ended text generation. However, appellate review is fundamentally a retrospective diagnostic problem: rather than predicting an unknown outcome, it requires an auditor to meticulously inspect existing judgment documents, detect subtle discrepancies between facts and statutory requirements, isolate the root cause among cascading errors, and derive a legally validated correction. Prior attempts to model appellate review in civil or administrative domains suffer from high evaluation subjectivity, as civil appeals heavily rely on discretionary proportional liability and contextual interpretation, precluding a verifiable ground truth for automated evaluation.

Criminal law under civil law traditions offers an ideal testbed for diagnostic reasoning due to its rigid adherence to the principle of legality (nullum crimen, nulla poena sine lege). In criminal proceedings, explicit factual elements map deterministically to statutory offenses, which in turn enforce strict intervals for prison terms and statutory ranges or multipliers for criminal fines. Exploiting this formal logical closure, this work introduces the task of APPELLATE REVIEW and constructs AR-BENCH, a controlled, counterfactually perturbed criminal judgment benchmark providing verifiable ground truth across 8,700 single-defendant cases. Core idea: shift legal AI evaluation from forward prediction to retrospective diagnostic auditing, formulating appellate review into an error detection-classification-correction pipeline under strict criminal statutory constraints, and benchmarking 16 LLMs on AR-BENCH to expose their fundamental breakdowns in holistic deductive reasoning and legal arithmetic.

Method

Overall Architecture

APPELLATE REVIEW formalizes judicial review into three decoupled subtasks to avoid error propagation and enable independent diagnostic assessment: 1. Error Detection: Given a finalized judgment document \(D\) encompassing case facts \(D_{\text{fact}}\), reasoning \(D_{\text{reason}}\), and final judgment \(D_{\text{judge}}\), the model must determine whether legal application errors exist without any prior hints, outputting a boolean decision \(\hat{y} \in \{\text{true}, \text{false}\}\); 2. Error Classification: Conditioned on the presence of an error, the model must penetrate cascading surface anomalies and classify the root violation into one of three primary statutory categories: Charge, Prison Term, or Fine; 3. Error Correction: Given the judgment document and the identified primary error category, the model is tasked with generating a localized, legally valid replacement \(\hat{C}\) within a strictly constrained target space (precise charge title or exact continuous penalty value).

To support this protocol, AR-BENCH assembles 8,700 real-world single-defendant criminal judgments, systematically injecting six fine-grained counterfactual error types (E1–E6) through semantic-level precedent swapping rather than superficial token mutations. The overall evaluation workflow is structured as follows:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Raw Criminal Judgment Corpus<br/>Case Facts + Reasoning + Judgment"] --> B["Three-Dimensional Counterfactual Anomaly Injection<br/>Charges·Prison Terms·Fines (E1-E6)"]
    B --> C["Semantic-Level Case Swapping & Expert Verification<br/>Preserve Local Coherence·Fleiss Kappa 0.79"]
    C --> D["Controlled Diagnostic Input Configurations<br/>S1 Baseline to S5 Adversarial Statutes"]
    D --> E["Error Detection<br/>Unguided Global Skepticism & Binary Verdict"]
    E --> F["Error Classification<br/>Root-Cause Isolation Among Cascading Anomalies"]
    F --> G["Error Correction<br/>Discrete & Continuous Distance-Aware Metrics"]
    G --> H["Comprehensive Legal Diagnostic Evaluation"]

Key Designs

1. Three-Dimensional Counterfactual Anomaly Injection: Eliminating Token Artifacts and Establishing Verifiable Ground Truth

To prevent models from exploiting superficial token-level heuristics (such as catching obvious linguistic artifacts or ungrammatical numbers), AR-BENCH establishes a rigorous taxonomy of six fine-grained synthetic error types (E1–E6) across the three pillars of criminal sentencing: - Charges: Encompasses Erroneous Conviction (E1), where confusing crimes sharing common features are misclassified (e.g., swapping contract fraud with basic fraud), and Omission of Charges (E2), where a multi-offense factual scenario is convicted on only a single primary charge while dropping concurring offenses. - Prison Terms: Encompasses Sentencing Outside Limits (E3), where sentences arbitrarily exceed or undercut statutory sentencing intervals without mitigating legal grounds, and Failure to Consider Sentencing Factors (E4), where statutory sentencing modifiers (such as confession, surrender, or recidivism) are ignored during sentence computation. - Fines: Encompasses Fixed-Amount Fine Outside Limits (E5), violating specific statutory monetary caps, and Percentage-Based Fine Outside Limits (E6), where proportional statutory multipliers (e.g., \(1\times\) to \(5\times\) of the illegal proceeds) are miscalculated.

Because each perturbed judgment is synthetically derived from an authentic adjudicated case, the original pre-modification attributes serve as uncontaminated, verifiable ground truth for error correction.

2. Semantic-Level Case Swapping & Expert Verification: Enforcing Deep Deductive Reasoning over Local Text Matching

To preserve the natural flow and professional tone of judicial writing, the benchmark rejects token replacements in favor of semantic-level structural swapping. For charge anomalies (E1 and E2), the pipeline retrieves companion cases from a verified single-charge repository that share identical sentencing factors and criminal circumstances, seamlessly exchanging the reasoning and sentencing text sections while adjusting subject entities. This ensures that the reasoning text appears internally consistent and grammatically pristine, compelling models to perform holistic deductive reasoning across the entire factual premise rather than detecting stylistic seams. Furthermore, a rigorous two-stage expert quality check was conducted by three certified legal experts who passed China's National Uniform Legal Profession Qualification Examination. After refining regex extraction on an initial 400 cases, a post-validation on a separate 400-case split demonstrated a 91.5% acceptance pass rate with a Fleiss' \(\kappa = 0.79\), validating the absence of data leaks and confirming logical soundness.

3. Controlled Diagnostic Input Configurations: Dissecting Contextual Components and Statutory Knowledge Dependencies

To evaluate how different textual components affect LLM auditing capabilities, AR-BENCH formulates five distinct input settings (S1–S5): - S1 (Case Facts + Anomalous Judgment): Mirrors the standard LJP setting with zero judicial reasoning; - S2 (S1 + Normal Reasoning Process): Supplies authentic reasoning paths, mitigating factors, and cited statutory articles; - S3 (S2 + Relevant Law Articles): Augments the prompt with complete text of relevant statutory articles via external retrieval; - S4 (Case Facts + Anomalous Reasoning + Anomalous Judgment): The primary real-world appellate review setup where the reasoning text internally matches the erroneous judgment outcome; - S5 (S4 + Erroneously Cited Law Articles): Supplies the verbatim statutory text of the misapplied articles, testing susceptibility to adversarial knowledge.

This structured breakdown enables precise ablation of whether LLMs reason from intrinsic parameters, leverage reasoning traces, or get distracted by legal knowledge injection.

Benchmark & Evaluation Protocol

AR-BENCH consists of 8,700 single-defendant criminal cases, partitioned across Charges (3,844 cases: 3,110 anomalous, 734 normal), Prison Terms (2,777 cases: 2,319 anomalous, 458 normal), and Fines (2,079 cases: 1,729 anomalous, 350 normal). The inclusion of balanced normal judgments is vital for evaluating false-positive hallucinations. - Detection and Classification: Evaluated using Accuracy (Acc), Macro-Precision (MaP), Macro-Recall (MaR), and Macro-F1 (MaF1) to account for distribution imbalances. - Correction: Discrete charges use classification metrics; continuous prison terms (months) and fines (monetary values) are evaluated both via discretized statutory intervals and distance-aware continuous metrics: ImpScore (Equations 4–6, penalizing log deviations via a 5-tier step function) and [email protected] (Equation 7, measuring predictions within a strict \(\pm 10\%\) tolerance of the ground truth):

\[d_i = |\log(t_i + 1) - \log(\bar{t}_i + 1)|\]
\[f(v) = \begin{cases} 1.0, & v \le 0.2 \\ 0.8, & 0.2 < v \le 0.4 \\ 0.6, & 0.4 < v \le 0.6 \\ 0.4, & 0.6 < v \le 0.8 \\ 0.2, & 0.8 < v \le 1.0 \\ 0.0, & v > 1.0 \end{cases}, \quad \text{ImpScore} = \frac{1}{M}\sum_{i=1}^M f(d_i)\]
\[\text{[email protected]} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}(|\hat{y}_i - y_i| \le 0.1 \cdot y_i)\]
  • Zero-Exclusion Reliability Policy: All models must output strictly valid JSON schemas; any structural formatting failure, markdown leakage, or hedging response is directly scored as incorrect, strictly mirroring production system reliability.

Key Experimental Results

Main Results

The evaluation benchmarks 16 diverse LLMs categorized into General-Domain LLMs, Reasoning-Enhanced LLMs, and Domain-Specific LLMs across all three subtasks.

Model Category Model Name Detection Acc Detection MaF1 Classification Acc Classification MaF1 Correction Acc Correction MaF1
General-Domain GPT-5 (no-thinking) 86.40 81.14 87.69 88.16 58.09 50.44
General-Domain GPT-4o 81.05 76.61 89.34 89.51 61.94 51.30
General-Domain Qwen3-Max 92.03 88.03 94.75 94.48 74.13 65.61
General-Domain DeepSeek-V3.2-Chat 78.87 73.26 84.67 83.88 65.68 58.04
General-Domain GLM-4.6-Chat 87.72 82.68 91.82 91.25 70.99 61.48
General-Domain MiniMax-M2 79.51 73.68 77.23 80.47 57.12 51.77
General-Domain Qwen3-14B-Chat 88.05 83.16 85.41 84.78 55.97 46.70
General-Domain Qwen3-8B-Chat 65.84 62.94 71.14 63.86 52.86 40.35
Reasoning-Enhanced GPT-5-Thinking 84.55 78.26 84.07 82.35 62.83 58.37
Reasoning-Enhanced GLM-4.6-Thinking 90.34 85.52 91.20 90.29 68.86 61.48
Reasoning-Enhanced Qwen3-Max-preview-Thinking 91.57 86.54 91.63 90.81 70.89 65.18
Reasoning-Enhanced DeepSeek-V3.2-Thinking 87.40 82.89 88.51 88.62 72.40 67.36
Reasoning-Enhanced Kimi-K2-Thinking 91.70 85.93 92.55 92.57 71.55 63.27
Reasoning-Enhanced Qwen3-Next-80B-A3B-Thinking 89.19 79.99 88.66 87.27 63.51 54.55
Domain-Specific Farui-Plus 48.05 47.27 66.22 63.56 52.60 41.10
Domain-Specific LawLLM-7B 26.34 25.04 39.85 34.20 31.38 31.65

Ablation Study & Human Performance Comparison

Evaluating the impact of input settings (S1–S5) on Qwen3-Max across the charge error subset, alongside a stratified human benchmark on 150 sampled cases:

Setting Detection Acc Detection MaF1 Classification Acc Classification MaF1 Correction Acc Correction MaF1
S1: Facts + Anomalous Judgment 91.79 87.09 92.56 48.07 86.30 86.50
S2: S1 + Normal Reasoning 97.06 95.09 99.97 49.99 97.71 96.01
S3: S2 + Ground-Truth Law Articles 95.79 92.78 99.97 49.99 97.34 95.46
S4: Facts + Anomalous Reasoning + Anomalous Judgment 94.63 91.47 96.67 49.15 95.46 90.93
S5: S4 + Erroneously Cited Law Articles 92.79 88.29 94.51 48.59 94.14 88.59
Evaluator Detection MaF1 Classification MaF1 Correction MaF1
Qwen3-Max (S4 Setting) 48.46 92.03 53.81
Certified Legal Experts (Human Baseline) 49.15 93.64 65.66

Key Findings

  1. The Inverted Task Difficulty Paradox: Error classification consistently achieves the highest scores across models, outperforming error detection despite detection logically preceding classification. This stems from conditional framing: classification prompts explicitly inform the model that an error exists, narrowing the target space to 3 choices, whereas error detection demands unguided, global skepticism across flawless and flawed documents.
  2. Knowledge-as-Noise Phenomenon: Adding statutory legal provisions systematically degrades performance rather than enhancing it (S3 drops compared to S2; S5 drops compared to S4). Unlike human judges who selectively cross-reference statutory clauses, LLMs struggle to isolate legally critical provisions from extensive statutory text, allowing superfluous legal jargon to act as contextual distractors.
  3. Severe Vulnerability of Legal-Specific Fine-Tuned Models: Specialized legal models (Farui-Plus, LawLLM-7B) collapse on appellate review (detection MaF1 of 47.27 and 25.04). Having been trained primarily on standard judicial templates and legal QA, they exhibit strong confirmation bias—assuming presented judgments are inherently valid—and completely lack adversarial diagnostic auditing capabilities.
  4. Weakness in Global Deduction and Fine-Grained Arithmetic: As depicted in Figure 4, models excel at identifying Erroneous Charges (E1 MaF1 of 90.93) where explicit conceptual clashes occur, but fail drastically on Omission of Charges (E2 MaF1 of 44.03) and Percentage-Based Fines (E6 [email protected] of 17.83). LLMs cannot holistically infer multiple concurring offenses from compound facts, and struggle with strict formulaic arithmetic under statutory percentage constraints.
  5. Instruction Compliance and Formatting Failures: As detailed in Table 19, while most models adhere to the JSON schema, reasoning-enhanced models (e.g., DeepSeek-V3.2-Thinking dropping to 96.44% in classification) occasionally break structural output formatting by generating verbose conversational explanations when encountering high uncertainty in long-tail charges.

Highlights & Insights

  • Shifting Legal AI from Outcome Prediction to Diagnostic Auditing: AR-BENCH pioneers the formalization of appellate judgment review, expanding legal NLP beyond superficial sentence prediction into deep, multi-tiered retrospective diagnostic reasoning.
  • Exploiting the Principle of Legality for Verifiable Ground Truth: By deliberately restricting the benchmark to criminal law, the authors circumvent the subjective liability balancing endemic to civil disputes, anchoring evaluation in mathematically and statutorily provable ground truth.
  • Antidote to Benchmark Contamination via Semantic Perturbation: Replacing full reasoning segments with matched factual precedents rather than substituting isolated lexical tokens forces models to engage in true deductive verification, setting a high standard for future reasoning benchmarks.

Limitations & Future Work

  • Jurisdictional and Civil Scope Constraints: The benchmark is strictly restricted to Chinese criminal law and single-defendant cases, leaving multi-defendant liability, civil disputes involving proportional torts, and common-law stare decisis jurisdictions for future exploration.
  • Exclusion of Evidentiary and Fact-Finding Disputes: The benchmark assumes case facts are established and immutable, testing only statutory application and sentencing mechanics without evaluating courtroom fact-finding or evidence admissibility.
  • Independent Modeling of Cascading Errors: Real-world trial errors often cascade (e.g., an erroneous charge directly corrupts prison terms and fines). AR-BENCH restricts evaluation to isolated primary root causes, leaving joint multi-error dependency modeling as an open challenge.
  • vs CAIL2018 / Auto-Judge / SwissJP: Early legal judgment prediction benchmarks focus exclusively on forward outcome prediction from facts using 2–4 coarse labels; AR-BENCH provides 20 fine-grained attributes and targets retrospective error diagnosis on finalized judgments.
  • vs AppealCase: AppealCase explores civil appeals where outcomes depend on subjective judicial discretion; AR-BENCH focuses on criminal statutory boundaries where error corrections are strictly deterministic and objectively verifiable.
  • vs JuDGE: JuDGE evaluates end-to-end forward generation of judicial documents; AR-BENCH deconstructs and audits finalized documents via an adversarial fact-checking paradigm.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates a pioneering retrospective appellate review diagnostic task with verifiable criminal ground truth.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluates 16 LLMs across 5 input variations, 6 fine-grained error types, expert human baselines, and granular failure modes.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear task mathematical formalization, clean structural layout, and sharp empirical insights.
  • Value: ⭐⭐⭐⭐⭐ Establishes a foundational testbed for deploying LLMs into high-stakes judicial auditing and regulatory compliance.