Skip to content

OBJECTION! Lawyer Agents Mitigate Guilty Bias in Legal Judgment Prediction

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Legal Judgment Prediction, Guilty Bias, Adversarial Lawyer Agent, Trichotomous Reasoning, Inference-time Debiasing

TL;DR

Addressing the critical "Guilty Bias" in Legal Judgment Prediction (LJP) caused by prosecutorial narratives in court records, this paper introduces OBJECTION—a training-free inference-time framework that embeds an adversarial Lawyer Agent into a three-step legal reasoning structure (Offense, Unlawfulness, Culpability)—alongside a new 3.4k real-world "Natural Innocent" benchmark, drastically slashing the False Guilty Rate from 82.93% to 16.69%.

Background & Motivation

The fundamental tenet of criminal justice, "It is better that ten guilty persons escape than that one innocent suffer," embodies the presumption of innocence. However, criminal proceedings are intrinsically non-neutral: by the time an indictment reaches a formal courtroom trial, investigative authorities have already established a substantial suspicion of guilt. Written judicial records inherently reflect prosecutorial narratives structured to justify conviction, compounded by massive label imbalances toward guilty outcomes in public legal corpora. Consequently, Large Language Models (LLMs) trained on standard LJP benchmarks systematically internalize this "Guilty Bias," naively accepting biased narrative descriptions as objective, neutral ground-truth facts. This causes unacceptable societal risks where models convict innocent individuals, yielding dangerously high False Guilty Rates (FGR).

Prior literature attempted to mitigate this bias using multi-step legal reasoning, syllogistic prompting, or explicit criminal constitutive element schemas. However, these methods operate under the fragile premise that the input fact summary is unbiased; when faced with one-sided prosecutorial prose, structured reasoning merely guides the model into an elaborate confirmation of guilt. Recent approaches that fine-tune models on synthetically rewritten counterfactual innocent cases (such as LJPIV) also fall short at inference time: synthetic innocent cases frequently inject artificial, unnatural exculpatory cues (e.g., bizarre causal behaviors), inducing models to rely on superficial shortcuts that fail when deployed on natural, implicit real-world acquittals. Meanwhile, generic critique-and-revise loops and unstructured multi-agent debates lack normative role asymmetry and often turn into echo chambers that reinforce the model's initial guilty assumptions.

To resolve this dilemma, the authors argue that narrative bias must be counteracted at inference time through an institutionalized adversarial mechanism reflecting legal due process. Rather than seeking passive consensus, the model must operationalize the legal principle of In Dubio Pro Reo (when in doubt, favor the accused). The core idea is to instantiate a single LLM into an adversarial Judge and Lawyer Agent, embedding the Lawyer Agent into each step of the criminal law trichotomous structure (Offense, Unlawfulness, Culpability) to actively excavate reasonable doubts anchored by a SOAM factual schema, thereby structurally neutralizing guilty bias at inference time without requiring any fine-tuning.

Method

Overall Architecture

The OBJECTION framework reframes the conventional direct prediction \(\hat{y} = \mathcal{M}(X_{\text{fact}})\) into an ordered, three-stage dialectical interaction between a Judge \(\mathcal{M}\) and an adversarial Lawyer Agent \(\mathcal{A}\). The reasoning flow maps onto the classical Continental civil law trichotomous framework of crime \(\mathcal{T} = \{\text{offense}, \text{unlawfulness}, \text{culpability}\}\).

For an input case description \(X_{\text{fact}}\), the pipeline first extracts a structured SOAM factual representation to ground subsequent defense arguments. It then proceeds sequentially through the three dogmatic stages: Constituent Elements of Offense, Justification of Unlawfulness, and Responsibility/Culpability. At each stage \(k \in \mathcal{T}\), the Judge generates an initial evaluation \(j_k\); the Lawyer Agent inspects \(j_k\) and the facts to construct a target-specific legal defense \(d_k\); and the Judge subsequently reviews the defense under strict fact-checking constraints to deliver the refined stage judgment \(\hat{y}_k\). Crucially, an early-exit mechanism terminates the pipeline and delivers an immediate acquittal if an exculpatory ground is sustained at any higher stage, preventing illogical subsequent discussions and cutting computational overhead.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Input Fact Description Xfact"] --> Step0["SOAM Factual Schema Extraction<br/>Subject / Object / Act / Mens Rea"]
    Step0 --> Step1["Trichotomous Reasoning & Issue Isolation: Offense<br/>Initial Judge: Check Constituent Elements"]
    Step1 --> Step1Adv["Adversarial Defense & In Dubio Pro Reo<br/>Lawyer challenges elements, intent & causation"]
    Step1Adv --> Step1Judge["Stage Judge Final Deliberation"]
    Step1Judge -->|Elements Not Met / In Doubt| ExitOffense["Early Exit: Acquittal by Offense"]
    Step1Judge -->|Offense Established| Step2["Trichotomous Reasoning & Issue Isolation: Unlawfulness<br/>Initial Judge: Check Justification Grounds"]
    Step2 --> Step2Adv["Adversarial Defense & In Dubio Pro Reo<br/>Lawyer argues Self-Defense / Necessity"]
    Step2Adv --> Step2Judge["Stage Judge Final Deliberation"]
    Step2Judge -->|Justification Established| ExitUnlaw["Early Exit: Acquittal by Unlawfulness"]
    Step2Judge -->|Unlawfulness Established| Step3["Trichotomous Reasoning & Issue Isolation: Culpability<br/>Initial Judge: Check Responsibility Capacity"]
    Step3 --> Step3Adv["Adversarial Defense & In Dubio Pro Reo<br/>Lawyer argues Insanity / Lack of Expectability"]
    Step3Adv --> Step3Judge["Stage Judge Final Deliberation"]
    Step3Judge -->|Excuse Established| ExitCulp["Early Exit: Acquittal by Culpability"]
    Step3Judge -->|All 3 Stages Upheld| Conviction["Granular Verdict Output: Conviction (Guilty)"]

Key Designs

1. SOAM Factual Schema Extraction: Decoupling charge-specific priors to anchor legal defense

Unstructured case narratives often interweave objective actions with subjective culpability, blinding models into equating tragic physical outcomes with criminal malice. Existing four-element information extraction approaches require manual rule definition across specific charges, limiting generalization. OBJECTION introduces the SOAM schema, a zero-shot extraction that decomposes raw narratives into Subject (S), Object (O), Actus Reus (A, strictly objective physical conduct and consequence), and Mens Rea (M, subjective states inferred from contextual clues). For instance, in an armed altercation where the defendant repeatedly dialed the emergency police hotline (112) while restraining an assailant who subsequently died of neck compression, the extractor assigns the fatal neck restraint to Actus Reus (A) but maps the urgent calls to the police into Mens Rea (M). This prevents the LLM from conflating a lethal result with murderous intent. Used exclusively in the Offense stage, SOAM provides the essential factual anchors that the Lawyer Agent needs to construct sharp, legally grounded rebuttals without hallucination.

2. Trichotomous Reasoning & Issue Isolation: Enforcing procedural dogmatics to halt bias cascade

LLMs prompted with plain chain-of-thought or naive multi-step prompts frequently exhibit cognitive contamination, mixing self-defense considerations into initial element checks or confusing diminished mental capacity with act unlawfulness. OBJECTION rigidly adopts the criminal law trichotomy: Step 1 (Offense) strictly determines whether the conduct satisfies statutory objective and subjective elements, explicitly banning consideration of justification or excuses; Step 2 (Unlawfulness) proceeds only if the offense is established, evaluating grounds such as self-defense or emergency evacuation; Step 3 (Culpability) isolates personal culpability, assessing insanity, minority, or the doctrine of expectability. By enforcing strict contextual isolation via stage-specific system prompts, the model is prevented from taking premature shortcuts or muddying the burden of proof across distinct procedural tiers.

3. Adversarial Defense & In Dubio Pro Reo: Breaking narrative echo chambers via asymmetric lawyer roles

Standard self-critique mechanisms act as passive validity checkers that inevitably amplify pre-existing biases when iterating over guilt-weighted inputs. OBJECTION injects a domain-specific Lawyer Agent \(\mathcal{A}\) whose explicit mandate is to reverse the burden of proof under the maxim of In Dubio Pro Reo. The interaction at stage \(k\) is formally characterized as: $\(j_k = \mathcal{M}(X_{\text{fact}}), \quad d_k = \mathcal{A}(j_k, X_{\text{fact}})\)$ $\(\hat{y}_k = \mathcal{M}(j_k, d_k, X_{\text{fact}})\)$ Upon receiving the Judge's initial ruling \(j_k\), the Lawyer Agent actively interrogates prosecutorial assumptions—citing textual evidence to negate intent in the Offense stage, highlighting urgency and unavoidable necessity in the Unlawfulness stage, or establishing duress and mental illness in the Culpability stage. The final stage judgment \(\hat{y}_k\) is rendered by the Judge role, explicitly prompted with 15 years of judicial experience to adopt only defense points verified by the raw facts, rejecting any ungrounded speculation. This asymmetric, institutionalized counterweight shatters the consensus loop without degenerating into unprincipled over-acquittal.

4. Granular Verdict Output & Early Exit: Unifying normative explainability with dynamic computational efficiency

In alignment with the hierarchical structure of criminal liability, OBJECTION introduces an Early Exit mechanism: the moment a higher-order stage is negated (e.g., constituent elements missing, or self-defense validated), the inference pipeline halts immediately and returns an acquittal, bypassing all subsequent stages. Normatively, this eliminates the doctrinal absurdity of deliberating self-defense for an act that does not satisfy an offense, while yielding granular, explainable verdicts (Acquittal by Offense, Unlawfulness, or Culpability). Computationally, while exhaustive multi-agent debates require a fixed token-heavy budget (e.g., Debate-Feedback demands 8.0 calls and 16–18s latency), early exits reduce OBJECTION's expected calls to 5.57–7.17 and latency to 5.77–6.77s on CAIL, slashing inference latency by 58%–69%.

Benchmark & Evaluation Protocol

To overcome the pitfalls of synthetic innocence datasets (which rely on counterfactual rewrites containing unnatural, explicit clues that lead to shortcut overfitting), the authors curate Natural Innocent, a real-world benchmark of 3,412 first-instance criminal judgments from South Korean courts. The dataset contains 1,291 guilty cases and 2,121 legitimate acquittals categorized across the trichotomous framework: 1,464 Offense-level acquittals (e.g., lack of proof, debt default vs. fraud), 431 Unlawfulness justifications (self-defense, justifiable acts), and 226 Culpability excuses (lack of expectability, mental incapacity).

Evaluation leverages normative error metrics reflecting legal risk alongside standard classification metrics: - False Guilty Rate (FGR): \(\text{FGR} = \frac{\text{FP}}{\text{TN} + \text{FP}}\), the ratio of innocent defendants wrongly convicted (measuring violations of the presumption of innocence). - False Not-Guilty Rate (FNR): \(\text{FNR} = \frac{\text{FN}}{\text{TP} + \text{FN}}\), the ratio of guilty offenders wrongly acquitted. - Standard class F1 scores: Guilty F1 (G-F1), Not-Guilty F1 (NG-F1), Macro-F1, and 3-Step Reasoning F1.

Key Experimental Results

Main Results

Evaluations across three open-weight backbones (Qwen-2.5-7B-Instruct, Llama-3.1-8B-Instruct, Gemma-2-9B-it) demonstrate OBJECTION's consistent ability to eliminate guilty bias while sustaining superior overall accuracy.

Dataset Backbone & Method G-F1 NG-F1 Macro-F1 3-Step F1 FGR (%)↓ FNR (%)↓
LJPIV-CAIL LJPIV (Fine-tuned SOTA) 0.80 0.83 0.82 0.61 9.29 27.14
(Qwen-2.5-7B) Baseline D (Schema+Tri) 0.77 0.62 0.70 0.39 53.21 3.93
Self-Refine 0.77 0.64 0.71 0.57 49.29 6.79
Debate-Feedback 0.74 0.76 0.75 0.62 22.14 27.50
OBJECTION (Ours) 0.75 0.80 0.78 0.82 10.71 33.93
LJPIV-ELAM LJPIV (Fine-tuned) 0.80 0.68 0.74 0.71 47.60 2.40
(Qwen-2.5-7B) Baseline D (Schema+Tri) 0.75 0.58 0.67 0.37 56.40 5.60
Self-Refine 0.76 0.58 0.67 0.59 57.60 3.20
Debate-Feedback 0.80 0.74 0.77 0.67 27.00 15.00
OBJECTION (Ours) 0.78 0.77 0.78 0.81 23.20 21.60
Natural Innocent LJPIV (Fine-tuned) 0.35 0.29 0.32 0.11 82.93 50.12
(Qwen-2.5-7B) Baseline D (Schema+Tri) 0.55 0.27 0.41 0.21 84.49 9.99
Self-Refine 0.57 0.23 0.40 0.24 86.28 2.87
Debate-Feedback 0.71 0.73 0.72 0.44 42.95 5.34
OBJECTION (Ours) 0.70 0.82 0.76 0.47 16.69 31.29
Natural Innocent Baseline D (Schema+Tri) 0.51 0.66 0.58 0.38 36.73 45.08
(Llama-3.1-8B) Self-Refine 0.53 0.33 0.43 0.34 78.55 16.89
Debate-Feedback 0.41 0.78 0.59 0.41 7.69 70.72
OBJECTION (Ours) 0.63 0.65 0.64 0.49 46.39 19.05
Natural Innocent Baseline D (Schema+Tri) 0.57 0.18 0.38 0.32 90.19 1.16
(Gemma-2-9B) Self-Refine 0.60 0.34 0.47 0.39 79.73 0.15
Debate-Feedback 0.49 0.69 0.59 0.47 37.01 50.74
OBJECTION (Ours) 0.60 0.73 0.67 0.50 29.89 36.10

On commercial API backbones (GPT-4.1-mini and Gemini-2.5-Flash), OBJECTION similarly maintains superior fairness and reasoning quality: on Natural Innocent, GPT-4.1-mini's 3-step reasoning F1 jumps from 0.08 to 0.34 while curbing FGR to 58.46% (vs. Baseline D's 59.71%); Gemini-2.5-Flash's 3-step F1 rises from 0.03 to 0.33 with FGR reduced from 76.75% to 61.95%.

Ablation Study

An ablation on the out-of-domain LJPIV-ELAM benchmark (using Qwen-2.5-7B) isolates the contribution of each architectural pillar.

Configuration G-F1 NG-F1 FGR (%)↓ FNR (%)↓ Finding & Operational Role
OBJECTION (Full) 0.78 0.77 23.20 21.60 Full pipeline balancing guilt detection and innocence defense
(-) Lawyer Agent 0.83 0.75 38.80 2.00 Removing lawyer causes FGR to surge by 15.6 points
(-) Schema (SOAM) 0.52 0.72 8.00 62.40 Without factual anchors, lawyer hallucinates and over-acquits
(-) 3-Step Structure 0.38 0.56 34.80 68.00 Without stage isolation, reasoning collapses across all fronts
Ours w/ Normal Critic 0.77 0.60 56.40 2.00 Generic critic confirms prosecutorial bias, yielding high FGR
LJPIV (Fine-tuned) 0.80 0.68 47.60 2.40 SOTA baseline exhibits heavy bias on out-of-domain natural cases
LJPIV + Lawyer Agent 0.01 0.67 0.40 99.60 Combining synthetic fine-tuning + lawyer creates hyper-acquittal

In a blinded human expert evaluation conducted by veteran police investigators on 50 randomly sampled Natural Innocent cases (5-point scale), OBJECTION scored 4.42 overall (Legal Relevance: 4.16, Argument Strength: 4.52, Factual Grounding: 4.86, Human Similarity: 4.12), vastly outperforming Debate-Feedback (2.87 overall). In step-wise analysis, the Lawyer Agent performed best in Unlawfulness with a perfect 5.00 in Factual Grounding and 4.57 overall.

Key Findings

  • Fine-tuned synthetic models catastrophically fail on natural acquittals: LJPIV, which achieves 9.29% FGR on the synthetic CAIL benchmark, collapses to an 82.93% FGR on Natural Innocent with a 3-step reasoning F1 of only 0.11. This proves that synthetic counterfactual samples instill brittle shortcut heuristics that fail when faced with subtle, implicit real-world innocence contexts.
  • Unstructured debate exhibits uncalibrated backbone volatility: Stochastic round-table debate (Debate-Feedback) lacks consistent steering. When running on Llama-3.1-8B, it falls into indiscriminate over-acquittal, misclassifying 70.72% of actual guilty defendants as innocent (FNR = 70.72%). In contrast, OBJECTION enforces a controlled, directional correction across all backbones, maintaining top-tier Macro-F1.
  • Factual schemas and adversarial personas are mutually indispensable: Removing the SOAM schema causes FNR to jump to 62.40%, as an unconstrained Lawyer Agent blindly invents doubt. Conversely, removing the Lawyer Agent while retaining the schema leads to an FGR spike to 38.80%. Structural anchors keep the defense grounded, while the adversarial persona forces the model to challenge prosecutorial certainty.
  • Early-exit yields significant latency and token reductions: On CAIL, OBJECTION requires only 5.57 to 7.17 average LLM calls compared to 8.0 fixed calls in Debate-Feedback. By exiting early in early tiers for clear acquittals, OBJECTION slashes inference latency by 57.9% on Qwen-2.5 (6.77s vs. 16.08s) and 69.1% on Llama-3.1 (5.77s vs. 18.70s).

Highlights & Insights

  • Architectural translation of "In Dubio Pro Reo": Rather than treating bias reduction as a data curation or fine-tuning problem, OBJECTION formalizes the legal defense right and presumption of innocence directly into the multi-agent inference graph.
  • Tunable steerability via persona modulation: By adjusting the aggression of the defense persona (the "OBJECTION-Soft" variant), practitioners gain an explicit control knob to dial between crime suppression (higher G-F1) and human rights protection (lower FGR), avoiding the black-box unpredictability of stochastic prompting.
  • Introduction of the Natural Innocent benchmark: The curation and release of 3,412 real-world Korean court judgments with three-tier legal ground annotations provide the community with an essential reality check against shortcut-learning traps in legal AI.

Limitations & Future Work

  • Civil Law doctrinal bias: The 3-step hierarchy (Offense, Unlawfulness, Culpability) is tailored to Continental Civil Law jurisdictions (e.g., Germany, South Korea, Japan). Translating this to Common Law systems, which center on case-law precedents and jury-centric reasonable doubt, requires further investigation.
  • Generic abstraction of the SOAM schema: While SOAM scales zero-shot across diverse offenses, it lacks jurisdiction- and crime-specific constituent elements, which could limit nuanced defenses in highly technical crimes (e.g., complex white-collar embezzlement or cyber fraud).
  • Binary verdict limitation: The current framework addresses only binary guilty/not-guilty verdicts. Extending adversarial agents to multi-class charge selection and sentencing prediction remains an important open direction.
  • vs. LJPIV (Zhang et al., EMNLP 2025): LJPIV pioneered trichotomous modeling but relied on fine-tuning over synthetic innocence data, resulting in acute shortcut learning and an 82.93% FGR on natural cases. OBJECTION demonstrates that a training-free, inference-time adversarial agent with factual anchors is far more robust and generalizable.
  • vs. Debate-Feedback (Chen et al., NAACL 2025): Debate-Feedback uses symmetric multi-agent debate that readily lapses into an unconstrained echo chamber or erratic over-acquittal (FNR up to 70.72%). OBJECTION establishes an asymmetric, legally disciplined Judge-Lawyer dynamic governed by procedural early-exit logic, achieving vastly higher factual grounding (4.86 vs. 3.64).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant formalization of procedural justice and the presumption of innocence into a test-time multi-agent architecture.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 3 open and 2 closed LLMs, synthetic and real datasets, ablation studies, efficiency benchmarks, and blinded expert evaluations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative bridging Continental criminal law doctrine and NLP reasoning mechanics.
  • Value: ⭐⭐⭐⭐⭐ Sets a new standard for high-stakes normative AI alignment, proving that adversarial inference can overcome foundational dataset biases.