How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awards¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Legal NLP, Non-Pecuniary Damages, Continuous Monetary Prediction, ECtHR, Extreme Tail Calibration
TL;DR¶
This paper introduces ECtHR-NPD, the first continuous monetary regression benchmark for predicting Article 41 non-pecuniary damage awards at the European Court of Human Rights across 14,575 cases, revealing that feature-based gradient-boosted trees and fine-tuned encoders consistently outperform frontier prompted LLMs and ReAct agents due to severe failures in zero-award identification and heavy-tail magnitude calibration.
Background & Motivation¶
Legal natural language processing benchmarks have predominantly focused on discrete classification, holding extraction, prior-case retrieval, and statutory rule-bounded predictions such as criminal sentencing ranges or tax liabilities. However, in real-world judicial adjudications, courts frequently make continuous monetary decisions where no statutory formula, tariff table, or transparent mathematical algorithm dictates the outcome. The European Court of Human Rights (ECtHR) operates under Article 41 of the European Convention on Human Rights (ECHR) to award just satisfaction, specifically non-pecuniary damage (NPD) for intangible harms such as emotional suffering, mental distress, humiliation, and loss of dignity. Because the Court assesses NPD purely on an equitable basis rather than through documentary financial receipts or precise statutory tables, predicting these awards presents an ideal testbed for evaluating whether NLP models can extract legally relevant facts and calibrate continuous monetary outcomes under judicial discretion.
Constructing a benchmark for continuous discretionary remedies involves substantial empirical and technical challenges. First, judicial judgments are long, heterogeneous documents where case descriptions, merits deliberations, and compensation proceedings are tightly intertwined. Without rigorous structural separation, benchmarks suffer from catastrophic target leakage: applicants' claimed compensation amounts act as artificial ceilings or anchors, and the judge's Article 41 reasoning directly reveals the award outcome before prediction occurs. Second, the ground-truth distribution of NPD awards is heavily zero-inflated and heavy-tailed: 20% to 33% of cases receive zero monetary awards due to procedural non-compliance or because the finding of a violation itself is deemed sufficient, while a tiny fraction of severe cases result in awards exceeding millions of euros. Standard language models, which process numbers as arbitrary token sequences, lack native distance metrics and fail to penalize magnitude errors proportionally.
Furthermore, real-world international litigation frequently involves joined applications with dozens or hundreds of applicants and multiple concurrent violations, where the Court typically issues a single global lump-sum award rather than a linear additive sum. When prompted with complex multi-applicant cases, existing large language models and chain-of-thought agents intuitively resort to linear per-applicant multiplication, resulting in catastrophic multi-order-of-magnitude over-predictions. To address these critical limitations, this work establishes a rigorous data extraction pipeline, strict leakage-control contracts, and chronological diagnostic evaluation views. Core idea: Establish ECtHR-NPD as the first leakage-controlled, chronologically partitioned benchmark for continuous non-pecuniary damage awards, demonstrating through six model families that prompted LLMs and agentic reasoning fail to beat structured gradient-boosted trees in continuous monetary calibration.
Method¶
Overall Architecture¶
The ECtHR-NPD task frames non-pecuniary damage award prediction as a continuous regression problem: given input features \(X\) containing case metadata, violated Convention articles, the factual background (The FACTS section), and external state-year macroeconomic covariates (GDP and GDP per capita), the model must predict the case-level nominal euro award \(\hat{y} \in \mathbb{R}_{\ge 0}\). Crucially, all award-related information—including applicants' Article 41 claims, Government observations, the Court's compensation reasoning, operative payment provisions, and appendix award tables—is strictly quarantined and used solely for deterministic target validation.
The end-to-end framework comprises: structural document segmentation and eight-fold target validation contracts, dual-modality representations (raw text and leakage-audited structured features with Late-Fusion encoders), chronological splitting with greedy quota matching for diagnostic views (In-Distribution, Out-of-Distribution, and Challenging), and a comprehensive multi-family evaluation protocol.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Raw HUDOC Judgments<br/>(18,367 English judgments)"] --> B["Leakage Isolation & Target Validation<br/>Quarantine Art. 41 & Operative text / 8 validation checks"]
B --> C["Dual-Modality Representation & Late Fusion<br/>Long FACTS encoding + Structured feature MLP branch"]
C --> D["Chronological Split & Diagnostic Matching<br/>Temporal 70/10/20 + Greedy Quota Matching (ID/OOD/Chal.)"]
D --> E["Multi-Family Model & Calibration Evaluation<br/>Trees, Encoders, Prompted Decoders & Sandboxed ReAct"]
E --> F["Case-Level Monetary Prediction Output<br/>(Continuous euro amount y in R>=0)"]
Key Designs¶
1. Leakage Isolation & Target Validation: Eliminating Causal Leakage via Strict Semantic Contracts
To prevent models from exploiting explicit claim amounts or compensation rationales that act as trivial outcome anchors, the benchmark enforces a rigorous boundary between model inputs and target construction. From 18,367 initial HUDOC judgments, the pipeline excludes all Article 41 submissions, Government observations, operative payment orders, and appendix award tables from the input \(X\). To ensure label fidelity, candidate awards undergo eight deterministic semantic validation checks: ① head separation, excluding multi-head totals combining pecuniary damages, costs, and NPD (retaining only 72 audited bundled-proxy cases with verified absence of financial harm); ② applicant-beneficiary consistency; ③ per-applicant sum reconciliation; ④ no-claim and positive-award incompatibility; ⑤ claim-award ceiling consistency; ⑥ historical currency normalisation to nominal euros; ⑦ operative-provision recoverability; and ⑧ an accepted six-class zero-award legal taxonomy (finding of violation sufficient, no claim submitted under Rule 60, unsubstantiated claim, Rule 60 procedural non-compliance, domestic redress sufficient, and deceased applicant without heirs). The resulting dataset contains 14,575 validated cases.
2. Dual-Modality Representation & Late Fusion: Compressing Long Legal Narratives with Hybrid Embeddings
Because raw ECtHR fact sections are exceptionally lengthy (with the 95th percentile reaching 5,337 words), models face significant attention degradation and context distraction. The benchmark introduces two complementary input representations: raw text (metadata, full facts, violated articles, macro covariates) and structured features (compact extracted case descriptors such as applicant counts, age groups, sex, violation duration, substantive versus procedural violation limbs, and log-transformed macroeconomic variables). The structured format compresses input length at the 95th percentile to just 237 words (a 95.6% reduction). For representation learning, a Late-Fusion architecture combines the best of both worlds: a long-context encoder (ModernBERT with 8,192 tokens or Legal-Longformer with 4,096 tokens) processes the raw text to produce a pooled \([CLS]\) embedding, which is concatenated with a 768-dimensional latent vector generated by a two-layer MLP (hidden dimension 256) over the structured features, followed by a linear regression head optimized via SmoothL1 loss on standardized \(\log(1 + y)\) targets.
3. Chronological Split & Diagnostic Matching: Preventing Future Leakage and Disentangling Distribution Shift
To simulate genuine prospective judicial forecasting and prevent chronological leakage, cases are sorted by judgment date and partitioned into a 70% training split (1968 to July 2019, 10,217 cases), a 10% validation split (July 2019 to December 2021, 1,461 cases), and a 20% held-out test pool (December 2021 to March 2026, 2,897 cases). Because post-2021 docket dynamics shifted dramatically (e.g., three-judge Committee summary judgments surged from 19.4% to 75.1%, and multi-applicant cases rose to 45.8%), the test pool is partitioned via a weighted Greedy Quota Matching algorithm (Algorithm 1) over eight structural dimensions. This isolates an In-Distribution (ID, \(n=1,000\), Mean Total Variation = 0.025) subset aligned with training distributions from the residual Out-of-Distribution (OOD, \(n=1,897\), Mean TV = 0.211) subset. Additionally, a legally expert-curated Challenging diagnostic subset (\(n=699\), Mean TV = 0.435) aggregates all 17-judge Grand Chamber judgments and cases combining multiple applicants with multiple concurrent violation findings, providing an explicit diagnostic lens on complex legal aggregation.
4. Multi-Family Model & Calibration Evaluation: Systematic Cross-Paradigm Auditing with Sandboxed ReAct
The benchmark conducts an exhaustive empirical evaluation across six distinct model families on identical linear euro targets: ① constant baselines (training median, mean, and constant zero); ② gradient-boosted decision trees (CatBoost, XGBoost, LightGBM) trained on log-transformed structured features; ③ precedent retrieval baselines (kNN on features, BM25S, and BGE-M3 dense/sparse representations taking the median award of top-20 training neighbours sharing violated articles); ④ fine-tuned long-sequence encoders (ModernBERT and Legal-Longformer text and late-fusion variants); ⑤ prompted frontier decoder LLMs (Qwen3.5 suite, GPT-OSS-20B, and GPT-5.4 under zero-shot, static 5-step CoT, and 5-shot retrieved CoT); and ⑥ knowledge-augmented ReAct agents operating via whitelisted tool controllers without web access. Models are audited not only on primary MAE, but also on extreme-award sensitivity (\(R^2\), RMSE), tail error (95AE), ranking correlation (Spearman \(\rho\)), zero-award classification (Zero-F1), and segmented error buckets.
Benchmark & Evaluation Protocol¶
Hyperparameters and model selection are strictly tuned on the validation set, with the test set locked until final scoring. Tree models employ cross-validated hyperparameter grids; encoders are optimized with AdamW, linear learning rate warmup and decay, batch size 16 (gradient accumulation 16), and bf16 mixed precision. Prompted decoder LLMs use greedy decoding (\(T=0\)) with strict JSON schema constraints. The ReAct agent environment enforces a 12-step budget and sandboxed tools (inspect_case, query_target_information, search_modules, resolve_empirical_priors, retrieve_train_references, assess_aggregation_pattern, and pre-prediction leakage_check). An "Expanded ReAct" diagnostic condition deliberately injects applicant claims and Article 41 reasoning to empirically quantify the extent to which LLMs depend on leaked target-adjacent cues.
Key Experimental Results¶
Main Results¶
Performance across the test pool (\(n=2,897\)) and diagnostic views (ID, OOD, Challenging) for all model families is summarized below (all monetary errors are reported in nominal EUR):
| Model | Variant / Configuration | Test MAE↓ | RMSE↓ | \(R^2\)↑ | MedAE↓ | 95AE↓ | Spearman \(\rho\)↑ | Zero-F1↑ | ID MAE↓ | OOD MAE↓ | Chal. MAE↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Constant baselines | |||||||||||
| Train median | — | 11,006 | 47,223 | -0.034 | 3,200 | 38,820 | 0.000 | 0.000 | 9,752 | 11,668 | 22,812 |
| Train mean | — | 14,940 | 46,468 | -0.001 | 11,010 | 29,010 | 0.000 | 0.000 | 14,465 | 15,190 | 25,150 |
| Constant zero | — | 11,713 | 47,904 | -0.064 | 3,400 | 42,020 | 0.000 | 0.495 | 10,309 | 12,454 | 22,916 |
| Gradient-boosted trees | |||||||||||
| CatBoost | struct. feats. | 9,881 | 44,231 | 0.093 | 2,315 | 37,819 | 0.533 | 0.190 | 8,143 | 10,797 | 20,987 |
| XGBoost | struct. feats. | 10,117 | 44,713 | 0.073 | 2,404 | 34,740 | 0.523 | 0.123 | 8,180 | 11,138 | 21,527 |
| LightGBM | struct. feats. | 10,240 | 50,785 | -0.195 | 2,395 | 33,020 | 0.518 | 0.104 | 9,312 | 10,729 | 21,085 |
| Retrieval baselines | |||||||||||
| BGE-M3 (sparse) | raw text | 10,226 | 43,839 | 0.109 | 3,200 | 33,000 | 0.321 | 0.530 | 8,796 | 10,980 | 20,378 |
| BGE-M3 (dense) | raw text | 10,177 | 43,401 | 0.127 | 3,150 | 33,020 | 0.323 | 0.425 | 8,354 | 11,138 | 21,467 |
| Trained Encoder LMs | |||||||||||
| ModernBERT | raw text | 10,133 | 42,232 | 0.173 | 2,997 | 35,759 | 0.329 | 0.209 | 8,438 | 11,026 | 21,544 |
| ModernBERT | late fusion | 10,074 | 41,579 | 0.199 | 2,999 | 36,391 | 0.312 | 0.067 | 8,404 | 10,955 | 21,024 |
| Legal-Longformer | late fusion | 10,244 | 42,439 | 0.165 | 2,760 | 36,021 | 0.487 | 0.226 | 8,695 | 11,060 | 21,013 |
| Prompted Decoder LMs | |||||||||||
| Qwen3.5-9B | zero-shot | 16,598 | 110,079 | -4.616 | 6,750 | 35,000 | 0.190 | 0.196 | 17,113 | 16,327 | 24,212 |
| Qwen3.5-9B | static CoT | 10,782 | 44,551 | 0.080 | 4,500 | 31,005 | 0.183 | 0.255 | 9,777 | 11,312 | 20,926 |
| Qwen3.5-Plus | zero-shot | 16,400 | 68,894 | -1.199 | 5,000 | 51,575 | 0.290 | 0.115 | 12,781 | 18,306 | 37,186 |
| Qwen3.5-Plus | few-shot CoT (5-ref)† | 24,141 | 79,094 | -1.900 | 6,000 | 105,000 | 0.235 | 0.076 | 13,518 | 29,741 | 60,814 |
| GPT-OSS-20B | few-shot CoT (5-ref)† | 11,674 | 46,410 | 0.002 | 5,000 | 36,725 | 0.155 | 0.065 | 10,007 | 12,553 | 24,688 |
| GPT-5.4 | zero-shot | 25,438 | 85,330 | -2.375 | 4,800 | 104,000 | 0.249 | 0.006 | 17,140 | 29,812 | 47,613 |
| GPT-5.4 | static CoT | 16,266 | 48,458 | -0.088 | 5,000 | 72,000 | 0.271 | 0.019 | 10,754 | 19,172 | 39,884 |
| Knowledge Agents | |||||||||||
| Qwen3.5-Plus | ReAct base agent† | 18,220 | 55,386 | -0.420 | 5,500 | 65,000 | 0.204 | 0.002 | 11,156 | 21,946 | 47,497 |
| MiniMax-M2.7 | ReAct base agent† | 16,571 | 52,914 | -0.298 | 5,000 | 59,840 | 0.199 | 0.029 | 10,911 | 19,555 | 40,758 |
Ablation Study¶
The breakdown of MAE performance across target-award ranges isolates boundary calibration errors at zero from extreme-value tail errors (amounts in EUR):
| Model System | Configuration | Zero Awards (n=953) | >0 ~ 10k (n=1,333) | 10k ~ 50k (n=493) | >50k Extreme Tail (n=118) |
|---|---|---|---|---|---|
| CatBoost | Structured features | 3,089 | 2,401 | 15,486 | 125,816 |
| BGE-M3 (sparse) | Retrieved median | 2,858 | 2,567 | 16,010 | 132,087 |
| ModernBERT | Late fusion | 3,367 | 2,484 | 15,722 | 126,386 |
| Qwen3.5-9B | Static CoT | 6,597 | 3,713 | 13,404 | 113,482 |
| GPT-OSS-20B | Few-shot CoT† | 7,926 | 3,077 | 14,791 | 126,038 |
| MiniMax-M2.7 | ReAct base agent† | 21,977 | 6,345 | 13,121 | 102,854 |
Diagnostic ablations provide crucial insights into model behavior: 1. Expanded ReAct Diagnostic: When claim amounts and Article 41 reasoning are exposed to the agent, MAE drops dramatically by 51.6% for Qwen3.5-Plus (from 18,220 to 8,810) and by 46.8% for MiniMax-M2.7 (from 16,571 to 8,819), while Zero-F1 surges from near zero (0.002 and 0.029) to near perfect (0.997 and 0.988). This proves that LLMs primarily lean on explicit settlement anchors rather than discerning intrinsic injury severity from factual narratives. 2. Input Serialisation Ablation: Replacing raw text facts with structured key-value features reduces MAE significantly for zero-shot models (Qwen3.5-9B improves by 3,380; Qwen3.5-27B by 4,117; GPT-OSS-20B CoT by 5,041), confirming that eliminating long-context cognitive load helps decoder LMs avoid extreme hallucinations, though it does not resolve magnitude scaling.
Key Findings¶
- Structured tree models consistently outperform frontier decoder LLMs and agentic reasoning: CatBoost attains the lowest aggregate test MAE (9,881 EUR), outperforming the training median by 10.2%. Paired bootstrap tests (\(Pr(A < B) = 1.000\)) confirm that CatBoost significantly outperforms all prompted decoder models and agent configurations. Frontier proprietary models (e.g., GPT-5.4, Qwen3.5-Plus) exhibit severe variance and frequently yield negative \(R^2\) values.
- Universal failure in zero-award identification and heavy-tail magnitude calibration: While zero-award judgments constitute 32.9% of the test pool, prompted LLMs and ReAct agents consistently predict substantial positive awards for them, resulting in massive zero-bucket MAEs (6,500 to 22,000 EUR). Conversely, in the upper tail (>50k EUR, representing only 4.1% of test cases), all model families suffer immense errors exceeding 100,000 EUR.
- Catastrophic linear scaling in multi-applicant litigation: In joined proceedings with large numbers of applicants, LLM reasoning traces generate plausible qualitative legal arguments but fail quantitatively by assuming linear per-applicant summation. For instance, in case 001-219675 (90 applicants, true global award 12,500 EUR), Qwen3.5-Plus predicted 450,000 EUR; in case 001-241738 (195 applicants, true global award 7,500 EUR), it predicted 1,500,000 EUR—an error factor of 200×.
Highlights & Insights¶
- A Bulletproof Anti-Leakage Protocol: By establishing deterministic semantic contracts that isolate Article 41 reasoning and operative awards, the benchmark closes causal loopholes prevalent in legal NLP. The Expanded ReAct experiment conclusively demonstrates that exposing claimed figures cuts error in half, validating that ECtHR-NPD genuinely tests judicial fact-to-award valuation.
- Greedy Quota Matching for Distribution Shift Diagnosis: Utilizing weighted Total Variation to construct matched In-Distribution and Out-of-Distribution test splits allows researchers to rigorously separate temporal baseline drift from inherent structural complexity (such as multi-applicant and multi-violation aggregation).
- A Reality Check on LLM Numeracy in Professional Domains: Demonstrates that scale and tool-augmented reasoning do not naturally solve numerical regression tasks. Standard token-level language modeling objectives treat numeric tokens identically to character tokens, lacking distance-sensitive metric loss landscapes necessary for monetary calibration.
Limitations & Future Work¶
- Jurisdictional and Language Scope: ECtHR-NPD is restricted to English-language judgments from the Strasbourg court and models case-level aggregate awards. It does not encompass French-language proceedings, domestic civil torts, pecuniary loss, legal cost apportionments, or individual victim-level allocations.
- Point-Metric Penalty on Legally Defensible Ranges: Because Article 41 just satisfaction is adjudicated on an equitable basis, a range of award amounts could be legally defensible on identical facts. Point-wise distance metrics (MAE, RMSE) penalize any deviation from the historical award, conflating legitimate judicial variation with predictive failure.
- Future Algorithmic Directions: Mitigating the twin hurdles of zero-inflation and extreme heavy tails requires dedicated numerical architectures, including two-stage hurdle networks (classifying zero versus positive awards prior to regression), quantile heads, and continuous distance-aware regression loss functions.
Related Work & Insights¶
- vs Traditional Legal Judgment Prediction (CAIL2018 / LexGLUE / ECHR-OD): Existing benchmarks focus almost exclusively on discrete multi-class or multi-label classification (charge prediction, article violation) or rule-constrained criminal sentencing; ECtHR-NPD pioneers continuous regression over unformulaic, discretionary monetary remedies.
- vs Statutory Numerical Legal Tasks (SARA): SARA evaluates deductive reasoning over formal tax statutes with deterministic arithmetic equations; ECtHR-NPD requires inductive equitable valuation from natural language narratives without explicit statutory formulas.
- vs LLM Numeracy Benchmarks: Generic mathematical benchmarks test algorithmic computation; ECtHR-NPD exposes the systemic failure of LLMs to map subjective real-world injury severity into calibrated monetary spaces.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering exploration of continuous, unformulaic discretionary judicial remedies; fills a major gap in legal NLP]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluates six model families, multiple prompt regimes, tool-augmented ReAct agents, and granular diagnostic views]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary integration of international human rights law doctrines and rigorous machine learning methodologies]
- Value: ⭐⭐⭐⭐⭐ [Provides an essential, highly challenging benchmark for numerical calibration, legal AI, and empirical legal studies]