Multi-Source Retrieval and Reasoning for Legal Sentencing Prediction¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cjj826/MSR2
Area: Information Retrieval
Keywords: Legal Sentencing Prediction, Multi-Source Retrieval, Reinforcement Learning, Process-Level Reward, Large Language Model Reasoning
TL;DR¶
Addressing the dual bottlenecks of fragmented objective regulatory criteria and vulnerable discretionary subjective reasoning in legal sentencing prediction, this paper proposes MSR2โa reinforcement learning framework that seamlessly interleaves multi-source legal retrieval into the LLM action space and guides intermediate sentencing factor extraction via a rubric-based process-level reward under GRPO, setting new state-of-the-art accuracy and interpretability benchmarks on CAIL2018 and CJO22.
Background & Motivation¶
Legal Judgment Prediction (LJP) represents a foundational milestone in judicial artificial intelligence, aiming to infer formal legal outcomes directly from textual descriptions of case facts. In general, LJP comprises three tightly linked subtasks: law article prediction, charge prediction, and legal sentencing prediction (LSP). While modern semantic representation learning and task-dependency logic frameworks have achieved considerable maturity in classifying articles and chargesโeven serving as operational aids in automated court docket systemsโsentencing prediction remains an exceptionally steep hurdle. Both traditional specialized neural classifiers and contemporary large language models prompted with standard chain-of-thought fail to produce dependable, legally sound sentencing determinations.
A forensic analysis of this persistent failure reveals that sentencing decision-making is fundamentally characterized by an acute objective-subjective duality. From the objective perspective, determining an accurate prison term hinges upon fine-grained regulatory metrics, such as the exact currency thresholds demarcating large, huge, and especially huge amounts in financial crimes, or specific statutory criteria defining egregious circumstances. Crucially, these quantitative standards are rarely articulated within the high-level statutes themselves; rather, they are scattered across regional judicial interpretations, provincial high court implementing guidelines, and academic treatises. Existing retrieval-augmented methods predominantly index statutory clauses or top-k precedent judgments, leaving this fragmented regulatory corpus untapped and leading to severe knowledge gaps. From the subjective perspective, criminal sentencing requires evaluating highly nuanced discretionary circumstances, such as defendant state of mind, intent versus negligence, confession, remorse, and victim restitution. Prior logic-based paradigms rely on rigid first-order formalisms or unconstrained LLM zero-shot generation, supervised solely by sparse binary outcome correctness. Consequently, models frequently learn spurious shortcutsโgenerating an ostensibly correct sentencing interval via coincidental keyword matching while executing completely flawed legal reasoning.
To overcome this core tension between fragmented objective knowledge requirements and fragile subjective discretionary reasoning, an effective sentencing system must dynamically interact with multi-source legal repositories and receive dense process-level supervision during trajectory rollout. Core idea: Introduce MSR2, a framework that integrates multi-source legal retrieval directly into the policy LLM's action space and enforces rubric-guided process-level rewards on intermediate sentencing-factor extraction, optimized end-to-end via Group Relative Policy Optimization (GRPO).
Method¶
Overall Architecture¶
Departing from conventional LJP formulations that reduce sentencing to crude multi-class classification over predetermined intervals, MSR2 operates directly over continuous sentence values \(\hat{y}\) before projecting them into standardized judicial bins, thereby preserving fine-grained discretionary gradations. The framework is composed of three interacting engines: an autoregressive policy LLM \(\pi_\theta\), a multi-source retrieval environment managing four distinct legal corpora \(\mathcal{S} = \{S_1, S_2, \dots, S_n\}\), and a rubric-based LLM-as-a-judge process reward module.
During inference rollout, the model executes reasoning following the judicial syllogism standard established by high-level judicial guidelines. Given a fact narrative \(x\), the policy LLM writes an initial crime analysis in <reasoning>, systematically extracts objective and subjective sentencing circumstances in <factors>, emits a <search> request with an explicit target source tag when knowledge is missing, incorporates the retrieved evidence injected into <information>, and concludes with the exact sentence in <answer>. During training, the extracted factor set is scored against a rigorous judicial rubric by the judge model to yield a process reward \(P_r\), which is linearly combined with the interval-level outcome reward \(O_r\) to optimize policy parameters via GRPO with retrieved-token masking.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Case Fact Description"] --> B["Action-Space Routed Multi-Source Retrieval"]
B --> C["Process-Level Reward on Sentencing Factors"]
C --> D["Retrieved-Token Masked Group Relative Policy Optimization"]
D --> E["Output Continuous Sentence & Mapped Interval"]
Key Designs¶
1. Action-Space Routed Multi-Source Retrieval: Dispersed regulatory criteria -> In-action target tagging -> Dynamic routing across specialized legal corpora
Prior legal retrieval augmentation typically targets a monolithic corpus of statutory articles or past judgments, failing to supply the fine-grained local sentencing guidelines and doctrinal interpretations essential for calculating prison terms. To resolve this objective information deficit, MSR2 constructs four dedicated knowledge sources: a statutory database containing 51,784 legal articles (indexed via 1,024-dimensional dense representations using Qwen3-Embedding and FAISS HNSW); an academic library of ~1,200 legal books partitioned into 780,035 passages (searched via hybrid BM25 and dense retrieval); a precedent corpus encompassing 10,714,118 official criminal rulings (indexed via BM25 over charge names and legal phraseology); and a specialized sentencing guideline corpus comprising 200 provincial-level implementing documents (queried via full-text BM25). Rather than relying on separate neural gating dispatchers, MSR2 treats source selection as an explicit action within the LLM generation space. The model outputs a search query \(q_t\) along with a source tag \(s_t \in \{\text{statute}, \text{guideline}, \text{precedent}, \text{book}\}\) inside <search><source> query </source></search>, directly dispatching the request to retriever \(R_{s_t}\) to fetch the top-\(k\) evidence blocks:
This zero-overhead action-space design tightly couples knowledge acquisition with legal deduction, empowering the model to proactively query provincial thresholds or doctrinal exceptions precisely when knowledge deficits arise.
2. Process-Level Reward on Sentencing Factors: Sparse outcome rewards tolerate reasoning hallucinations -> Rubric-guided LLM-as-a-judge scoring -> Grounding subjective discretion in case facts
Training language models exclusively with binary outcome rewards \(O_r \in \{0, 1\}\) over final sentencing intervals incentivizes reward-hacking: the policy may arrive at the ground-truth interval through spurious lexical cues while severely misinterpreting mitigating circumstances. To enforce genuine judicial deliberation, MSR2 introduces a process-level reward targeting intermediate sentencing-factor identification. The policy is mandated to generate a distinct <factors> span explicitly enumerating both objective factors (monetary amounts, repeat offenses, operational modality) and subjective factors (plea confession, victim forgiveness, financial restitution, subjective malice). A dedicated judge model (Qwen3-32B guided by a detailed judicial rubric) evaluates the extracted factor set \(\mathcal{F} = \{f_1, \dots, f_m\}\) across four core dimensions: factual support, legal relevance, factor coverage, and reasoning credibility. The rubric assigns an integer score from 1 to 10, triggering severe point deductions (\(-3\) to \(-5\)) for hallucinated or conflicting factors, normalized into \(P_r \in [0, 1]\). The overall trajectory reward balances process fidelity with terminal accuracy:
Empirically parameterized with \(\lambda_r = 0.2\), this process reward ensures the model develops rigorous, case-faithful discretionary reasoning habits rather than ungrounded guesswork.
3. Retrieved-Token Masked Group Relative Policy Optimization: External knowledge injection contaminates gradients -> Selective token masking -> Robust end-to-end policy exploration
MSR2 leverages Group Relative Policy Optimization (GRPO) to eliminate the substantial GPU memory overhead of a dedicated value network. For each case description \(x\), the policy samples a cohort of \(G\) trajectories \(\mathcal{Y} = \{y_1, \dots, y_G\}\) from \(\pi_{\text{old}}\), computing the cohort reward mean \(\bar{R} = \frac{1}{G}\sum_{i=1}^G R_i\) and standard deviation \(\sigma_R\) to obtain the standardized advantage \(\tilde{A}_i = \frac{R_i - \bar{R}}{\sigma_R + \epsilon}\). Because dynamic multi-turn retrieval injects lengthy external documents into <information> blocks within the trajectory, standard policy gradient estimation would compute log-likelihood ratios over non-generated statutory tokens, injecting catastrophic noise into parameter updates. MSR2 solves this by introducing a retrieval-span mask indicator \(I(y_{i,t}) \in \{0, 1\}\) (\(1\) for model-generated tokens, \(0\) for retrieved tokens) and normalizing advantages by the count of active tokens \(Z_i = \sum_{t=1}^{|y_i|} I(y_{i,t})\):
where \(r_{i,t} = \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t}; \mathcal{R})}{\pi_{\text{old}}(y_{i,t} \mid x, y_{i,<t}; \mathcal{R})}\), and the KL divergence penalty is identically masked. This ensures policy gradients reflect solely the quality of the model's retrieval decisions, factor assessments, and final deductions.
Multi-Source Retrieval and Sentencing Reasoning Mechanism¶
To illustrate how MSR2's multi-source retrieval and intermediate reasoning interact in practice, consider a representative fraud case from Hunan Province (detailed in Figure 4):
1. Fact Digestion & Factor Extraction: Defendant Deng falsely claimed ownership of over 1,000 tons of timber, defrauding victim Shi of RMB 50,000 as a deposit. Following arrest, Deng truthfully confessed, repaid the financial loss, and obtained Shi's formal forgiveness. MSR2 records the initial facts and extracts in <factors>: [Amount: RMB 50,000, Truthful confession, Guilty plea, Restitution and victim forgiveness].
2. Iterative Multi-Source Retrieval:
- The model issues <search><statute> Find the statutory provision for fraud </statute></search>, receiving Article 266 of the Criminal Law: large amount carries up to 3 years; huge amount carries 3 to 10 years.
- Recognizing a critical knowledge gap, MSR2 reasons in <reasoning>: Article 266 does not define whether RMB 50,000 constitutes a large or huge amount in Hunan.
- The policy issues <search><guideline> Find fraud sentencing guidelines/standards </guideline></search>, retrieving Hunan High Court standards confirming that RMB 50,000 meets the threshold for "huge amount," anchoring the statutory baseline to 3โ10 years.
- To calibrate discretionary leniency, it queries <search><guideline> Find how forgiveness adjusts the baseline </guideline></search>, confirming that restitution with forgiveness permits reducing the baseline by up to 40%.
3. Discretionary Synthesis & Adjudication: Synthesizing the mitigating circumstances with the 3โ10 year base bracket, MSR2 determines that the sentence should land at the statutory minimum, outputting <answer> 3 years </answer> (36 months), exactly matching the human judge's ground truth. In contrast, BERT predicts a generic "one to two years," and PLJP misclassifies RMB 50,000 as a "large amount" due to missing local guidelines, erroneously yielding "one to two years."
Key Experimental Results¶
Main Results¶
Experiments were conducted on the benchmark CAIL2018 dataset (82,138 carefully filtered cases with single-charge, single-article criteria) and the out-of-distribution real-world CJO22 test set (1,698 complex judicial documents). Models are evaluated across four standard classification metrics: Accuracy (Acc), Macro-Precision (Ma-P), Macro-Recall (Ma-R), and Macro-F1 (Ma-F).
| Model Paradigm | Baseline Model | CJO22 Acc | CJO22 Ma-P | CJO22 Ma-R | CJO22 Ma-F | CAIL2018 Acc | CAIL2018 Ma-P | CAIL2018 Ma-R | CAIL2018 Ma-F |
|---|---|---|---|---|---|---|---|---|---|
| Classification | TFIDF+SVM (1995) | 33.16 | 23.17 | 19.70 | 20.01 | 42.05 | 28.56 | 26.42 | 26.66 |
| CNN (1989) | 34.28 | 15.34 | 16.47 | 13.79 | 39.39 | 19.38 | 17.51 | 16.08 | |
| BERT (2019) | 33.57 | 19.86 | 17.88 | 16.47 | 42.34 | 23.78 | 23.51 | 22.64 | |
| RoBERTa (2019) | 35.34 | 23.11 | 20.24 | 19.27 | 42.12 | 28.90 | 24.14 | 23.08 | |
| TopJudge (2018) | 27.14 | 19.76 | 17.69 | 17.94 | 35.70 | 18.59 | 18.43 | 17.63 | |
| LADAN (2020) | 33.69 | 26.40 | 22.94 | 24.55 | 36.14 | 20.86 | 18.03 | 16.58 | |
| NeurJudge (2021) | 26.80 | 26.81 | 26.85 | 25.97 | 36.84 | 20.99 | 16.81 | 18.51 | |
| Direct Inference | Llama3-8B-DI | 12.96 | 16.63 | 12.45 | 8.69 | 12.54 | 14.91 | 13.12 | 8.60 |
| Qwen3-8B-DI | 15.14 | 35.81 | 15.40 | 11.28 | 11.38 | 31.92 | 14.04 | 8.72 | |
| Qwen3-14B-DI | 14.43 | 36.78 | 18.14 | 12.23 | 12.82 | 26.41 | 16.35 | 10.92 | |
| Qwen3-32B-DI | 28.45 | 27.82 | 22.67 | 22.06 | 26.49 | 27.68 | 27.28 | 23.43 | |
| Qwen3-235B-A22B-DI | 37.28 | 34.81 | 26.35 | 27.29 | 38.05 | 29.36 | 28.55 | 26.82 | |
| Direct Reasoning | Qwen3-8B-DR | 22.67 | 22.91 | 22.87 | 18.87 | 20.57 | 20.43 | 21.07 | 16.73 |
| Qwen3-14B-DR | 27.80 | 24.18 | 26.51 | 22.42 | 25.58 | 19.35 | 22.44 | 18.03 | |
| Qwen3-32B-DR | 30.09 | 26.20 | 27.96 | 24.53 | 27.69 | 23.21 | 26.57 | 22.22 | |
| RAG | PLJP-GPT-4o-mini | 27.92 | 24.88 | 20.90 | 18.63 | 27.53 | 23.65 | 19.56 | 18.03 |
| Qwen3-8B-RAG | 29.03 | 22.75 | 24.26 | 20.95 | 28.16 | 20.54 | 21.56 | 18.70 | |
| Qwen3-14B-RAG | 30.57 | 27.07 | 24.05 | 23.81 | 29.68 | 22.82 | 21.00 | 19.57 | |
| Supervised Fine-Tuning | Llama3-8B-FT | 31.68 | 32.05 | 19.48 | 17.26 | 37.15 | 26.83 | 21.37 | 19.38 |
| Qwen3-8B-FT | 37.63 | 36.45 | 21.85 | 19.87 | 41.84 | 29.57 | 23.89 | 22.31 | |
| Reinforcement Learning | MSR2-Qwen3-8B (Ours) | 44.29 | 37.43 | 30.73 | 28.07 | 46.18 | 36.74 | 31.70 | 28.84 |
Ablation Study¶
The table below details the performance trajectories across training checkpoints on CJO22 when disabling the process reward (w/o PR, \(\lambda_r = 0\)) or restricting retrieval to statutory articles only (w/o MR):
| Training Checkpoint | Variant | CJO22 Acc (%) | CJO22 Ma-P (%) | CJO22 Ma-R (%) | CJO22 Ma-F (%) | Analysis & Observations |
|---|---|---|---|---|---|---|
| Step 0 | MSR2 | 29.03 | 22.75 | 24.26 | 20.95 | Initial policy prior to RL exploration |
| w/o PR | 29.03 | 22.75 | 24.26 | 20.95 | Identical baseline at initialization | |
| w/o MR | 28.09 | 24.50 | 22.94 | 21.77 | Slightly handicapped without multi-source access | |
| Step 40 | MSR2 | 35.92 | 32.61 | 28.12 | 25.07 | Rapid early gains from structured guidance |
| w/o PR | 33.04 | 24.59 | 23.11 | 22.47 | Absence of PR drops accuracy by 2.88% | |
| w/o MR | 32.69 | 23.92 | 22.80 | 21.85 | Single-source restriction drops accuracy by 3.23% | |
| Step 80 | MSR2 | 35.57 | 26.16 | 29.08 | 26.12 | Balanced progression across minority bins |
| w/o PR | 34.51 | 25.13 | 23.78 | 23.33 | Pronounced gap in Macro-Recall (-5.30%) | |
| w/o MR | 34.39 | 24.87 | 24.78 | 23.14 | Macro-F1 trails full model by ~3 points | |
| Step 100 | MSR2 | 37.40 | 32.12 | 29.26 | 26.02 | Sustained trajectory divergence |
| w/o PR | 33.51 | 25.33 | 25.46 | 23.59 | Accuracy gap widens to 3.89% | |
| w/o MR | 35.10 | 26.50 | 25.49 | 24.07 | Guideline deficit impedes boundary convergence | |
| Step 120 | MSR2 | 38.40 | 30.54 | 28.10 | 24.96 | Peak plateau performance reached |
| w/o PR | 35.51 | 25.15 | 25.12 | 23.25 | Drops 2.89% without process feedback | |
| w/o MR | 34.22 | 25.17 | 24.68 | 22.96 | Drops 4.18%, proving MR is critical late-stage | |
| Step 140 | MSR2 | 38.40 | 30.24 | 28.31 | 24.44 | Final checkpoint convergence |
| w/o PR | 36.22 | 25.70 | 24.31 | 22.62 | Lacks fine-grained factor constraint | |
| w/o MR | 35.10 | 27.44 | 22.89 | 22.90 | Accuracy trails by 3.30% |
In addition, an agreement study on 200 randomly sampled cases with three law-school legal experts revealed that Qwen3-32B achieves Pearson \(r = 0.587\), Spearman \(\rho = 0.554\), and Kendall \(\tau = 0.512\) against aggregated human ratings (\(p < 0.01\)), closely approaching inter-human agreement (\(r = 0.637\)) and substantially outperforming DeepSeek-V3-0324 (\(r = 0.451\)) and GPT-4o (\(r = 0.324\)).
Key Findings¶
- Staged Complementarity Between PR and MR: Process rewards provide vital gradient direction during early training (Steps 20โ60), rapidly stabilizing legal factor extraction; multi-source retrieval becomes the primary performance differentiator in later phases (Steps 100โ140), where lacking provincial guidelines caps accuracy below 35% even with extended exploration.
- RL Beats Extreme Scale Expansion: Operating on an 8B backbone, MSR2 achieves 44.29% accuracy on CJO22, significantly outperforming zero-shot reasoning on the 235B Qwen3 MoE (37.28%) and supervised fine-tuning on 8B (37.63%). Tailoring the reasoning action space with RL offers far superior cost efficiency over raw scale.
- Robust Cross-Distribution Resiliency: When evaluated on CJO22, traditional classifiers and SFT models suffer dramatic accuracy declines (BERT drops from 42.34% to 33.57%), while MSR2 experiences only a minor drop (46.18% on CAIL2018 to 44.29% on CJO22), demonstrating that grounded multi-source retrieval prevents superficial pattern memorization.
Highlights & Insights¶
- In-Action Retrieval Routing: By allowing the policy LLM to emit the target repository tag as a native action, MSR2 avoids complex auxiliary gating networks and enables the policy to learn when and where to query legal knowledge end-to-end.
- Retrieved-Token Masking in GRPO: Selectively zeroing the gradient contribution of injected external context ensures that policy updates reward only the model's active reasoning and retrieval decisions, preventing gradient contamination from non-generated tokens.
- Continuous Sentencing Value Modeling: Formulating sentencing as continuous prediction before discretizing into judicial evaluation intervals mirrors real-world legal reasoning and provides smooth reward signals for RL optimization.
Limitations & Future Work¶
- Exploration Restricted to 8B Scale: Due to RL compute overheads across multi-turn trajectories, evaluations were confined to Qwen3-8B; scaling MSR2 to 32B or 70B architectures remains an unverified opportunity.
- Jurisdictional Scope: The current multi-source framework is tailored to the civil law statutory framework of mainland China; adapting it to common-law jurisdictions with extensive case-law precedent networks requires further research.
- Fixed Multi-Turn Budget: The rollout employs a hard cap of 8 interaction turns, which may prove insufficient for highly intricate corporate fraud or multi-defendant racketeering litigations.
Related Work & Insights¶
- vs PLJP (Wu et al., 2023): PLJP employs a separate neural retriever for precedents and uses LLMs solely for context-conditioned generation without process-level RL; MSR2 incorporates dynamic multi-source guideline routing and GRPO, boosting CJO22 accuracy by 16.37% (44.29% vs 27.92%).
- vs NeurJudge (Yue et al., 2021) / LADAN (Xu et al., 2020): These earlier neural architectures encode static topological dependencies between subtasks or confusable law articles; MSR2 replaces rigid topological constraints with dynamic, explainable reasoning traces backed by on-demand external evidence.
Rating¶
- Novelty: โญโญโญโญโญ [Pioneering integration of action-space multi-source legal routing and factor-level process rewards within GRPO]
- Experimental Thoroughness: โญโญโญโญโญ [Extensive cross-dataset evaluations, full checkpoint ablations, and legal expert agreement audits]
- Writing Quality: โญโญโญโญโญ [Exceptionally cohesive formulation bridging algorithmic design and judicial procedural reality]
- Value: โญโญโญโญโญ [Provides a practical, auditable template for deploying trustworthy AI in high-stakes legal adjudication]