Skip to content

JustifIT-QA: Evidence-Grounded Legal Question Answering in Italian

Conference: EMNLP 2026
Paper: ACL Anthology
Code: GitHub
Area: NLP Understanding
Keywords: Legal Question Answering, Evidence Extraction, Dense Retrieval, LLM-as-a-Judge, Self-Correction

TL;DR

Introduces JustifIT-QA, the first Italian statutory QA dataset featuring expert-authored answers and fine-grained legal evidence spans, alongside an end-to-end framework combining the domain-adapted dense retriever DARv2, a judge-driven conversational refinement loop, and token containment-based evidence extraction.

Background & Motivation

Legal question answering requires outputs that are not only linguistically fluent but strictly grounded in applicable statutory provisions. In critical legal domains such as civil liability, criminal sanctions, anti-money laundering regulations, and data privacy, even minor inaccuraciesโ€”such as misrepresenting deontic force, reversing burden-of-proof conditions, or citing non-existent exceptionsโ€”can lead to severe legal malpractice. However, general-purpose large language models remain prone to hallucinations, generating fabricated authorities or speculative arguments that detach from statutory texts. Even within standard retrieval-augmented generation (RAG) paradigms, models frequently produce answers that only partially adhere to the retrieved passages, making unconstrained generation inherently risky for professional legal counseling.

At the same time, grounded statutory resources outside the English language are severely limited. While prior statutory benchmarks such as JuriFindIT and BSARD pioneered article-level retrieval for civil law jurisdictions, they fundamentally lack expert-drafted reference answers and granular, span-level evidence annotations that isolate the precise legislative sentences supporting a given answer. Furthermore, conventional one-pass RAG pipelines lack explicit self-auditing mechanisms capable of catching legal discrepancies between the generated answer and the source texts. Human expert evaluation is too costly and unscalable for continuous auditing, while binary automated metrics fail to distinguish substantive legal distortions from harmless omissions of secondary procedural details.

To bridge this critical gap, this paper investigates grounded statutory QA in Italian across four heterogeneous normative domains. The authors construct a fully expert-annotated benchmark pairing natural-language legal inquiries with gold answers and sentence-level statutory excerpts, while engineering a closed-loop system that couples high-coverage retrieval with a judge-guided refinement loop and verbatim evidence extraction. Core idea: Introduce JustifIT-QA, the first Italian statutory QA dataset with expert-written answers and fine-grained evidence spans, and propose an end-to-end verifiable legal QA framework integrating the contrastively fine-tuned retriever DARv2, a rule-grounded LLM judge feedback loop, and token-containment evidence extraction.

Method

Overall Architecture

The JustifIT-QA framework addresses the three fundamental bottlenecks of statutory legal QA: incomplete article coverage during initial retrieval, unsupported or distorted claims during generation, and the absence of granular source-linked proof. The end-to-end pipeline operates in three coherent stages: first, given a natural language legal inquiry, a task-adapted bi-encoder retriever DARv2 coupled with a cross-encoder reranker retrieves and re-ranks the top legislative provisions to construct a clean statutory context. Next, a generator model drafts an initial answer, which is iteratively reviewed by an LLM-based judge checking adherence against five expert-defined legal criteria; if discrepancies are detected, conversational critiques guide the generator to self-correct across dialogue rounds. Finally, conditioned on the validated answer, an evidence extraction module pinpoints the exact verbatim statutory spans justifying the conclusions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Natural Language Legal Query q<br/>Statutory Corpus C"] --> B["Domain-Adapted Dense Retriever DARv2<br/>Bi-Encoder Retrieval + Qwen3-4B Reranking"]
    B --> C["Rule-Grounded Judge Feedback Loop<br/>5 Legal Criteria + Multi-Turn Self-Correction"]
    C --> D["Token Containment-Based Evidence Extraction<br/>Extract Verbatim Spans + Source-Linked Audit"]
    D --> E["Verified Legal Answer and Evidence Spans<br/>Grounded Answer y* + Extracted Spans S"]

Key Designs

1. Domain-Adapted Dense Retriever DARv2: Mitigating Cross-Article Confusions with Synthetic Supervision

Standard dense retrievers evaluated in zero-shot settings struggle with statutory legal text due to heavy syntactic subordination, dense legal terminology, and subtle distinctions between related articles. To maximize recall over candidate provisions, DARv2 is constructed by fine-tuning jina-embeddings-v5-text-nano (239M parameters, supporting 8,192 input tokens) using contrastive learning. The training dataset enriches the human-authored queries with professionally styled synthetic questions generated by Qwen3.5-27B for each statutory article. These synthetic questions are explicitly optimized to have higher noun density, longer sentences, and reduced syntactic subordination, closely mirroring the factual patterns of real-world legal queries. The contrastive loss is optimized with in-batch negatives:

\[ \mathcal{L}(q_i, a_i^p, \mathcal{N}(q_i)) = -\log \frac{\exp(s_\theta(q_i, a_i^p) / \tau)}{\sum_{a \in \{a_i^p\} \cup \mathcal{N}(q_i)} \exp(s_\theta(q_i, a) / \tau)} \]

where \(s_\theta(q, a) = \cos(\text{Enc}_\theta(q), \text{Enc}_\theta(a))\). On the JuriFindIT validation set, DARv2 achieves 85.88% Recall@20 (a statistically significant \(+2.71\%\) absolute gain over its predecessor DAR at \(83.16\%\)) and 93.79% Recall@100, providing an expansive candidate pool that is subsequently refined by Qwen3-Reranker-4B down to the top-5 authoritative provisions.

2. Rule-Grounded Judge Feedback Loop: Conversational Self-Correction under Expert Compliance Criteria

To eliminate legal hallucinations and prevent the model from drifting outside the retrieved articles, the framework formalizes an iterative verification loop mediated by an LLM judge. The judge model evaluates the draft answer \(\hat{y}\) exclusively against the retrieved context \(\tilde{F}\) based on five expert-crafted legal principles: factual coverage, absence of contradiction, preservation of normative force and precision (e.g., distinguishing "must" from "may"), preservation of subject/temporal scope, and faithful synthesis without external speculation. Instead of a simplistic binary verdict, the judge outputs an ordinal compliance label: NON-COMPLIANT, PARTIALLY-COMPLIANT, or FULLY-COMPLIANT. When deviations occur, the judge produces a targeted critique \(f^{(t)}\), which is appended to the generator's conversation history:

\[ \hat{y}^{(t+1)} = G\left(H^{(t)} \oplus f^{(t)}\right) \]

The loop halts upon reaching FULLY-COMPLIANT or capping at \(T=2\) iterations. Expert blind evaluations demonstrate that for questions where relevant context is retrieved, this feedback loop boosts expert-rated correctness on core legal content from 76.74% to 83.72% and side details from 69.77% to 74.42%, actively curbing unwarranted overgeneralizations and missing statutory conditions.

3. Token Containment-Based Evidence Extraction: Granular Legal Justification without Boundary Penalties

Providing an answer without precise textual citations leaves human verifiers with the burden of reading entire statutory articles. JustifIT-QA introduces an evidence extraction layer \(E(q, \tilde{F}, y^*) \to \hat{S}\) that extracts structured verbatim pairs \(\hat{s}_i = (\text{id}(a), \hat{\tau})\) directly supporting the chosen answer \(y^*\). To accommodate legitimate minor boundary discrepancies (such as punctuation or introductory article numbering), the authors establish an automated token containment criterion:

\[ \text{GoldCont}(\hat{\tau}, \tau) = \frac{|\text{Tok}(\hat{\tau}) \cap \text{Tok}(\tau)|}{|\text{Tok}(\tau)|} \]

A candidate excerpt \(\hat{\tau}\) is counted as a true positive if \(\text{GoldCont}(\hat{\tau}, \tau) \ge 0.95\), reinforced with one-to-one bipartite matching to prevent duplicate claims over the same reference span. Macro-averaged results reveal that larger instruction-tuned models like Qwen3-32B achieve the highest extraction F1 of 42.55% (53.08% Recall, 41.07% Precision), establishing a transparent audit trail for downstream legal compliance.

A Worked Example

Consider a query from the Italian Civil Code: "Mario takes out a life insurance policy on Lucia without having the authority to do so. Can Lucia ratify the policy after learning of it only after a claim has occurred?" 1. Retrieval: DARv2 retrieves provisions related to unauthorized representation and insurance; Qwen3-Reranker-4B places Article 1891 of the Italian Civil Code into the top-5 context. 2. Initial Generation: Generator drafts: "Yes, Lucia can ratify the insurance policy at any time, and Mario is freed from all legal duties." 3. Judge Critique: The judge reviews Article 1891 and detects an unsupported overgeneralization, labeling it PARTIALLY-COMPLIANT with feedback: "Article 1891 allows ratification after the loss has occurred, but explicitly dictates that Mario remains personally liable for all obligations until the insurer receives notice of ratification or rejection. Correct the claim regarding Mario's immediate release." 4. Conversational Revision: The generator revises: "Yes, Lucia may ratify the policy even after the loss has occurred. However, until the insurer receives notice of ratification or rejection, Mario remains personally liable for the obligations arising from the contract." 5. Evidence Extraction: The extractor locates Article 21095, character offsets [0, 537], extracting the exact statutory sentence into \(\hat{S}\), delivering a fully grounded, transparent legal response.

Loss & Training

DARv2 was trained using SentenceTransformers and PyTorch on a single NVIDIA A100 (80GB) GPU. The optimization protocol employed the AdamW optimizer with a learning rate of \(3.5 \times 10^{-5}\) under linear schedule decay, weight decay of \(10^{-2}\), a batch size of 768, and 35 epochs. In-batch negative sampling provided 767 challenging negative articles per query. For generation, judging, and extraction, open-weight models (Qwen3, Qwen3.5, GPT-oss, and Gemma 4) were deployed using their official inference configurations (e.g., Qwen3 in no-thinking mode with temperature \(\tau=0.7, \text{top-p}=0.80, \text{top-k}=20\)).

Key Experimental Results

Main Results

Table 1 summarizes first-stage retrieval performance on the JuriFindIT validation pool, contrasting zero-shot generalist encoders with domain-adapted bi-encoders.

Retriever Model Params Mode Recall@5 Recall@20 Recall@100 nDCG@5 MRR@5 MAP@5
granite-311m-multilingual-r2 311M Zero-shot 54.98 64.80 72.49 47.57 47.30 43.47
harrier-oss-v1-270m 270M Zero-shot 55.88 67.19 74.39 50.87 51.79 47.01
snowflake-arctic-embed-m-v2.0 305M Zero-shot 63.56 71.90 79.62 58.09 59.17 53.49
pplx-embed-v1-4B 4B Zero-shot (Best) 74.42 81.41 88.63 72.95 76.34 68.64
DAR (Previous SOTA) 305M Domain Fine-tuned 76.11 83.16 91.18 73.29 76.50 68.56
DARv2 (Ours) 239M Domain Fine-tuned 77.38 85.88 93.79 74.94 76.88 70.09

Table 2 evaluates various LLMs acting as reference-based judges against expert human correctness labels on main content.

Judge Model Mode Precision (%) Recall (%) F1 (%)
Qwen3-32B No-thinking 89.57 95.42 92.40
Qwen3.5-27B No-thinking 93.89 80.39 86.62
Qwen3.5-27B Thinking 94.57 79.74 86.52
Qwen3.6-27B No-thinking 95.38 81.05 87.63
GPT-oss 20b Thinking Med 92.37 79.08 85.21
GPT-oss 20b Thinking Low 92.65 82.35 87.20
Gemma 4 31B No-thinking 95.71 87.58 91.47
Gemma 4 26B-A4B No-thinking 93.20 89.54 91.33
Qwen3.6 35B-3A No-thinking 93.64 67.32 78.33

Ablation Study

Table 3 presents controlled fine-tuning ablations varying the encoder backbone, synthetic training questions, and model capacity evaluated at \(k=5\).

Encoder Backbone Training Supervision Source Recall@5 (%) nDCG@5 (%) MRR@5 (%) MAP@5 (%)
jina-embeddings-v5-text-nano Original real queries only 70.52 66.49 69.16 62.02
snowflake-arctic-embed-m-v2.0 Original real queries only 71.08 65.29 65.99 60.31
harrier-oss-v1-270m JuriFindIT synth + real 67.41 64.00 67.63 59.29
harrier-oss-v1-270m JustifIT-QA synth + real 70.27 66.39 68.83 61.34
DAR (snowflake-arctic-embed-m) JuriFindIT synth + real 76.11 73.29 76.50 68.56
snowflake-arctic-embed-m-v2.0 JustifIT-QA synth + real 76.42 73.07 76.00 68.04
jina-embeddings-v5-text-nano JuriFindIT synth + real 75.33 73.81 76.56 69.41
DARv2 (jina-nano) JustifIT-QA synth + real 77.38 74.94 76.88 70.09
jina-embeddings-v5-text-small JustifIT-QA synth + real 74.63 72.88 75.25 68.67

Table 4 highlights the effect of the judge-based feedback loop on expert-rated answer correctness on queries with non-zero retrieval recall.

Evaluation Dimension Initial Answer (No Feedback) (%) Judge-Refined Answer (%) Absolute Gain (%)
Main Legal Correctness 76.74 83.72 +6.98
Side Detail Correctness 69.77 74.42 +4.65

Key Findings

  • Upstream retrieval recall is the primary bottleneck for answer factuality: Errors in answers judged incorrect by human experts correlate strongly with low statutory recall (\(\sim 0.50\) vs \(\sim 0.87\) for correct answers). When retrieval recall is zero, feedback loops are counterproductive, as the model hallucinates modifications without authoritative backing.
  • Syntactic profile of synthetic queries governs dense representation quality: The JustifIT-QA synthetic questions generate substantial improvements across backbones by matching human-authored queries in length, increasing noun representation, and removing distractive syntactic subordination.
  • LLM judges exhibit distinct safety-versus-coverage profiles: Qwen3-32B provides the highest recall (95.42%) and F1 (92.40%), making it ideal for maintaining valid legal answers. In contrast, Gemma 4 31B and Qwen3.6-27B achieve \(>95\%\) precision, serving as stringent filters against subtle compliance violations.
  • Evidence extraction remains challenging under verbatim containment: Even top-performing models like Qwen3-32B achieve an extraction F1 of 42.55% (41.07% Precision, 53.08% Recall), showing that models tend to over-extract broad context rather than isolating exact minimal normative clauses.

Highlights & Insights

  • Backward annotation workflow prevents anchoring: Legal annotators drafted answers before identifying supporting article spans, preventing topical word-overlap bias and ensuring spans represent genuine legal grounds.
  • Conversational self-refinement avoids heavy RL training: Treating judge critiques as conversational turns within dialogue history allows immediate, prompt-based self-correction without complex reward modeling or RL fine-tuning.
  • Multi-domain coverage illuminates structural divergence in statutory evidence: The empirical statistics expose stark differences between compact civil/criminal codes (\(\sim 1.07-1.69\) spans/query) versus highly fragmented, heavy-tailed privacy and anti-money laundering regulations (\(\sim 24.8-32.8\) spans and thousands of words per query).

Limitations & Future Work

  • Context is capped at the top-5 retrieved articles due to computational and memory constraints, leading to missing provisions in complex multi-statute domains like Privacy and AML/CTF.
  • The multi-turn feedback loop increases latency compared to standard single-pass RAG, presenting practical challenges for high-throughput real-time deployment.
  • The token containment metric (\(\text{GoldCont} \ge 0.95\)), while more resilient than exact string matching, is still surface-oriented and does not capture deeper paraphrastic or semantic equivalence in legal reasoning.
  • vs JuriFindIT / BSARD: Prior datasets provided only coarse article-level retrieval mappings; JustifIT-QA expands JuriFindIT with 895 expert answers and paragraph/clause-level evidence spans.
  • vs LLeQA / JEC-QA: LLeQA focuses on French administrative documents and JEC-QA on Chinese bar exam multiple-choice questions; JustifIT-QA targets open-ended statutory QA across 4 major Italian legal domains.
  • vs Standard Self-Refine / Reflexion: Unlike general self-refinement frameworks that evaluate outputs without authoritative grounds, JustifIT-QA binds the critique directly to retrieved legislative texts under strict statutory compliance rubrics.

Rating

  • Novelty: โญโญโญโญโญ First Italian statutory QA dataset pairing expert answers with fine-grained span annotations and an evidence-grounded judge feedback loop.
  • Experimental Thoroughness: โญโญโญโญโญ Includes paired bootstrap significance testing, retriever ablations, multi-model judge comparisons, and human inter-annotator agreement analysis.
  • Writing Quality: โญโญโญโญโญ Rigorous mathematical formalization, extensive linguistic profiling (Profiling-UD), and transparent reporting of negative results.
  • Value: โญโญโญโญโญ A foundational benchmark and framework for grounded legal QA, explainable AI, and trustworthy statutory NLP.