Beyond Static Grounding: Behavioral Contracts for Trustworthy Retrieval-Augmented Generation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Retrieval-Augmented Generation, Dynamic Evidence Contracts, Behavioral Compliance, Knowledge Conflicts, Intervention Invariance
TL;DR¶
Addressing the blind spot of static RAG evaluations that operate on fixed contexts and cannot detect whether systems causally depend on retrieved passages, this work introduces Dynamic Evidence Contracts with the Intervention Compliance Score (ICS) and presents DOVE-RAG, a single-call reference pipeline using NLI-based verification and deterministic three-tiered reorganization to achieve explicit, structured evidence adaptation.
Background & Motivation¶
Retrieval-Augmented Generation (RAG) grounds language model generations in externally retrieved passages, serving as a primary defense against hallucination and enabling verifiable outputs across enterprise deployments. However, existing RAG evaluation benchmarks—including RAGAS, ARES, and ALCE citation accuracy metrics—are fundamentally static: they measure answer quality and citation correctness only when evidence is held fixed. This operational assumption conceals a critical failure mode: static grounding verifies whether a claim matches the retrieved passages, but cannot determine whether the claim causally derives from those passages or merely coincides with memorized parametric knowledge. A system that produces accurate answers on the initial context but ignores deletions, overlooks conflicts, or silently defaults to pre-trained weights when context shifts is evidence-ignorant, yet passes every conventional static metric.
In practical production environments, retrieval context is inherently dynamic: databases undergo incremental updates, documents are revised, and retrieval rankings drift. A trustworthy RAG system must demonstrate genuine evidence responsiveness: explicitly acknowledging information insufficiency when supporting passages are removed, reporting factual contradictions when conflicting evidence emerges, and preserving output stability when irrelevant documents are injected or passage order is permuted. In reality, existing open-source and commercial pipelines exhibit severe behavioral pathologies under evidence interventions: when supporting evidence vanishes, models fall back to parametric knowledge without signaling uncertainty; when conflicting evidence is supplied, models arbitrarily pick one side or ignore the dispute; and citation markers often function as post-hoc annotations rather than faithful causal traces.
To bridge the gap between static surface grounding and counterfactual evidence dependence, this work draws on software contract testing to formulate Dynamic Evidence Contracts. The framework specifies explicit behavioral expectations across four canonical evidence transformations, establishing intervention invariance as a foundational desideratum for trustworthy RAG. Core idea: formalize RAG trustworthiness as a dual requirement of static grounding and dynamic evidence responsiveness across deletion, conflict, distraction, and permutation interventions; operationalize this via the Intervention Compliance Score (ICS); and implement DOVE-RAG, a single-LLM-call reference system that combines sentence-level NLI verification with deterministic three-tiered output restructuring to convert latent verification signals into explicit, auditable user-facing behavior.
Method¶
Overall Architecture¶
DOVE-RAG implements evidence contracts under a strict "separation of concerns" philosophy: generation produces raw content, verification independently checks compliance, and structured reorganization enforces the contract. The entire pipeline executes with exactly one LLM generation call, avoiding the excessive latency and iterative divergence common in multi-agent or self-reflection paradigms. Given a user query \(q\) and top-\(k\) retrieved passages \(P = \{p_1, \dots, p_k\}\), the generator first emits a contract-aware draft containing citation anchors; a batched natural language inference (NLI) model then checks every claim against every passage to construct a contract graph; finally, a deterministic string reorganization engine partitions the draft into a verified core tier, an uncertain claims tier, and an explicit conflict report tier without invoking additional language models.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Query q and Retrieved Passages P"] --> B["Contract-Aware Draft Generation<br/>Single-call prompt enforcing citations & conflict awareness"]
B --> C["Sentence-Level NLI Verification & Contract Graph<br/>Batched inference & Noisy-OR evidence aggregation"]
C --> D["Deterministic Three-Tiered Output Reorganization<br/>Zero-LLM string partitioning into structured tiers"]
D --> E["Contract-Compliant Final Response<br/>Verified Core / Uncertain Claims / Conflict Report"]
Key Designs¶
1. Contract-Aware Draft Generation: Constraining Hallucination and Surfacing Latent Conflict Signals Standard RAG prompt templates allow models to blend parametric knowledge seamlessly with external context, causing evidence absence or contradictions to be masked during decoding. This design addresses this unconstrained blending by guiding the generator in a single forward pass via four universal behavioral directives: (i) every factual claim must cite at least one supporting passage; (ii) when retrieved passages contradict each other, the model must explicitly acknowledge the dispute rather than arbitrarily siding with one; (iii) the model must not incorporate external parametric knowledge absent from the retrieved passages; and (iv) citations must strictly point to passages that genuinely substantiate the claim. The generator produces an initial draft \(\hat{y}\) with sentence-level citations, focusing purely on candidate drafting while delegating compliance auditing to external components.
2. Sentence-Level NLI Verification & Contract Graph: Independent Assessment of Evidence Accumulation Relying on LLM self-reflection introduces self-reinforcing evaluation blind spots and uncalibrated confidence. To establish objective evidence dependence, the draft \(\hat{y}\) is decomposed into distinct sentences \(S = \{s_1, \dots, s_n\}\) and verified against all retrieved passages \(p_j \in P\) using a DeBERTa-v3-large verifier fine-tuned on FEVER. The verifier outputs probability triples \((p_{sup}, p_{ref}, p_{neu})\) in batched execution. To properly account for the accumulation of multiple weakly supporting or refuting passages, support \(S(s_i)\) and refutation \(R(s_i)\) scores are aggregated via noisy-OR pooling: $\(S(s_i) = 1 - \prod_{p_j \in Z^+(s_i)} (1 - w_{ij}), \quad R(s_i) = 1 - \prod_{p_j \in Z^-(s_i)} (1 - w_{ij})\)$ where \(Z^+\) and \(Z^-\) represent supporting and refuting passage sets filtered at threshold \(\theta = 0.3\), and \(w_{ij}\) denotes verifier confidence. In conjunction with redundancy rate \(\rho(s_i)\) and calibrated thresholds (\(\tau_s = 0.70, \tau_r = 0.20, \tau_\rho = 0.90, \delta = 0.1\)), each sentence is mapped to one of four mutually exclusive contract states: LICENSED (LIC), OVERCITED (OVER), CONTRADICTED (CONTR), or UNSUPPORTED (UNSUP), yielding contract graph \(G = (S, P, R, W)\).
3. Deterministic Three-Tiered Output Reorganization: Translating Verification Metadata into Observable Compliance A diagnostic score remains ineffective unless translated into transparent, observable user-facing responses, yet invoking secondary LLM calls for text rewriting introduces latency and secondary hallucinations. Serving as the operational bridge between verification metadata and final text, Phase 3 reorganizes the original draft via deterministic string operations without changing the wording of individual sentences. Validated claims labeled LIC or OVER are placed in the primary Verified Core tier; claims flagged as UNSUP are moved to an explicit "[Uncertain Claims]" section, signaling that supporting evidence is insufficient; and claims classified as CONTR are converted into a structured "[Conflict Report]" that explicitly specifies the disputed proposition alongside the contrasting passage IDs (e.g., supported by passage [1], contradicted by passage [2]). This zero-LLM transformation guarantees reproducible, audit-compliant behavior at negligible compute cost.
Benchmark & Evaluation Protocol¶
To quantitatively probe system responsiveness to context interventions, the evaluation framework introduces four counterfactual intervention operators \(T_r\) and a unified composite metric: - Deletion Operator (\(T_{delete}\)): Removes 50% of the retrieved passages (or targets only cited supporting passages in a support-critical audit), evaluated via Deletion Acceptance Rate (DAR) to measure whether sentence support drops by \(\Delta Sup(s_j) > \tau_d\) (\(\tau_d = 0.15\)); - Conflict Operator (\(T_{conflict}\)): Injects contradictory factual passages, evaluated via Conflict Detection Rate (CDR) to verify whether any sentence triggers a refutation score \(R(s_j) > \tau_r\) (\(\tau_r = 0.20\)); - Distraction Operator (\(T_{distract}\)): Injects 4 irrelevant passages sampled from other benchmark questions, evaluated via Distract Immunity (DI) using ROUGE-L F1 between original and intervened outputs; - Permutation Operator (\(T_{permute}\)): Shuffles passage order across two independent trials, evaluated via Permutation Stability (PS) using ROUGE-L F1 between permutations.
These behavioral dimensions are synthesized into the Intervention Compliance Score (ICS): $\(\text{ICS} = \frac{1}{4} \left( \underbrace{\text{DAR} + \text{CDR}}_{\text{Behavioral Correctness}} + \underbrace{\text{DI} + \text{PS}}_{\text{Behavioral Stability}} \right)\)$
Key Experimental Results¶
Main Results¶
Experiments evaluate 8 baselines spanning dense, hybrid, long-context, and self-reflective architectures across five benchmark datasets: ASQA, ELI5, and QAMPARI from the ALCE benchmark; GaRAGe (multi-hop reasoning); and MAGIC (inter-context conflict resolution). All models utilize Qwen3-8B-Instruct as the base generator, BGE-M3 with Qwen3-Reranker-0.6B for retrieval, and DeBERTa-v3-large for verification.
Dynamic Intervention Results (DAR, CDR, DI, PS, and Overall ICS)
| Dataset | Method | DAR (↑) | CDR (↑) | DI (↑) | PS (↑) | ICS (↑) |
|---|---|---|---|---|---|---|
| ASQA | Vanilla RAG | 0.528 | 0.796 | 0.170 | 0.259 | 0.438 |
| Dense-RAG | 0.564 | 0.736 | 0.168 | 0.259 | 0.432 | |
| Hybrid-RAG | 0.519 | 0.789 | 0.170 | 0.259 | 0.434 | |
| Long-context RAG (top-30) | 0.533 | 0.831 | 0.191 | 0.259 | 0.454 | |
| Self-RAG-style | 0.455 | 0.817 | 0.150 | 0.262 | 0.421 | |
| DOVE-RAG (Ours) | 0.632 | 0.997 | 0.548 | 0.418 | 0.649 | |
| ELI5 | Vanilla RAG | 0.663 | 0.587 | 0.153 | 0.219 | 0.406 |
| Hybrid-RAG | 0.662 | 0.610 | 0.154 | 0.219 | 0.411 | |
| DOVE-RAG (Ours) | 0.876 | 0.984 | 0.470 | 0.377 | 0.677 | |
| QAMPARI | Vanilla RAG | 0.696 | 0.790 | 0.165 | 0.251 | 0.476 |
| Long-context RAG | 0.700 | 0.787 | 0.182 | 0.252 | 0.480 | |
| DOVE-RAG (Ours) | 0.673 | 0.999 | 0.559 | 0.413 | 0.661 | |
| GaRAGe | Vanilla RAG | 0.910 | 0.311 | 0.353 | 0.427 | 0.500 |
| Hybrid-RAG | 0.901 | 0.396 | 0.346 | 0.427 | 0.518 | |
| DOVE-RAG (Ours) | 0.873 | 0.594 | 0.482 | 0.437 | 0.597 | |
| MAGIC | Vanilla RAG | 0.582 | 0.812 | 0.264 | 0.738 | 0.599 |
| Self-RAG-style | 0.332 | 0.847 | 0.306 | 0.736 | 0.555 | |
| DOVE-RAG (Ours) | 0.948 | 0.505 | 0.494 | 0.737 | 0.671 |
Static Generation Quality and Citation Metrics (ALCE Protocol)
| Dataset | Method | Answer Quality AnsQ | Citation Precision Cite-P | Citation Recall Cite-R |
|---|---|---|---|---|
| ASQA | Vanilla RAG | 0.252 | 0.975 | 0.301 |
| Hybrid-RAG | 0.248 | 0.973 | 0.304 | |
| Long-context RAG | 0.220 | 0.986 | 0.496 | |
| Self-RAG-style | 0.262 | 0.687 | 0.126 | |
| DOVE-RAG (Ours) | 0.245 (Full: 0.178) | 0.982 | 0.633 | |
| ELI5 | Vanilla RAG | 0.116 | 0.709 | 0.210 |
| DOVE-RAG (Ours) | 0.128 | 0.987 | 0.833 | |
| QAMPARI | Vanilla RAG | 0.062 | 0.945 | 0.363 |
| DOVE-RAG (Ours) | 0.068 | 0.974 | 0.606 | |
| GaRAGe | Vanilla RAG | 0.223 | 0.962 | 0.485 |
| DOVE-RAG (Ours) | 0.213 | 0.973 | 0.642 | |
| MAGIC | Vanilla RAG | 0.102 | 0.732 | 0.254 |
| DOVE-RAG (Ours) | 0.197 (Full: 0.084) | 1.000 | 0.967 |
Ablation Study¶
Ablations systematically remove pipeline components to isolate the operational drivers of behavioral compliance. License Compliance Score is computed as \(\text{LCS} = R_{lic} \times (1 - R_{contr}) \times (1 - R_{over})\).
ASQA Component Ablation (N=948)
| Configuration | LCS | Cite-P | Cite-R | ContrRate (↓) | LicRate | AnsQ | Note |
|---|---|---|---|---|---|---|---|
| DOVE-RAG (Full Pipeline) | 0.774 | 0.982 | 0.633 | 0.012 | 0.853 | 0.245 | Full pipeline: prompt + NLI + Phase 3 tiering |
| w/o Structured Output | 0.275 | 0.742 | 0.339 | 0.014 | 0.436 | 0.232 | Retains verification, drops structured prompt rules |
| w/o Phase 3 (Output Restructuring) | 0.104 | 0.961 | 0.663 | 0.446 | 0.235 | 0.199 | Drops tiering, outputs unparsed raw draft |
GaRAGe Component Ablation (N=355)
| Configuration | LCS | Cite-P | Cite-R | ContrRate (↓) | LicRate | AnsQ |
|---|---|---|---|---|---|---|
| DOVE-RAG (Full Pipeline) | 0.914 | 0.973 | 0.642 | 0.043 | 0.917 | 0.213 |
| w/o Structured Output | 0.604 | 0.748 | 0.224 | 0.041 | 0.709 | 0.203 |
| w/o Phase 3 (Output Restructuring) | 0.223 | 0.967 | 0.659 | 0.155 | 0.276 | 0.232 |
Key Findings¶
- Separation of Passive NLI Detection from Explicit Behavioral Action: While baselines achieve moderate passive NLI refutation scores (CDR of 0.283–0.847, rising to 0.812–0.847 on MAGIC), manual and control audits reveal that baseline models virtually never articulate conflict in natural language (Explicit-CDR is 0.004 on ASQA and 0.149 on MAGIC). Baselines shift their output text under deletion because they fall back to parametric knowledge, not because they intentionally acknowledge insufficiency.
- Phase 3 Restructuring Serves as the Crucial Operational Bridge: Removing Phase 3 post-hoc tiering causes LCS to collapse from 0.774 to 0.104 on ASQA and from 0.914 to 0.223 on GaRAGe, with unresolved contradiction rates jumping from 0.012 to 0.446. Verification alone does not modify generated text; actionable reorganization is required to render internal verification observable.
- Targeted Evidence Loss vs. Random Deletion: Under random 50% passage deletion on ASQA, 54.3% of test samples experience no loss of supporting evidence due to context redundancy. In a support-critical audit (N=200) where only cited supporting passages are removed, DOVE-RAG's DAR increases from 10.0% to 97.5%, demonstrating acute sensitivity to the loss of necessary evidence.
- Blinded Human Audit Corroborates Empirical Superiority: In an 80-sample blinded audit (40 ASQA, 40 MAGIC), DOVE-RAG achieved 97.5% explicit conflict acknowledgment (39/40) and 92.5% precise evidence localization (37/40), yielding an 87.5% all-three compliance rate (35/40). In contrast, Vanilla RAG achieved only 22.5% explicit acknowledgment and 20.0% all-three compliance.
Highlights & Insights¶
- Formulating Counterfactual Intervention Contracts for Causal RAG Auditing: Moves beyond static reference matching by introducing programmatic deletion, conflict, distraction, and permutation probes, unmasking systems that use post-hoc citations to disguise memorized parametric fallback.
- High-Trust Enforcement with Negligible Computational Overhead: Disentangles evidence compliance into single-call generation, batched lightweight NLI scoring, and deterministic string tiering, adding only 2.6–3.5s per sample on an A800 GPU without requiring expensive agentic re-generation loops.
- Architectural Generalization across Generators and Verifiers: The pipeline generalizes smoothly to Phi-3-mini-4k-instruct (achieving LCS=0.895 and ICS=0.597 on ASQA). Furthermore, cross-evaluating with an independent RoBERTa-large-MNLI verifier yields consistent baseline rankings (Spearman \(\rho = 0.84, p < 0.001\)) and reinforces DOVE-RAG's compliance superiority across all datasets.
Limitations & Future Work¶
- Controlled Synthetic Contradictions: The conflict intervention operator relies on synthetic, prompt-injected contradictions; scaling the paradigm to naturally occurring, temporal, or perspective-based web conflicts remains an open frontier.
- Sentence-Level Granularity and Over-Filtering: Decomposing claims at sentence granularity can penalize compound sentences containing both supported facts and minor unsupported clauses, occasionally causing benign sentences to be categorized as UNSUPPORTED.
- Post-Hoc Reorganization rather than Constrained Decoding: DOVE-RAG reorganizes drafts post-hoc rather than enforcing contract adherence end-to-end within the generator's autoregressive decoding phase; integrating contract rewards into decoding or reinforcement learning is a promising direction.
Related Work & Insights¶
- vs Conventional Benchmarks (RAGAS, ARES, ALCE): Existing benchmarks measure generation quality on static contexts, failing to differentiate genuine evidence dependence from parametric memorization; Dynamic Evidence Contracts probe counterfactual responsiveness across evidence states.
- vs Self-Reflective RAG (CRAG, Self-RAG): Self-reflective models rely on internal critique tokens or iterative multi-turn LLM generation, incurring significant compute costs while remaining susceptible to knowledge conflict hallucinations; DOVE-RAG leverages external, deterministic NLI verification and structured output tiering in a single generation round.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates an innovative behavioral contract testing paradigm that rigorously separates static grounding from dynamic causal evidence dependence.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 5 diverse datasets and 8 baselines, backed by ablations, weight sensitivity sweeps, cross-model/cross-verifier validations, and blinded human audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear definitions, cohesive narrative prose, disciplined mathematical formulations, and rigorous distinction between passive detection and active behavior.
- Value: ⭐⭐⭐⭐⭐ Provides crucial evaluation methodology and an actionable, low-overhead blueprint for building verifiable, audit-ready RAG deployments in high-stakes domains.