Skip to content

The RAT: A Unified Bayesian Model for RAG Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/vodezhaw/rat
Area: Information Retrieval
Keywords: RAG Evaluation, Bayesian Model, Abstention Behavior, Policy Adherence, Uncertainty Quantification

TL;DR

The paper introduces RAT, a unified Bayesian evaluation framework for Retrieval-Augmented Generation (RAG) that factorizes retrieval success, abstention behavior, and answer correctness along the pipeline's information flow, disentangles task success from generator policy adherence, reveals through information theory why retrieval annotations are far more informative than task correctness annotations, and seamlessly integrates calibrated noisy LLM judges.

Background & Motivation

Retrieval-Augmented Generation (RAG) pipelines have emerged as the foundational paradigm for anchoring large language models to external, verifiable knowledge sources and mitigating hallucination risks. Despite their ubiquity, evaluating RAG systems comprehensively remains notoriously difficult and fraught with structural blind spots. Prevailing evaluation methodologies either isolate individual components by evaluating retrieval engines with standard information retrieval metrics and generators via synthetic question sets, or evaluate the system strictly end-to-end based solely on whether the final generated string matches a gold-standard reference. However, RAG systems are fundamentally sequential pipelines governed by strong cascading error dependencies: when the retrieval step fails to fetch the necessary evidence, the downstream generator lacks any legitimate grounding to produce a factual answer, and the only faithful, safe behavior is to explicitly abstain from answering.

The central limitation of end-to-end evaluation is that it conflates two fundamentally distinct concepts of success: task success, which reflects whether the end user received a correct answer, and generator success (or policy adherence), which evaluates whether the generator behaved appropriately given the actual retrieval outcome. This conflation obscures severe behavioral deficiencies. An aggressive generator that never abstains may achieve respectable marginal task accuracy purely through memorized parametric weights or lucky guesses while systematically violating safety policies under retrieval failures. Conversely, two pipelines exhibiting identical end-to-end task accuracy can exhibit wildly disparate reliability when retrieval fails. Marginal metrics collapse the pipeline into a black box, rendering it impossible to diagnose error propagation or quantify estimation uncertainty under budget constraints.

To overcome these structural limitations, an evaluation methodology must explicitly capture the conditional dependencies connecting retrieval outcomes, abstention decisions, and answer correctness. The core idea of this paper is to formulate a unified Bayesian evaluation framework that factorizes retrieval success, abstention behavior, and task correctness according to the pipeline's directed information flow, formalizing generator policy adherence as a deterministic logical variable to enable exact uncertainty propagation, budget-optimal sample allocation, and principled calibration of noisy LLM judges.

Method

Overall Architecture

RAT (Retrieval-Abstention-Task Bayesian Model) formulates RAG evaluation as a probabilistic graphical model whose factorization follows the physical flow of information through the pipeline. The framework models three primary binary variables: retrieval success \(R \in \{0, 1\}\), indicating whether the retrieved context contains all necessary evidence to answer the query; abstention behavior \(A \in \{0, 1\}\), indicating whether the generator declines to answer by emitting an explicit refusal string (such as "I DO NOT KNOW"); and task success \(T \in \{0, 1\}\), representing whether the generated answer matches the reference (with \(T = 0\) deterministically when \(A = 1\)).

Under this information flow, the joint distribution is factorized into the prior probability of retrieval success \(P(R)\), the retrieval-conditioned abstention probability \(P(A \mid R)\), and the conditional answer correctness \(P(T \mid A=0, R)\). Generator policy adherence \(G \in \{0, 1\}\) is modeled as a deterministic function of these random variables, capturing whether the generator abstains when retrieval fails and answers correctly when retrieval succeeds. The entire model is implemented in Stan and estimated via Hamiltonian Monte Carlo with the No-U-Turn Sampler (NUTS), naturally supporting partial observation marginalization and calibrated noisy observations from automated LLM judges.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RAG Evaluation Pipeline Input<br/>(Query, Context, Output)"] --> B["Information-Flow Conditional Factorization<br/>P(R) · P(A|R) · P(T|A=0, R)"]
    B --> C["Deterministic Policy Adherence Derivation<br/>G = (R=0 ∧ A=1) ∨ (R=1 ∧ A=0 ∧ T=1)"]
    C --> D["Bayesian Uncertainty Inference<br/>Stan NUTS Sampling & Partial Marginalization"]
    D --> E["Information-Theoretic Sample Allocation<br/>Target-Aware Optimal Annotation Budgeting"]
    D --> F["Calibrated LLM-as-a-Judge Extension<br/>Dirichlet Priors & Noise Likelihood Matrix"]

Key Designs

1. Information-Flow Conditional Factorization: Decoupling Retrieval, Abstention, and Supported Generation

Traditional marginal metrics evaluate overall accuracy while failing to explain how component interactions generate errors. RAT addresses this by decomposing the joint probability distribution strictly along the pipeline's directed acyclic graph: \(P(R, A, T) = P(R) P(A \mid R) P(T \mid A, R)\). Since task success is structurally impossible when the generator abstains (\(A=1 \implies T=0\)), the system's operational space is fully parameterized by five behavioral probabilities:

\[P(R=1) = \theta_R\]
\[\theta_{A-} = P(A=1 \mid R=0), \quad \theta_{A+} = P(A=1 \mid R=1)\]
\[\theta_{T-} = P(T=1 \mid R=0, A=0), \quad \theta_{T+} = P(T=1 \mid R=1, A=0)\]

Here, \(\theta_R\) quantifies baseline retriever quality, \(\theta_{A-}\) measures the appropriate abstention rate under retrieval failure, \(\theta_{A+}\) captures unwarranted false-alarm abstention despite successful retrieval, \(\theta_{T-}\) denotes ungrounded guessing accuracy from parametric knowledge, and \(\theta_{T+}\) reflects supported generation accuracy when valid context is provided. This decomposition cleanly separates the generator's intrinsic parametric knowledge from its context utilization fidelity.

2. Deterministic Policy Adherence Derivation: Decoupling Desired Behavior from Marginal Task Accuracy

Conventional evaluation rewards generators that fabricate answers upon retrieval failure whenever they happen to hit the ground-truth string, while penalizing conservative models that adhere to safety instructions. RAT resolves this tension by introducing generator success \(G\) as a deterministic variable over the joint state space:

\[G = (R = 0 \land A = 1) \lor (R = 1 \land A = 0 \land T = 1)\]

Under this policy definition, the generator is deemed successful if and only if it follows one of two compliant behavioral trajectories: it recognizes retrieval failure (\(R=0\)) and abstains (\(A=1\)), or it receives complete context (\(R=1\)), attempts an answer (\(A=0\)), and answers correctly (\(T=1\)). Any hallucinated response attempted without supporting evidence (\(R=0, A=0\)) or any failure to answer when evidence was present (\(R=1, A=1\)) constitutes a policy violation (\(G=0\)).

3. Bayesian Uncertainty Inference and Marginalization: Stan Posterior Computation

In practical deployment, human annotations are expensive and unevenly accessible—abstention \(A\) can be detected at zero cost via regex string matching, whereas validating retrieval sufficiency \(R\) and answer correctness \(T\) requires costly expert verification. RAT frames evaluation as a Bayesian inference problem, assigning independent \(\mathrm{Uniform}(0, 1)\) priors over all five primitive parameters and performing posterior inference in Stan via the No-U-Turn Sampler (using 5 Markov chains with 2,000 warmup iterations and 10,000 posterior draws per chain). For unobserved variables in partially annotated data, the Bayesian model integrates over unobserved states via discrete marginalization, enabling partially labeled instances to contribute statistical strength while propagating uncertainty into exact 95% credible intervals for \(P(G=1)\).

4. Information-Theoretic Sample Allocation: Structural Asymmetry in Retrieval Annotations

When operating under fixed annotation budgets, practitioners must decide whether to allocate scarce human annotations to retrieval validation \(R\), task correctness \(T\), or both. RAT formalizes this choice through conditional mutual information \(I(G; \cdot \mid A)\). Within the contingency table of four \((R, A)\) state cells, three cells resolve \(G\) deterministically: \((R=0, A=1) \implies G=1\), \((R=0, A=0) \implies G=0\), and \((R=1, A=1) \implies G=0\). Only the single cell \((R=1, A=0)\) leaves \(G\) unresolved, where \(G = T\).

This structural asymmetry reveals that observing \(R\) directly resolves \(G\) in 75% of the state space, whereas observing \(T\) resolves \(G\) only in the single \((A=0, T=0)\) cell (which yields \(G=0\)). Consequently, when estimating policy adherence \(P(G=1)\), allocating annotations to retrieval success provides significantly higher per-sample mutual information than allocating them to task correctness.

5. Calibrated LLM-as-a-Judge Extension: Incorporating Automated Judges as Noisy Observations

To leverage inexpensive automated evaluators without inheriting their uncalibrated systematic biases, RAT extends the joint distribution to include binary judge observations \(R_J\) and \(T_J\): \(P(R, A, T, R_J, T_J) = P(R, A, T) P(R_J, T_J \mid R, A, T)\). Rather than enforcing fragile conditional independence assumptions, RAT models a 4-category conditional distribution over \((R_J, T_J)\) across all 6 feasible latent \((R, A, T)\) states with symmetric Dirichlet priors. By embedding the empirical true positive rate (TPR) and false positive rate (FPR) of the judge into the observation likelihood, the framework allows practitioners to pool thousands of noisy automated judgments with small sets of gold human labels while correctly penalizing judge uncertainty in the posterior.

Key Experimental Results

Main Results

The framework was evaluated on 27 distinct RAG configurations across three knowledge-intensive datasets from the KILT benchmark: FEVER (fact verification, single-evidence), HotpotQA (multi-hop QA, 2 required documents), and Natural Questions (NQ, open-domain QA, single-evidence). The configurations combine three retrieval strategies—Sparse (BM25), Dense (Jina-embeddings-v3 with HNSW via FAISS), and Hybrid (Reciprocal Rank Fusion, RRF)—with three open-weight generators: Apertus 8B, Gemma3 12B, and Qwen3.5 9B.

The table below reports marginal probabilities across all 27 configurations, showing retrieval success \(P(R=1)\), abstention rate \(P(A=1)\), task success \(P(T=1)\), and policy adherence \(P(G=1)\):

Dataset Retriever Generator \(P(R=1)\) \(P(A=1)\) \(P(T=1)\) \(P(G=1)\)
FEVER Dense Apertus 8B 0.603 0.135 0.753 0.608
FEVER Dense Gemma3 12B 0.603 0.180 0.784 0.736
FEVER Dense Qwen3.5 9B 0.603 0.199 0.751 0.744
FEVER Hybrid Apertus 8B 0.655 0.089 0.801 0.614
FEVER Hybrid Gemma3 12B 0.655 0.096 0.864 0.701
FEVER Hybrid Qwen3.5 9B 0.655 0.112 0.838 0.703
FEVER Sparse Apertus 8B 0.456 0.124 0.759 0.504
FEVER Sparse Gemma3 12B 0.456 0.221 0.744 0.644
FEVER Sparse Qwen3.5 9B 0.456 0.246 0.708 0.657
HotpotQA Dense Apertus 8B 0.151 0.038 0.260 0.110
HotpotQA Dense Gemma3 12B 0.151 0.311 0.253 0.391
HotpotQA Dense Qwen3.5 9B 0.151 0.489 0.260 0.574
HotpotQA Hybrid Apertus 8B 0.207 0.032 0.304 0.138
HotpotQA Hybrid Gemma3 12B 0.207 0.218 0.315 0.337
HotpotQA Hybrid Qwen3.5 9B 0.207 0.375 0.329 0.503
HotpotQA Sparse Apertus 8B 0.218 0.033 0.318 0.152
HotpotQA Sparse Gemma3 12B 0.218 0.226 0.318 0.360
HotpotQA Sparse Qwen3.5 9B 0.218 0.391 0.326 0.535
NQ Dense Apertus 8B 0.347 0.028 0.239 0.164
NQ Dense Gemma3 12B 0.347 0.256 0.242 0.414
NQ Dense Qwen3.5 9B 0.347 0.355 0.243 0.496
NQ Hybrid Apertus 8B 0.348 0.024 0.267 0.163
NQ Hybrid Gemma3 12B 0.348 0.198 0.267 0.359
NQ Hybrid Qwen3.5 9B 0.348 0.307 0.266 0.451
NQ Sparse Apertus 8B 0.244 0.039 0.233 0.136
NQ Sparse Gemma3 12B 0.244 0.300 0.220 0.412
NQ Sparse Qwen3.5 9B 0.244 0.434 0.215 0.531

Conditional Decomposition & Ablation Insights

The model-implied posterior mean estimates for the conditional parameters demonstrate the latent mechanisms governing pipeline behavior:

Dataset Retriever Generator \(\theta_{A-} = P(A=1 \mid R=0)\) \(\theta_{A+} = P(A=1 \mid R=1)\) \(\theta_{T-} = P(T=1 \mid R=0, A=0)\) \(\theta_{T+} = P(T=1 \mid R=1, A=0)\)
FEVER Dense Apertus 8B 0.245 0.063 0.810 0.903
FEVER Dense Gemma3 12B 0.422 0.021 0.941 0.962
FEVER Dense Qwen3.5 9B 0.462 0.027 0.894 0.954
FEVER Hybrid Apertus 8B 0.158 0.053 0.831 0.901
FEVER Hybrid Gemma3 12B 0.242 0.019 0.940 0.961
FEVER Hybrid Qwen3.5 9B 0.274 0.027 0.913 0.955
FEVER Sparse Apertus 8B 0.197 0.038 0.828 0.906
FEVER Sparse Gemma3 12B 0.391 0.018 0.942 0.963
FEVER Sparse Qwen3.5 9B 0.428 0.028 0.916 0.956
HotpotQA Dense Apertus 8B 0.044 0.010 0.229 0.492
HotpotQA Dense Gemma3 12B 0.361 0.026 0.313 0.570
HotpotQA Dense Qwen3.5 9B 0.568 0.044 0.458 0.638
HotpotQA Hybrid Apertus 8B 0.039 0.005 0.258 0.519
HotpotQA Hybrid Gemma3 12B 0.270 0.021 0.331 0.608
HotpotQA Hybrid Qwen3.5 9B 0.463 0.036 0.456 0.678
HotpotQA Sparse Apertus 8B 0.041 0.006 0.264 0.552
HotpotQA Sparse Gemma3 12B 0.286 0.011 0.326 0.632
HotpotQA Sparse Qwen3.5 9B 0.493 0.028 0.446 0.704
NQ Dense Apertus 8B 0.039 0.007 0.161 0.402
NQ Dense Gemma3 12B 0.382 0.021 0.192 0.485
NQ Dense Qwen3.5 9B 0.513 0.060 0.256 0.494
NQ Hybrid Apertus 8B 0.034 0.006 0.179 0.408
NQ Hybrid Gemma3 12B 0.290 0.024 0.210 0.499
NQ Hybrid Qwen3.5 9B 0.439 0.061 0.277 0.503
NQ Sparse Apertus 8B 0.050 0.009 0.187 0.407
NQ Sparse Gemma3 12B 0.387 0.032 0.216 0.505
NQ Sparse Qwen3.5 9B 0.550 0.077 0.293 0.513

Furthermore, when evaluating Apertus on HotpotQA + Hybrid using 200 fully annotated base samples supplemented with \(n_{\mathrm{add}} \in \{0, 500, 5000\}\) automated GPT-4o-mini judge annotations, the noise characteristics severely limited performance improvements:

Added Judge Samples \(n_{\mathrm{add}}\) \(P(G=1)\) MAE \(P(G=1)\) 95% CI Width \(P(T=1)\) MAE \(P(T=1)\) 95% CI Width
0 0.0174 0.0941 0.0232 0.1246
500 0.0166 0.0838 0.0243 0.1176
5,000 0.0170 0.0802 0.0230 0.1150

Key Findings

  • Divergence of Marginal Accuracy and Policy Adherence: On NQ with Dense retrieval, all three generators exhibit virtually indistinguishable task success rates (Apertus: 0.239, Gemma3: 0.242, Qwen3.5: 0.243). However, their policy adherence metrics diverge radically (Apertus: 0.164, Gemma3: 0.414, Qwen3.5: 0.496). The conditional decomposition reveals that this disparity is entirely governed by abstention under failure \(\theta_{A-}\) (0.039 for Apertus versus 0.513 for Qwen3.5).
  • Asymmetric Value in Sample Budgeting: For estimating policy adherence \(P(G=1)\), the All-R annotation strategy captures 58.0% to 65.6% of the mutual information delivered by the full joint strategy, whereas the All-T strategy captures only 29.3% to 40.8%. This empirically confirms the theoretical derivation that observing retrieval resolves \(G\) in 3 out of 4 joint cells, whereas observing answer correctness resolves only 1.
  • Diminishing Returns from Uncalibrated LLM Judges: GPT-4o-mini exhibits high sensitivity (TPR 0.77–0.94) but substantial false positive rates (FPR 0.17–0.43 for retrieval and task judgments). Because of this severe noise, adding 5,000 automated judge labels reduces the 95% credible interval width for \(P(G=1)\) only marginally from 0.0941 to 0.0802 without improving MAE, underscoring that automated judges cannot replace small-scale human gold annotations unless calibration is exceptionally sharp.

Highlights & Insights

  • Causal Decomposition Paradigm: Transforms RAG evaluation from a bag of uncoupled heuristic scores (like independent context-relevance and answer-relevance metrics in RAGAs/ARES) into an information-flow Bayesian graphical model with rigorous uncertainty quantification.
  • Information-Theoretic Annotation Guidance: Provides mathematical justification for why engineering teams evaluating RAG systems should prioritize retrieval auditing over output labeling when evaluating system trust and safety compliance.
  • Principled Hybrid Evaluation Framework: Offers a plug-and-play Bayesian formulation capable of combining scarce gold human judgments with abundant, noisy automated judge outputs in a single consistent posterior.

Limitations & Future Work

  • Binary Variable Simplification: The base model restricts retrieval quality and answer correctness to binary variables (\(0/1\)). In industrial IR settings, continuous ranking metrics (NDCG, MAP) provide finer granularity, and extending the Bayesian framework to continuous observations remains open.
  • Static Retrieve-then-Generate Scope: The formulation assumes a single-turn pipeline, omitting multi-turn conversational dependencies, query reformulation passes, and iterative agentic RAG loops.
  • Strict Deterministic Policy: Policy adherence \(G\) assumes a rigid policy where generating an answer is acceptable only if retrieval is completely successful. Soft policies that permit partial answers under incomplete retrieval would require probabilistic relaxations of \(G\).
  • vs RAGAs (Es et al., 2024) / ARES (Saad-Falcon et al., 2024): Existing triad frameworks treat retrieval relevance, faithfulness, and answer relevance as mutually independent evaluation dimensions; RAT models the explicit statistical dependencies along the pipeline's information flow, decoupling task success from behavioral policy compliance.
  • vs Feb4RAG (Wang et al., 2024): Feb4RAG assesses generator robustness by artificially injecting adversarial retrieved contexts; RAT natively models the generator's intrinsic abstention policy and evaluates system fidelity under natural operational distributions.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ First principled Bayesian graphical model for RAG pipeline evaluation with information-theoretic sample budgeting.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 27 system configurations and exhaustive MCMC sampling validation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, elegant narrative flow, and tightly integrated theoretical analyses.
  • Value: ⭐⭐⭐⭐⭐ Directly applicable to industrial RAG benchmarking, offering actionable advice for maximizing annotation ROI and calibrating LLM judges.