Skip to content

EARAL: Efficient Attribution of Retrieval-Augmented Answer Generation using Logits

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/lucassA/EARAL
Area: Information Retrieval & RAG
Keywords: Retrieval-Augmented Generation, Document Attribution, Token Logits, Faithful Attribution, Low-Latency Inference

TL;DR

EARAL prefixes retrieved documents with special identifier tokens and jointly fine-tunes LLMs to calibrate identifier logits alongside answer generation, enabling faithful, generation-time document attribution directly from pre-softmax token logits with zero extra inference passes and a 104x speedup over ablation-based baselines.

Background & Motivation

Retrieval-Augmented Generation (RAG) significantly mitigates factual hallucinations in Large Language Models (LLMs) by conditioning text generation on retrieved external documents. However, in mission-critical applications such as healthcare and legal reasoning, generating correct answers is insufficient; models must reliably link each generated statement to its supporting evidence via verifiable document attributions. Existing attribution paradigms suffer from fundamental trade-offs across faithfulness and computational efficiency. Self-generated citation approaches prompt or tune models to emit bracketed citation markers inline during decoding, but empirical studies show these often represent post-hoc rationalizations rather than faithful reflections of internal model dynamics. Retrieval-based attribution pipelines decompose generated statements into claims and execute secondary retrieval passes, yet they inevitably introduce latency overheads and tend to identify semantically plausible documents rather than the actual passages that causally drove token generation. Meanwhile, model-internal approaches track attention distributions, feature gradients, or input ablations (e.g., MIRAGE, ContextCite) to capture authentic internal signals, but they demand dozens of auxiliary forward passes or complex surrogate fitting, inflating inference latency by orders of magnitude.

The core tension lies in collecting faithful, token-level attribution signals directly during autoregressive decodingโ€”reflecting the model's true decision state at generation timeโ€”without paying prohibitive multi-pass latency penalties. Intuitively, when retrieved passages are tagged with unique identifier tokens, the pre-softmax logits assigned to those identifiers at each generation step naturally encode the instantaneous relevance of each document to the newly generated token. However, reading raw logits directly from frozen language models fails due to two severe systemic distortions: selection bias (extreme sensitivity to the arbitrary ordering and positioning of context passages) and token bias (prior lexical frequency artifacts inherited from pre-training corpora that artificially boost or suppress specific identifier symbols regardless of content).

This paper's angle of attack is that identifier logits can be cleanly debiased and regularized by treating attribution as a calibrated multi-task regression problem during fine-tuning, preserving the natural language generation flow while explicitly teaching identifier logits to track true evidence grounding. Core idea: prefix retrieved documents with lightweight identifier tokens and jointly fine-tune the generator on language modeling and mean-squared-error logit calibration, thereby extracting faithful token-level document contributions from pre-softmax logits in a single forward pass and aggregating them via dual-threshold pooling into low-latency statement-level citations.

Method

Overall Architecture

The EARAL architecture operates in three closely coupled phases: structured identifier marking over retrieved documents, debiased joint fine-tuning for simultaneous generation and logit calibration, and token-to-statement attribution aggregation. Given a user query \(q\) and a set of \(K\) retrieved context documents \(C = \{d_1, \dots, d_K\}\), the system prefixes each passage with a unique, low-frequency single-token identifier to form a marked context. During autoregressive decoding of each answer token \(s_{ij}\), the model outputs standard next-token predictions while simultaneously exposing pre-softmax logits corresponding to each document identifier token as continuous contribution scores. Finally, a pooling operator evaluates the temporal persistence of these contribution scores across the statement's token span, mapping continuous logits into discrete document citations while deploying a safety refusal fallback if no document reaches attribution thresholds.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Retrieval Input Context<br/>Query q + Context C"] --> B["Identifier Marking Strategy<br/>markingBA wraps documents with special tokens"]
    B --> C["Debiased Joint Fine-Tuning<br/>Cross-entropy generation loss + MSE logit loss"]
    C --> D["Autoregressive Contribution Extraction<br/>Read pre-softmax identifier token logits"]
    D --> E["Dual-Threshold Pooling Aggregation<br/>Contribution threshold ฯ•prop and ratio threshold ฮป"]
    E -->|No supporting document passed| F["Refusal Fallback Mechanism<br/>Emit standardized refusal string"]
    E -->|Supported by evidence| G["Faithful Attributed Statements<br/>Discrete statement-level citations"]

Key Designs

1. Identifier Marking Strategy: Leveraging Code-Syntax Priors for Minimal Context Distortion To enable language models to assign distinct vocabulary probabilities to specific source documents at each decoding step, every retrieved passage \(d_k\) is associated with a unique identifier token \(\text{id}_k\). Using multi-token strings or complex prefixes fragments subword tokenization and confuses step-wise probability assignment. EARAL selects 10 single-token identifiers from the tokenizer vocabulary that exhibit near-zero natural frequency in standard training and evaluation corpora (AA, BB, CC, DD, EE, FF, GG, HH, II, JJ). The authors evaluate three marking granularities: wrapping the entire document before and after using angle brackets (\(\text{markingBA}\), e.g., < AA>...< /AA>), wrapping every individual sentence within each passage (\(\text{markingBAS}\)), and repeating the identifier before every single word (\(\text{markingAW}\)). Because modern foundation LLMs are extensively pre-trained on code datasets containing paired XML/HTML-style closing tags, the macro-level angle-bracket enclosure of \(\text{markingBA}\) activates native code-block parsing representations with minimal prompt inflation, consistently outperforming fragmented word-level marking during supervised fine-tuning.

2. Debiased Joint Fine-Tuning: Decoupled Multi-Task Supervision in Logit Space Directly probing identifier logits in frozen LLMs yields unreliable attributions because positional order and uncalibrated token priors dominate small raw logit variations. To eliminate selection and token biases without corrupting natural language generation quality, EARAL fine-tunes the backbone model using a joint multi-task objective \(\mathcal{L}_{aa}\): $\(\mathcal{L}_{aa} = \sum_{(q, C, \hat{A}) \in \mathcal{D}_{train}} \sum_{\hat{S}_i \in \hat{A}} \left[ (1 - \alpha)\mathcal{L}_{ans} + \alpha\mathcal{L}_{att} \right]\)$ The language modeling objective \(\mathcal{L}_{ans}\) is standard token-level cross-entropy \(\mathcal{L}_{ans} = -\sum \log p(\hat{s}_{ij})\), which preserves conversational fluency and factual accuracy. The attribution objective \(\mathcal{L}_{att}\) is formulated as a Mean Squared Error (MSE) regression loss applied strictly to the tiny subset of vocabulary logits corresponding to active context identifiers (\(K \ll |\mathcal{V}|\)): $\(\mathcal{L}_{att} = \sum_{\hat{s}_{ij} \in \text{tok}(\hat{S}_i)} \sum_{k=1}^K \left( l_k - \hat{l}_k \right)^2\)$ where \(l_k = \mathcal{M}(\text{id}_k \mid q, \hat{A}_{S<i}, S_{i<j}, C)\) represents the predicted pre-softmax logit. The target labels \(\hat{l}_k\) encode distinct hierarchical levels of evidential support: true supporting gold documents for the current statement receive \(\hat{l}_k = 4\); irrelevant passages present in the prompt context receive \(\hat{l}_k = 2\); randomly sampled uninformative passages receive \(\hat{l}_k = 0\); and out-of-context vocabulary identifiers receive \(\hat{l}_k = -2\). Calibrating in unconstrained logit space prevents the gradient squashing and competitive suppression inherent in Softmax probability distributions, forcing identifier activations to scale linearly with evidential dependency.

3. Dual-Threshold Pooling Aggregation: Mapping Continuous Token Dynamics to Discrete Citations During autoregressive generation, the model produces a continuous vector of document contribution scores \(\text{ctrk}_{ij} = \{l_1, \dots, l_K\}\) for every generated token \(s_{ij}\). While these continuous traces offer fine-grained interpretability, downstream evaluation and human inspection require discrete sentence-level citations. Naive max-pooling over-attributes documents based on isolated punctuation spikes, whereas mean-pooling dilutes critical entity contributions across non-informative stop words. EARAL introduces a robust top-k proportion aggregation function \(\Psi\): $\(a_i = \left\{ \text{id}_k \;\middle|\; \sum_{j=1}^{n_i} \mathbf{1}\left( \text{ctrk}_{ij}^k > \phi_{prop} \right) \ge \lambda n_i \right\}_{k=1}^K\)$ where \(\phi_{prop}\) is the instantaneous contribution threshold (tuned to 3), \(\mathbf{1}(\cdot)\) is the indicator function, and \(\lambda\) is the token proportion threshold (fixed at 0.75). Under this formulation, a document is cited if and only if its calibrated identifier logit exceeds 3 across at least 75% of the tokens comprising the statement. If no context document satisfies this requirement for a given answer span, the system triggers an automatic refusal fallback, preventing ungrounded hallucinated completions from masquerading as attributed facts.

A Worked Example

Consider a factual query from the ASQA benchmark: "Who serves as the president of the senate in the Texas legislature?" 1. Context Marking: Three retrieved documents are prepended with identifiers. Document \(d_1\) states "Texas Lieutenant Governor Dan Patrick...", marked with < AA>...< /AA>; document \(d_2\) contains irrelevant procedural commentary, marked with < BB>...< /BB>; document \(d_3\) explains general state governance ("The Lieutenant Governor of the state..."), marked with < CC>...< /CC>. 2. Online Logit Extraction: As the model generates the statement The president of the senate... is Texas Lieutenant Governor Dan Patrick., internal pre-softmax logits are captured at each step. At the critical token _Lieutenant, the extracted logits are \(\text{AA} = 3.47\), \(\text{BB} = 1.47\), and \(\text{CC} = 2.71\). Document \(d_1\) (\(\text{AA}\)), which uniquely identifies Dan Patrick, comfortably clears the \(\phi_{prop} = 3\) barrier, whereas background document \(d_3\) falls short and distractor \(d_2\) is suppressed. 3. Proportional Pooling & Intervention: Across the entire statement, \(d_1\) maintains an active logit \(>3\) across 92% of generated tokens (exceeding \(\lambda = 0.75\)), yielding discrete attribution \(a_1 = \{\text{AA}\}\). Under a counterfactual ablation test where the name "Dan Patrick" is removed from \(d_1\), the \(\text{AA}\) logit immediately collapses to 2.34, dropping below the proportion threshold and triggering the refusal fallback clause, verifying faithful causal grounding.

Key Experimental Results

Main Results

EARAL was evaluated against leading attribution methods on Llama 3.1 8B using the Trust-Align benchmark across three standard long-form QA datasets: ASQA (ambiguous factoid), QAMPARI (multi-entity factoid), and ELI5 (open-ended explanations). Metrics include answer correctness (\(F1_{AC}\)), appropriate refusal (\(F1_{GR}\)), grounded citation (\(F1_{GC}\)), aggregate trustworthiness score (\(\text{TRUST}\)), and average query latency in milliseconds.

Model & Method ASQA TRUST QAMPARI TRUST ELI5 TRUST Avg Latency (ms) Attribution Category
PostCite* (Llama3 8B) 38.10 16.35 24.91 - Post-hoc Retrieval
PostAttr* (Llama3 8B) 41.62 22.78 27.18 - Post-hoc TRUE-NLI Entailment
FRONT* (Llama3 8B) 56.67 14.09 22.64 - Fine-Grained Self-Citation
Trust-Align* (Llama3 8B) 66.45 50.99 44.83 - Trustworthiness Multi-Task SFT
ContextCite (Llama3.1 8B) 61.38 31.37 44.32 6031.12 Surrogate Linear Modeling
MIRAGE (Llama3.1 8B) 59.01 29.54 41.63 35024.44 Input Ablation & Sensitivity
Llama 3.1 8B Zero-Shot (markingBA) 57.15 26.40 39.03 341.22 Raw Logits Zero-Shot
Llama 3.1 8B FT w/ Trust-Align 67.83 43.53 44.18 329.73 Text Citation Supervised Tuning
EARAL (markingBA) 70.86 47.63 48.94 335.55 Proposed Logit Attribution
EARAL (markingBAS) 65.47 42.27 44.14 341.54 Sentence-Level Marking
EARAL (markingAW) 66.44 42.91 46.23 407.11 Word-Level Marking

Note: Starred () baseline metrics are cited directly from Song et al. (2025). EARAL achieves state-of-the-art TRUST performance across all three benchmarks. Crucially, EARAL operates at 335.55 ms per queryโ€”matching normal generation latency while executing 104.38x faster than MIRAGE (35,024.44 ms) and 17.97x faster than ContextCite (6,031.12 ms).*

Ablation Study

The ablation analysis investigates the impact of logit debiasing supervision and alternative contribution aggregation functions using the primary \(\text{markingBA}\) configuration.

Configuration ASQA TRUST QAMPARI TRUST ELI5 TRUST Note / Relative Change
EARAL (Full Model) 70.86 47.63 48.94 Full model with MSE logit debiasing and top-k pooling
w/o debiasing (Raw Logits) 55.13 25.13 30.01 Eliminates MSE loss: ASQA -22.20%, QAMPARI -47.21%, ELI5 -38.68%
w/o aggregation + Max operator 67.76 47.27 46.65 Replaces \(\Psi\) with max pooling: ASQA -4.37%, QAMPARI -0.69%, ELI5 -4.68%
w/o aggregation + Mean operator 69.31 45.55 46.98 Replaces \(\Psi\) with mean pooling: ASQA -2.19%, QAMPARI -4.31%, ELI5 -4.00%
Alternative: ablate-repeat 50.36 26.53 29.99 Prompt ablation probability shifts (latency inflates to 771 ms)
Alternative: KL-proba 62.01 39.44 43.18 KL divergence on vocabulary probabilities (ASQA lags at 62.01 vs 70.86)

Key Findings

  • Logit debiasing is indispensable for faithful attribution: Removing logit supervision and relying on raw generation logits causes dramatic performance collapses, with TRUST dropping by 47.21% on QAMPARI and 38.68% on ELI5. This proves that uncalibrated model logits are heavily corrupted by pre-training priors and position bias.
  • MSE in logit space significantly outperforms probability-space KL divergence: Direct MSE supervision on pre-softmax identifier activations yields a +14.27% gain on ASQA (70.86 vs. 62.01) compared to KL-divergence over Softmax probability distributions, confirming that linear logit space prevents destructive gradient saturation.
  • Coarse-grained document marking outperforms dense insertions: In fine-tuned regimes, document-level < AA>...< /AA> marking achieves 70.86 TRUST, noticeably higher than sentence-level (65.47) and word-level (66.44) marking. Dense in-text insertions fragment natural language sequences and degrade syntactic coherence.
  • Remarkable order invariance and threshold stability: As training proceeds past 0.8 epochs, performance across random vs. alphabetical token assignments and standard vs. reversed document orderings completely converges. Furthermore, varying the proportion threshold \(\lambda\) across \(\{0.25, 0.50, 0.75\}\) produces less than 3.40% variation in TRUST scores, proving that calibrated identifier logits remain decisively high or low across entire statements.

Highlights & Insights

  • Zero-overhead model-internal attribution via vocabulary repurposing: By binding contextual passages to unused single-token identifiers, EARAL extracts internal attribution signals directly from standard decoder output layers, bypassing the need for auxiliary attention pooling or external classification heads.
  • A 104x speedup bridging the gap to industrial deployment: Prior model-internal attribution techniques were confined to academic benchmarks due to massive multi-forward inference bottlenecks (MIRAGE requires ~35 seconds per query). EARAL maintains native conversational speed (335 ms), unlocking practical real-time grounded generation.
  • Counterfactual ablation confirms true causal grounding: On the ASQA dataset, ablating the critical answer-bearing text from the cited document triggered answer or attribution changes in 94.80% of test queries, whereas substituting irrelevant context documents induced changes in only 1.13% of cases.

Limitations & Future Work

  • Constrained identifier pool and retrieval depth: Experiments currently validate a fixed set of 10 single-token identifiers on top-5 retrieval settings. Extending the approach to long-context RAG environments with dozens of retrieved passages requires exploring expanded identifier vocabularies without diluting output attention mass.
  • Empirically tuned target logit values: The stepped target values (\(\hat{l}_k \in \{4, 2, 0, -2\}\)) were optimized heuristically via validation set tuning. Formulating end-to-end adaptive logit scaling bounds represents an important theoretical refinement.
  • Single model family evaluation: Empirical evaluations are restricted to 8B parameter variants of the Llama family. Exploring scaling dynamics on 70B+ models and mixture-of-experts (MoE) architectures remains an open direction.
  • vs. MIRAGE (Qi et al., 2024): MIRAGE identifies sensitive tokens via token-level input ablation and calculates contrastive feature importance, suffering from a 35-second inference latency. EARAL obtains attribution scores natively from generation logits, achieving a 104x speedup and improving \(F1_{GC}\) citation quality by 56.25% on ASQA (82.69 vs 52.92).
  • vs. Trust-Align / FRONT (Song et al., 2025; Huang et al., 2024a): Trust-Align relies on auto-regressive generation of citation brackets inline with text tokens. EARAL decouples the citation signal from the output string into latent vocabulary logits, preventing citation syntax from biasing natural language generation trajectories.
  • vs. ContextCite (Cohen-Wang et al., 2024): ContextCite fits query-specific surrogate linear models over internal representations, requiring ~6,031 ms per query and showing brittle performance on entity-dense queries. EARAL amortizes attribution learning into pre-training/fine-tuning weights, operating at constant computational complexity during inference.

Rating

  • Novelty: โญโญโญโญโญ Elegant formulation framing RAG document attribution as a calibrated logit regression problem with zero inference overhead.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across three Trust-Align benchmarks, extensive ablations, and rigorous counterfactual faithfulness testing.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative progression connecting theoretical motivation, bias mechanics, and experimental validation.
  • Value: โญโญโญโญโญ Solves the longstanding latency vs. faithfulness dilemma in trustworthy RAG, providing an immediately deployable blueprint for verifiable enterprise AI.