Does Faithfulness-Guided Alignment Hurt Accuracy? Unlocking Accurate and Faithful Post-Retrieval Reasoning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Ameame1/CRAFT
Area: LLM Reasoning
Keywords: Retrieval-Augmented Generation, Multi-hop QA, Faithfulness Alignment, Reinforcement Learning, Process Supervision
TL;DR¶
Addressing the "right-answer-wrong-reason" shortcut dilemma in multi-hop retrieval-augmented generation (RAG), this paper presents CRAFT, a reinforcement learning framework that couples structured, machine-auditable reasoning traces with a dual-reward design (deterministic rules plus decoupled LLM semantic judge auditing), demonstrating that faithfulness alignment promotes rather than sacrifices task accuracy from 1.5B parameters upward.
Background & Motivation¶
Retrieval-augmented generation (RAG) grounds large language models (LLMs) in retrieved documents and serves as the prevailing paradigm for multi-hop question answering (QA), where solving complex queries requires composing interdependent evidence across passages—such as resolving a bridge entity and associating its attributes with downstream facts. However, in realistic retrieval scenarios with significant noise, retrieved document sets inevitably contain distractors that share high surface-level n-gram overlap with the query. Standard post-retrieval alignment predominantly optimizes outcome-level answer correctness, leaving the generation policy prone to exploiting spurious shortcuts. Consequently, models frequently exhibit severe "right-answer-wrong-reason" failures: they produce the correct final target entity while the supporting free-form rationale hallucinates nonexistent links, misattributes entities, or cites entirely irrelevant distractor passages. Such flaws remain completely invisible to black-box outcome evaluation metrics, substantially jeopardizing the reliability and transparency of RAG deployments in safety-critical domains.
Prior research tackling this challenge broadly falls into three categories, yet none directly resolves post-retrieval reasoning fidelity. First, chain-of-thought (CoT) prompting and question decomposition elicit step-by-step rationales, but these free-text traces lack formal binding to retrieved context; under distractor noise, models easily hallucinate plausible-sounding explanations that bypass the actual evidence. Second, post-hoc citation and attribution frameworks assess evidence support or claim verification, yet they operate almost exclusively on completed generations or final answers, leaving intermediate multi-hop deductions hard to audit at scale. Third, reinforcement learning for RAG primarily optimizes retrieval decision trajectories or final response correctness rather than enforcing semantic faithfulness over a fixed evidence corpus. These limitations have sustained a longstanding concern across the community: whether enforcing rigorous faithfulness constraints inherently hurts final task accuracy.
The central insight of this work is that prior faithfulness penalties degraded accuracy primarily because unstructured, free-form traces make semantic consistency verification fragile and prone to degeneration; structuring reasoning into an explicit, verifiable "chain of faithfulness" allows planning, evidence commitment, deductive reasoning, and answer extraction to mutually constrain one another. Core idea: propose CRAFT (Calibrated Reasoning with Answer-Faithful Traces), a reinforcement learning framework using Group Relative Policy Optimization (GRPO) that couples a family of configurable XML-structured output templates with a dual-reward auditor combining deterministic constraints (format compliance, citation coverage, normalized answer match) and decoupled LLM-judge semantic auditing to unlock accurate and faithful post-retrieval reasoning.
Method¶
Overall Architecture¶
CRAFT focuses specifically on the response-generation stage of RAG-based LLMs after candidate documents have been fetched. Given a question \(q\), a retrieved document set \(D = \{d_1, \dots, d_K\}\) combining gold supporting documents \(S^* \subseteq \{1, \dots, K\}\) with distracting passages, and a ground-truth answer set \(A^*\), the goal is to optimize a policy \(\pi_\theta\) that generates a trace \(y = (\pi, E, \rho, a)\) satisfying structural parsability, exact answer matching, and semantic reasoning faithfulness. The CRAFT framework integrates three complementary pillars: a family of structured trace variants (CRAFTv1–CRAFTv5) spanning an auditability spectrum, a decomposed reward auditor delivering multi-signal supervision, and an algorithmic faithfulness audit principle enabling fine-grained, machine-checkable verification.
The end-to-end execution pipeline flows seamlessly: the model first decodes structured XML traces containing sub-question planning, evidence indices, step-by-step reasoning, and extracted answers; deterministic checkers immediately verify schema compliance, citation precision, and answer correctness; concurrently, an independent LLM judge conducts four-dimensional binary consistency checks over the parsed segments, before GRPO computes group-relative advantage normalization to perform token-level clipped updates.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Question q + Mixed Documents D"] --> B["Stage 1: Structured Reasoning Trace Generation<br/>Output plan / gold_docs / reason / answer"]
B --> C["Stage 2: Deterministic Rule Constraints<br/>Strict schema validation + Citation and answer match"]
C --> D["Stage 3: Decoupled LLM Semantic Judge<br/>4-dimensional cross-field consistency audit"]
D --> E["Stage 4: GRPO Group-Normalized Policy Optimization<br/>Critic-free token-level clipped advantage update"]
E --> F["Output: Accurate & Faithful Auditable Reasoning Policy"]
Key Designs¶
1. Structured Reasoning Trace Generation: Unconstrained free-form reasoning traces lack auditability and facilitate hallucinated shortcuts -> Define an auditable XML output template family decomposing generation into sequential constraints -> Enforce a verifiable chain of faithfulness
To replace opaque free-form generation with machine-checkable derivations, CRAFT introduces an XML-structured template family spanning an auditability spectrum. In full CRAFTv1, the output is factorized into four sequential fields establishing a chain of faithfulness: <plan> specifies the sub-question decomposition blueprint; <gold_docs> pre-commits to the candidate evidence indices \(E \subseteq \{1, \dots, K\}\) prior to reasoning; <reason> is constrained to deduce step-by-step claims strictly grounded within the declared boundary \(E\) while addressing the plan; and <answer> must be logically entailed by the reasoning block. Intermediate variants explore auditability trade-offs: CRAFTv2 ablates <plan>, CRAFTv3 removes <gold_docs>, CRAFTv4 retains only <reason> and <answer>, and CRAFTv5 serves as a direct answer-only baseline. This progressive structure guarantees that unfaithful deductions or distractor-induced shortcut leaps manifest as verifiable constraint violations across adjacent fields.
2. Deterministic Rule Constraints: Relying solely on black-box LLM scoring causes high variance and formatting collapse -> Integrate rule-based checks for schema validity, citation set coverage, and normalized answer exact-match -> Establish low-latency and objective process supervision
CRAFT balances semantic checks with three deterministic reward functions computed directly from output structure and ground-truth annotations without LLM involvement. First, format compliance \(R_{\mathrm{fmt}}(y) = \mathbb{I}[y \in \mathcal{V}_v]\) strictly validates whether trace \(y\) adheres to the expected XML schema and tag sequence for variant \(v\), acting as a mandatory gatekeeper that zeroes out subsequent citation and answer rewards if breached. Second, citation set reward \(R_{\mathrm{gold}}(y)\) assesses the predicted document indices \(E\) against gold supports \(S^*\): it awards 1.0 for an exact match (\(E = S^*\)), 0.5 for partial non-empty overlap (\(E \cap S^* \neq \emptyset \wedge E \neq S^*\)), and 0.0 for disjoint sets (\(E \cap S^* = \emptyset\)), discouraging the citation of distractors. Third, answer correctness reward \(R_{\mathrm{ans}}(y, A^*)\) checks whether the normalized string extracted from <answer> matches any ground-truth alias in \(A^*\) after stripping punctuation, articles, underscores, and extra whitespace.
3. Decoupled LLM Semantic Judge: Open-ended natural language reasoning cannot be supervised by exact string rules -> Introduce decomposed binary verification criteria for cross-stage consistency and claim attribution -> Provide dense, calibrated process-level faithfulness signals
Direct cross-entropy supervision on open-ended reasoning chains tends to induce brittle memorization and format collapse. CRAFT therefore employs an external high-capacity judge (Qwen3-30B-A3B) to evaluate intermediate semantic fidelity. Rather than scoring on an uncalibrated scalar rubric, the judge performs four decomposed binary evaluations: (i) plan adherence \(c_\pi \in \{0, 1\}\) (does reasoning follow sub-questions in plan order?), (ii) citation boundary adherence \(c_E \in \{0, 1\}\) (are all citations in reasoning confined to declared indices in <gold_docs>?), (iii) answer entailment \(c_a \in \{0, 1\}\) (is the extracted answer a valid logical conclusion of reasoning?), and (iv) evidence grounding \(c_g \in \{0, 1\}\) (are all fact claims in reasoning verifiable against the cited document texts?). During training, the arithmetic mean of applicable checks supplies a dense process reward:
$\(R_{\mathrm{faith}}(y) = \frac{1}{m_v} \sum_{k=1}^{m_v} r_k(y) \in [0, 1]\)$
At evaluation time, the judge emits a holistic binary overall-consistency verdict \(F_{\mathrm{eval}}(y) \in \{0, 1\}\). Human expert validation on 500 MuSiQue samples demonstrated a 93.0% macro agreement rate (Cohen's \(\kappa = 0.78\)) with the judge.
4. GRPO Group-Normalized Policy Optimization: Traditional Actor-Critic architectures incur excessive memory overhead for long reasoning sequences -> Adopt critic-free group-relative advantage estimation with token-level clipping -> Stabilize multi-objective alignment under combined reward signals
To optimize the combined objective \(R(y) = \sum_{c \in C_v} w_c R_c(y)\) (where all weights \(w_c = 1.0\)), CRAFT leverages Group Relative Policy Optimization (GRPO), eliminating the separate critic network. For each input \(x = (q, D, v)\), the policy generates a group of \(G=8\) rollouts \(\{y_i\}_{i=1}^G\). Advantages are normalized within the group via \(\hat{R}_i = (R(y_i) - \mu) / (\sigma + \epsilon)\), where \(\mu\) and \(\sigma\) denote group mean and standard deviation. The policy parameters are updated via token-level clipped surrogate optimization bounded by a reference policy KL divergence penalty:
$\(\ell_{\mathrm{clip}, i, t}(\theta) = \min \left( r_{i,t}(\theta) \hat{R}_i, \operatorname{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{R}_i \right)\)$
with clipping parameter \(\varepsilon=0.2\) and KL coefficient \(\beta=0.04\). By offloading value network overhead, GRPO enables efficient distributed training of 7B policies on 4 H100 GPUs alongside 4 co-located H100 GPUs serving local judge inferences.
Loss & Training¶
CRAFT is implemented using the MS-Swift framework with vLLM acceleration on an 8-GPU node of NVIDIA H100 96GB GPUs (4 GPUs for policy rollouts and updates, 4 GPUs hosting Qwen3-30B-A3B). The training mixture comprises 20,000 multi-hop instances: 10,000 from MuSiQue, 5,000 from HotpotQA, and 5,000 from 2WikiMultiHopQA. Each update step processes 64 prompts with 8 generations per prompt (512 effective completions per step) across 312 training steps. Training runs under bfloat16 precision with Flash Attention 2. Learning rates are set to \(5 \times 10^{-6}\) for 0.5B–3B scales and \(3 \times 10^{-6}\) for 7B, scheduled via Cosine decay with 10% warmup. Supervised fine-tuning (SFT) baselines are trained on the same corpus with cross-entropy loss at learning rate \(2 \times 10^{-5}\) for 3 epochs.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on HotpotQA, 2WikiMultiHopQA, and MuSiQue (1,000 test samples per benchmark, averaged over 10 runs), evaluating open-source base models, commercial APIs, SFT models, and CRAFT (v1 full trace) across parameter scales.
| Model | Type | MuSiQue (EM / F1 / Faith.) | HotpotQA (EM / F1 / Faith.) | 2WikiMHQA (EM / F1 / Faith.) |
|---|---|---|---|---|
| Qwen2.5-0.5B Base | Pre-Trained | 0.25 / 1.32 / 0.19 | 1.76 / 5.58 / 1.69 | 6.02 / 9.95 / 1.64 |
| Qwen2.5-1.5B Base | Pre-Trained | 0.91 / 3.63 / 5.13 | 9.96 / 17.28 / 14.63 | 10.10 / 15.95 / 9.22 |
| Qwen2.5-3B Base | Pre-Trained | 12.03 / 19.62 / 24.28 | 32.61 / 44.34 / 56.00 | 30.62 / 38.65 / 44.59 |
| Qwen2.5-7B Base | Pre-Trained | 35.83 / 47.99 / 58.47 | 56.47 / 71.58 / 83.30 | 57.87 / 66.56 / 76.92 |
| Qwen3-4B Base | Pre-Trained | 24.11 / 32.40 / 52.68 | 51.30 / 65.58 / 87.81 | 54.24 / 64.33 / 86.20 |
| Qwen3-30B-A3B Base | Pre-Trained | 40.12 / 48.95 / 68.80 | 59.57 / 74.24 / 92.89 | 55.73 / 66.19 / 90.78 |
| GPT-5-mini | API | 53.59 / 68.42 / 83.42 | 61.00 / 80.22 / 96.43 | 70.47 / 81.19 / 95.98 |
| DeepSeek-V3.2 | API | 53.70 / 67.81 / 76.89 | 63.56 / 79.51 / 94.31 | 69.20 / 77.74 / 96.80 |
| Gemini-2.5-Flash | API | 52.37 / 63.62 / 77.55 | 63.60 / 78.80 / 93.41 | 69.76 / 79.35 / 94.42 |
| Qwen2.5-7B SFT | Trained | 31.01 / 41.78 / 51.22 | 57.81 / 71.30 / 83.37 | 62.69 / 71.50 / 79.00 |
| CRAFT-1.5B (Ours) | Trained | 26.87 / 36.00 / 28.35 | 46.35 / 59.47 / 60.27 | 43.57 / 51.19 / 48.46 |
| CRAFT-3B (Ours) | Trained | 41.44 / 49.96 / 58.69 | 56.19 / 69.92 / 78.52 | 56.75 / 63.45 / 62.11 |
| CRAFT-7B (Ours) | Trained | 53.13 / 62.97 / 83.87 | 64.06 / 78.86 / 96.31 | 78.23 / 84.18 / 96.79 |
Ablation Study¶
Systematic ablations were performed on Qwen2.5-7B across trace templates (CRAFTv1 through CRAFTv5) and judge reward supervision (\(R_{\mathrm{faith}}\)).
| Configuration | MuSiQue (EM / F1 / Faith.) | HotpotQA (EM / F1 / Faith.) | 2WikiMHQA (EM / F1 / Faith.) | Note |
|---|---|---|---|---|
| CRAFTv1 Full | 53.13 / 62.97 / 83.87 | 64.06 / 78.86 / 96.31 | 78.23 / 84.18 / 96.79 | Full 4-field trace with dual reward |
| CRAFTv1 w/o judge | 51.53 / 60.88 / 81.96 | 62.49 / 76.86 / 94.81 | 76.47 / 82.18 / 94.67 | Rule rewards only; consistent drops |
| CRAFTv1 SFT | 31.01 / 41.78 / 51.22 | 57.81 / 71.30 / 83.37 | 62.69 / 71.50 / 79.00 | Supervised baseline degrades in fidelity |
| CRAFTv2 Full (no plan) | 51.31 / 63.06 / 82.49 | 65.28 / 80.10 / 96.38 | 75.26 / 82.51 / 94.40 | Removes plan; peaks on HotpotQA EM |
| CRAFTv3 Full (no cite) | 51.86 / 62.37 / 84.60 | 63.81 / 78.55 / 96.23 | 76.62 / 83.23 / 95.97 | Removes gold_docs; high MuSiQue faith |
| CRAFTv4 Full (reason only) | 48.02 / 59.94 / 83.17 | 62.64 / 77.84 / 94.96 | 75.60 / 82.04 / 92.90 | Compact derivation (avg 97.5 tokens) |
| CRAFTv5 (answer only) | 49.95 / 60.51 / - | 59.80 / 74.79 / - | 60.34 / 67.72 / - | Direct answer generation without trace |
Key Findings¶
- Faithfulness alignment enhances rather than diminishes accuracy: On Qwen2.5-7B, full CRAFTv1 improves EM over the base model by +17.30 on MuSiQue, +7.59 on HotpotQA, and +20.36 on 2WikiMHQA, while boosting Faithfulness by 13.01–25.40 percentage points. Furthermore, CRAFT-7B surpasses competitive commercial APIs (GPT-5-mini, DeepSeek-V3.2, Gemini-2.5-Flash) in EM on HotpotQA and 2WikiMHQA.
- Judge reward serves as an indispensable semantic regularizer: Removing the LLM judge reward (\(R_{\mathrm{faith}}\)) caused performance drops across all 12 evaluated 7B cells. The full reward configuration yielded an average gain of +1.75 EM, +1.89 F1, and +1.89 Faithfulness points over the w/o-judge setup, proving that the judge prevents the policy from converging to shortcut reasoning paths.
- Capacity threshold and template sensitivity at smaller scales: Multi-hop structured reasoning unlocks from 1.5B parameters upward. Conversely, the 0.5B model collapses to near-zero accuracy under the complex CRAFTv1 schema (0.00% EM on MuSiQue; format compliance 24.97%), but prospers under the direct answer-only CRAFTv5 variant (23.87%–36.70% EM; 100% format compliance). Sub-1B models exhibit capacity bottlenecks that favor minimal template overhead.
- Strong out-of-distribution transfer: Evaluated on the held-out MultiHopRAG benchmark built on a distinct news domain, CRAFT7B achieved +6.86 EM, +7.18 F1, and +29.71 Faithfulness gains over the base model, demonstrating genuine transfer of structured causal deduction rather than dataset-specific memorization.
Highlights & Insights¶
- Deconstructing ungrounded reasoning into verifiable chains: By sequencing
<plan>\(\rightarrow\)<gold_docs>\(\rightarrow\)<reason>\(\rightarrow\)<answer>, CRAFT transforms open-ended semantic faithfulness into machine-checkable local dependencies, forcing the model to explicitly commit to evidence boundaries before generating claims. - Hybrid reward formulation mitigating judge gaming and format failure: Assigning schema adherence, set overlap, and exact answers to fast deterministic rules while reserving decomposed binary checks for claim verifiability eliminates judge reward hacking and accelerates RL training stability.
- Tunable auditability vs. efficiency trade-off: The trace family offers concrete deployment operating points—from 11.3 tokens (v5 answer-only) and 88.5 tokens (v2 compact plan-free) to 144.6 tokens (v1 comprehensive auditability)—enabling seamless adaptation to inference budgets.
Limitations & Future Work¶
- Author-acknowledged constraints: Human validation was restricted to 500 samples from MuSiQue due to the steep cost of expert annotations; compute requirements for online GRPO with a 30B judge (adding 22.6% latency overhead, ~2.16 hours) capped policy scale at 7B parameters and 20K training examples.
- Post-retrieval isolation: CRAFT operates under the assumption that documents are pre-retrieved and frozen, without updating the retriever or retrieval search policy. If retrieval fails completely to retrieve relevant context, the post-retrieval generator cannot dynamically issue secondary searches.
- Future directions: Integrating CRAFT's dual-reward paradigm into multi-agent interleaved search-and-reason loops, and distilling the 30B judge into a lightweight dedicated reward verifier to slash training overhead.
Related Work & Insights¶
- vs. Post-hoc verification methods (RAGAs, Generate-but-Verify): Prior work performs quality screening or attribution checks after generation completes; CRAFT directly internalizes faithfulness criteria as process-level rewards within GRPO reinforcement learning to teach the policy self-grounded reasoning.
- vs. RAG reinforcement learning (Self-RAG, R3-RAG): Existing frameworks optimize retrieval triggers or interleaved search steps; CRAFT targets the post-retrieval generation phase, solving distractor-induced "right-answer-wrong-reason" failures under fixed candidate document sets.
- vs. Prompting strategies (CoT, Least-to-Most): Standard CoT lacks structural evidence boundaries; CRAFT enforces prior document commitment and cross-stage consistency, preventing models from exploiting plausible-sounding distractor rationales.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Rigorously formulates and dispels the accuracy-faithfulness trade-off in post-retrieval RAG reasoning via a chain-of-faithfulness template family and dual-reward design.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-scale evaluation (0.5B–7B), multi-template ablations, base/SFT/RL comparisons, OOD benchmark testing, and 500-instance human correlation validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Lucid narrative flow, clear visual and algorithmic formalizations, and honest analysis of small-model capacity bottlenecks.
- Value: ⭐⭐⭐⭐⭐ Delivers an open-source, highly effective RL recipe for high-assurance, auditable RAG reasoning systems.