Skip to content

FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/cxcscmu/FaithMed
Area: LLM Reasoning
Keywords: Evidence-based medicine reasoning, Process reward model, Reinforcement learning, Step-level credit assignment, Faithful evidence grounding

TL;DR

This paper introduces FaithMed, a reinforcement learning framework that formalizes five evidence-based medicine (EBM) principles into 15 refined checklist rubrics, employing anchor-state advantage grouping and step-level process reward modeling to train LLMs for faithful evidence retrieval, critical appraisal, and patient-specific reasoning—achieving a 9.0% average accuracy improvement over agentic-search baselines across seven medical benchmarks and boosting EBM rubric compliance by 15.5%.

Background & Motivation

In high-stakes clinical decision-making, large language models (LLMs) cannot merely act as black-box predictors of diagnostic labels or drug choices; their internal reasoning pathways must remain transparent, medically valid, and rigorously grounded in trustworthy clinical evidence. Real-world evidence-based medicine (EBM) follows a systematic clinical protocol wherein practitioners synthesize detailed patient history, physical findings, and authoritative clinical guidelines to formulate interventions while balancing clinical benefits against adverse risks. However, contemporary medical LLMs predominantly rely on internal parametric memory to perform intuitive, unconstrained heuristic reasoning, which frequently introduces cognitive leaps, specious rationalizations, and catastrophic medical hallucinations. Even when an LLM happens to guess the correct multiple-choice option, its intermediate reasoning trace may cite non-existent physiological mechanisms or draw clinically erroneous conclusions.

To mitigate hallucination and enhance factual grounding, recent research has actively explored retrieval-augmented generation (RAG) and agentic multi-turn search paradigms, empowering models to query medical corpora like PubMed or clinical guideline databases. Nevertheless, standard RAG and reinforcement learning frameworks suffer from acute process-level unfaithfulness. Outcome-only reinforcement learning (such as vanilla GRPO) provides scalar reward signals based exclusively on terminal answer correctness, inadvertently distributing credit to reasoning trajectories that arrived at the right choice through fundamentally flawed inferences. Furthermore, existing process reward models (PRMs) generally rely on unconstrained scalar judgments or post-hoc trajectory reranking, which fail to penalize decorative citations, hallucinated relevance, or a model's failure to critically appraise retrieved passages against specific patient demographics.

Bridging this gap requires translating real-world evidence-based medicine workflows—specifically the five core dimensions of Ask, Acquire, Appraise, Apply, and Assess—into granular, discriminative, and verifiable step-level evaluation criteria, and directly coupling these criteria with localized policy gradient updates. Core idea: formalize evidence-based medicine principles into fine-grained checklist rubrics through metric-driven iterative refinement, and optimize medical agent reasoning via step-level process reward modeling paired with anchor-state advantage grouping to enforce rigorous evidence retrieval, critical appraisal, and patient-specific synthesis.

Method

Overall Architecture

FaithMed frames clinical reasoning as a multi-turn agent–environment interaction under a Partially Observable Markov Decision Process. Given a clinical question \(x\) establishing the initial state \(s_1\), the policy at turn \(k\) inspects the running observation context \(s_k\) and generates an intermediate thought trace and action tuple \(y_k = (t_k, a_k)\), where \(a_k \in \{\text{search}, \text{answer}\}\). Executing a search action retrieves external medical snippets \(e_k = \{e_{k,j}\}\) from a corpus index, wrapped inside <information> tags with structured identifiers [Tk-Ej], which updates the environment to \(s_{k+1}\); executing an answer action yields the final clinical conclusion and terminates the episode.

The FaithMed pipeline operates across three foundational modules: first, clinical experts draft seed criteria covering the five EBM dimensions, which are refined through metric-driven recursive pruning and rewriting into 15 discriminative binary checklist rubrics; second, for each reasoning step, an applicable rubric subset is dynamically assigned based on local action semantics, scored by a batched LLM judge, centered within its applicable subset, and normalized across anchor-state groups sharing similar retrieved evidence; third, the step-level process advantage is blended with the trajectory-level outcome advantage to guide online GRPO policy optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Clinical Case Input x"] --> B["Metric-Driven Rubric Refinement<br/>5 EBM Dimensions · Statistical Pruning & Rewriting"]
    B --> C["Multi-Turn Agent Environment<br/>ask / acquire / appraise / apply / assess Action Space"]
    C --> D["Step-Level Rubric Judging & Centering<br/>Context-Bound Rubric Assignment · Baseline Subtraction"]
    D --> E["Anchor-State Advantage Grouping<br/>Evidence-Similarity Clustering · Local Advantage Normalization"]
    E --> F["Dual Advantage Policy Optimization<br/>GRPO · Blended Step Process & Global Outcome Advantage"]
    F -->|Policy Update & Online Rollouts| C
    F --> G["Faithful Evidence-Based Medical Reasoning<br/>Verifiable Inline Citations & Transparent Decisions"]

Key Designs

1. Metric-Driven Rubric Refinement: Eliminating Redundancy and Ambiguity in Clinical Criteria

Directly using human-written clinical guidelines as reinforcement learning rewards often leads to reward gaming or unstable optimization due to overlapping criteria and ambiguous triggering conditions. To construct a reliable supervisory rubric set, FaithMed decomposes the five EBM dimensions into 26 checklist-style binary criteria following CheckEval principles. The system then executes an automated recursive refinement procedure over 300 teacher-generated medical QA reasoning trajectories. In each iteration, an LLM judge evaluates all trajectory–rubric pairs, computing three key statistics: non-applicability rate, MET rate, and pairwise Pearson correlation \(\rho\).

Based on these statistics, an automated LLM editor executes targeted refinements: (1) MERGE: any pair of rubrics exhibiting an absolute Pearson correlation \(|\rho| > 0.70\) is merged to eliminate double-counting of identical reasoning behaviors; (2) REMOVE: non-discriminative rubrics with near-constant scores (\(\text{MET} > 0.95\) or \(\text{MET} < 0.05\)) are discarded; (3) REWRITE: rubrics with non-applicability rates exceeding \(0.80\) are rewritten to clarify narrow or ambiguous preconditions. The refinement converges after three iterations, and a final review by a clinical expert establishes 15 core rubrics (exactly 3 per dimension across Ask, Acquire, Appraise, Apply, and Assess) that balance clinical rigor with empirical discriminativeness.

2. Step-Level Process Reward Assignment: Contextual Binding and Rubric-Level Centering

Global outcome-based rewards and episode-level process rewards suffer from severe credit assignment mismatch, incorrectly penalizing early search steps for downstream errors or rewarding flawed reasoning in lucky guesses. FaithMed establishes a context-bound dimension assignment: step 1 always includes Ask; search actions trigger Acquire; subsequent steps conditioned on retrieved evidence \(e_{k-1}\) include Appraise and Apply; and answer actions trigger Assess.

Given the applicable rubric subset \(R_k = \bigcup_{d \in D_k} R_d\), the LLM judge outputs a binary evaluation vector \(J_k \in \{0, 1\}^{|R_k|}\). Because natural rubric pass rates differ substantially, raw summation would bias policy learning toward easier criteria. FaithMed computes the mean pass rate \(\mu_{i,k,m}\) for each applicable rubric \(m\) within the local rollout group and applies mean-centering: $\(r^P_{i,k} = \sum_{m \in R_{i,k}} \left( J_m(y_{i,k}, e_{i,k-1}, R_{i,k}) - \mu_{i,k,m} \right)\)$ This ensures that intermediate steps are credited solely for exceeding the baseline quality expected under the same clinical context.

3. Anchor-State Advantage Grouping: Context-Normalized Local Comparison

In interactive multi-turn search, different candidate trajectories inevitably retrieve heterogeneous document snippets. Comparing reasoning steps across disparate evidence sets introduces substantial variance into policy gradients. FaithMed adapts the GiGPO framework to construct anchor-state groups \(\mathcal{G}(\tilde{s})\), where the anchor state \(\tilde{s}_{i,1} = x\) for initial steps, and \(\tilde{s}_{i,k} = e_{i,k-1}\) for subsequent retrieval-conditioned steps. Using a fuzzy string similarity threshold \(\eta = 0.8\), steps with comparable retrieved contexts are clustered together: $\(\mathcal{G}(\tilde{s}) = \{(j, l) : \text{sim}(\tilde{s}_{j,l}, \tilde{s}) \ge \eta\}\)$

The step-level process advantage is calculated by standardizing the centered process rewards exclusively within each anchor group: $\(A^P_{i,k} = \frac{r^P_{i,k} - \text{mean}(\{r^P_{j,l} : (j,l) \in \mathcal{G}(\tilde{s}_{i,k})\})}{\text{std}(\{r^P_{j,l} : (j,l) \in \mathcal{G}(\tilde{s}_{i,k})\}) + \epsilon}\)$ Consequently, an agent's search or appraisal action is reinforced when it outperforms alternative rollouts facing the same clinical information barrier, isolating reasoning quality from retrieval luck.

4. Blended Dual-Advantage Policy Optimization: Unifying Task Correctness with EBM Rigor

Over-optimizing process rewards in isolation risks producing verbose, superficial clinical jargon that fails to resolve the patient's core diagnostic question. Conversely, pure outcome supervision invites unfaithful shortcuts. FaithMed unifies both supervisory streams by constructing a composite advantage for step \(k\) in trajectory \(i\): $\(A_{i,k} = A^O_i + \lambda A^P_{i,k}\)$ where \(A^O_i = \frac{R^O(\tau_i) - \text{mean}(\{R^O(\tau_j)\})}{\text{std}(\{R^O(\tau_j)\})}\) represents the group-normalized episode outcome advantage from standard GRPO, and \(\lambda\) balances process guidance against task success (\(\lambda = 0.05\) identified as optimal).

The composite advantage \(A_{i,k}\) is optimized using the clipped GRPO objective across all tokens generated in step \(y_{i,k}\), constrained by a reference-policy KL penalty. This end-to-end objective simultaneously elevates answer accuracy and enforces clinical fidelity across the entire reasoning trajectory.

Loss & Training

  • Training Pipeline:
  • Supervised Fine-Tuning (SFT): 2,650 clinical QA questions sampled from MedQA, MedMCQA, HeadQA, and MedCalc-Bench. Using DeepSeek-R1 with agentic search prompts, 15,900 valid, format-compliant trajectories yielding correct answers were collected (28,368 step-level instances). Qwen3 backbones were fine-tuned for 1 epoch (learning rate \(5 \times 10^{-5}\), cosine decay, 1% linear warmup).
  • Reinforcement Learning (RL): Built on verl-agent with batch size 32, 8 rollouts per prompt, for 300 optimization steps. Gemini-2.5-Flash-Lite acts as the step-level batched LLM judge. Max sequence length is 3072 tokens.
  • Retrieval Infrastructure: Five authoritative medical corpora—PubMed, StatPearls, medical textbooks, Wikipedia (from MedRAG), and MediTron-70B clinical guidelines—segmented and encoded with MiniCPM-Embedding-Light via Tevatron, indexed with DiskANN for fast nearest neighbor vector lookup.

Key Experimental Results

Main Results

FaithMed was evaluated across seven medical benchmarks covering knowledge-intensive QA (MedQA, MedMCQA, HeadQA, MMLU-Pro-Health), expert clinical reasoning (MedXpertQA), calculation (MedCalc-Bench), and application QA (MedBullets). Results (accuracy %) are summarized below († denotes in-distribution, * denotes out-of-distribution):

Method / Backbone Scale MedQA† MedMCQA† HeadQA† MedCalc† MedBullets* MMLU-Pro* MedXpert* Overall Acc.
Llama 3.1 Instruct 8B 61.8 57.3 67.6 21.4 52.2 56.0 11.8 46.9
Gemma 3 12B 69.0 59.0 73.0 27.0 51.0 59.0 16.4 50.6
Qwen3 8B 79.4 66.3 83.4 44.9 61.8 69.7 19.2 60.7
DeepSeek-R1 - 94.2 72.8 93.4 72.7 73.2 74.2 42.9 74.8
Qwen3 Base 1.7B 48.9 48.0 66.2 16.9 35.7 43.4 9.3 38.3
MedRAG 1.7B 51.4 51.3 67.5 8.4 40.8 48.5 14.2 40.3
Agentic Search 1.7B 49.2 46.8 65.5 15.3 36.0 46.4 11.1 38.6
Agentic Search + GRPO 1.7B 50.9 50.1 68.5 27.1 31.0 48.1 11.5 41.0
FaithMed 1.7B 61.4 56.9 75.6 33.2 38.5 52.0 17.1 47.8
Qwen3 Base 4B 69.3 59.7 81.3 13.7 46.3 63.0 16.0 49.9
MedRAG 4B 70.2 59.8 83.9 10.3 52.1 55.5 19.0 50.1
Agentic Search 4B 70.6 63.4 81.5 29.6 47.3 55.4 19.6 52.5
Agentic Search + GRPO 4B 71.3 63.2 84.4 39.0 49.8 66.1 22.0 56.5
FaithMed 4B 74.8 64.1 85.7 56.5 55.6 63.5 28.2 61.2

Evaluating EBM process rubric compliance (scores 0–100) demonstrates dramatic gains across all five clinical dimensions:

Model Ask Acquire Appraise Apply Assess Macro Avg.
DeepSeek-R1 (Teacher) 94.7 52.7 48.5 46.2 14.6 51.3
Qwen3-4B Base 87.0 30.2 26.1 23.3 9.0 35.1
Qwen3-4B + Ep. PR 86.7 41.4 36.4 34.0 12.5 42.2
FaithMed-4B (Step PR) 90.4 53.9 48.5 41.4 19.1 50.7

Ablation Study

Training recipe ablations across SFT, outcome rewards (OR), episode-level process rewards (Ep. PR), and step-level process rewards (Step PR):

Recipe Variant SFT OR PR Type Qwen3-1.7B Acc. Qwen3-4B Acc.
Base Agentic Search × × None 38.6 52.5
Outcome-Only GRPO × ✓ None 41.0 (+2.4) 56.5 (+4.0)
Episode-Level PR × ✓ Ep. 41.5 (+2.9) 56.8 (+4.3)
FaithMed (Step-Level PR) × ✓ Step 47.8 (+9.2) 61.2 (+8.7)
SFT-Only ✓ × None 41.0 54.2
SFT + Outcome-Only GRPO ✓ ✓ None 46.0 (+5.0) 57.4 (+3.2)
SFT + Episode-Level PR ✓ ✓ Ep. 43.8 (+2.8) 57.1 (+2.9)
SFT + FaithMed (Step-Level) ✓ ✓ Step 47.1 (+6.1) 59.5 (+5.3)

Dimension ablations on FaithMed 1.7B: - w/o Acquire: Acquire drops by -8.28, causing cascading drops in Appraise (-5.92) and Assess (-4.17). - w/o Appraise: Appraise drops by -9.35, and Assess drops by -2.48. - w/o Apply: Apply score plunges by -8.51. - w/o Assess: Assess drops by -8.44, and Appraise falls by -2.92.

Process reward weight (\(\lambda\)) sensitivity on Qwen3-1.7B: - \(\lambda = 0\): 41.0% | \(\lambda = 0.01\): 43.8% | \(\lambda = 0.05\): 47.8% | \(\lambda = 0.5\): 39.9% (excessive weighting degrades final task performance).

Key Findings

  • Step-level granularity is essential: Step-level process rewards deliver a 9.2-point gain on 1.7B and an 8.7-point gain on 4B without SFT. In contrast, episode-level PR yields minor gains and actually degrades accuracy when applied after SFT (46.0% down to 43.8%), driven by severe off-step credit assignment mismatches.
  • Process supervision drives task success: Rather than trading accuracy for transparency, enforcing clinical EBM behaviors raises task success; FaithMed-4B (61.2%) surpasses the much larger Qwen3-8B model (60.7%).
  • Interdependent reasoning stages: The ablation of earlier dimensions (Ask, Acquire) impairs downstream dimensions (Appraise, Assess), proving that evidence-based clinical reasoning forms a tightly coupled, cumulative decision chain.

Highlights & Insights

  • Metric-driven self-refining checklist rubrics: FaithMed avoids arbitrary prompt engineering by using non-applicability, pass rate, and Pearson correlation to systematically merge, remove, and rewrite criteria into 15 robust, discriminative EBM rubrics.
  • Anchor-state clustering for fair local comparisons: Grouping reasoning steps by observation context similarity prevents agents from being unfairly penalized for uninformative retrievals, stabilizing policy gradient signals.
  • Exposing the flaws of outcome-only RL in high-stakes reasoning: Standard RL rewards lucky guesses despite faulty logic. FaithMed establishes that fine-grained process constraints are vital to prevent models from learning deceitful shortcuts.

Limitations & Future Work

  • Constrained to textual multiple-choice benchmarks: The evaluation focuses on standardized textual QA, omitting real-world complexities such as longitudinal EHR data, dense physiological telemetry, multi-modal imaging (CT/MRI), and conversational nuance.
  • Judge bias and alignment limits: The framework relies on Gemini-2.5-Flash-Lite as an automated proxy; despite 80% inter-annotator agreement with Gemini 3, models could potentially overfit to the linguistic quirks of the LLM judge.
  • Static knowledge corpus: Experiments rely on a pre-indexed vector pool. Extending FaithMed to live web queries and dynamic, updating medical databases remains an open direction.
  • vs MedRAG / MedCite: Traditional medical RAG focuses on static pre-retrieval context injection or citation generation. FaithMed trains models to actively query, critically evaluate, and contextualize evidence across dynamic multi-turn interactions.
  • vs MedPrompt / Huatuo-o1: Prior medical reasoning work employs chain-of-thought prompting or rejection sampling. FaithMed directly injects EBM rubrics into online RL, eliminating the need for expensive test-time search.
  • vs Med-PRM / MedS3: Existing medical PRMs train standalone verifiers used for best-of-4 inference reranking. FaithMed bakes EBM-aligned process rewards directly into policy optimization, surpassing best-of-4 PRM baselines in single-pass inference.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formalizes real-world EBM principles into refined process rubrics with anchor-state advantage grouping and step-level RL.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across seven medical benchmarks, deep ablations of training recipes and dimensions, and detailed error attribution diagnostics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptional clarity, mathematically grounded formulations, and well-structured empirical narratives.
  • Value: ⭐⭐⭐⭐⭐ Provides a foundational blueprint for transforming clinical AI assistants from heuristic guessers into rigorous, evidence-based reasoning agents.