Skip to content

auditing evidence use in medical llm diagnosis

content_hash: 3f51fe36948f743e ---# Auditing Evidence Use in Medical LLM Diagnosis

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Rosita777/medical-llm-evidence-audit
Area: Medical NLP
Keywords: Medical LLMs / Diagnostic Evidence Audit / Interaction Mining / Counterfactual Evaluation / Differential Diagnosis

TL;DR

Addressing the limitation where diagnostic accuracy hides improper clinical reasoning, this paper presents a behavioral evidence-use audit via controlled subset interventions and diagnostic margin interaction mining, establishing a diagnosis-relative framework with multi-stage stability filtering to isolate true evidence failures.

Background & Motivation

Large language models have demonstrated impressive performance on medical examination and case-based diagnostic benchmarks, frequently rivaling expert clinicians in option-selection accuracy. However, aggregate accuracy provides a misleading surface signal. In high-stakes clinical settings, a model can easily select the correct diagnosis by exploiting spurious shortcuts—such as explicit disease mentions in family history or irrelevant anatomical findings—while overweighting misleading evidence or underweighting decisive findings. Clinical evidence is fundamentally not a flat, homogeneous list of features; symptoms, physical signs, risk factors, negative exclusions, and medical history play heterogeneous roles across competing candidate diseases.

The core tension in auditing clinical diagnosis lies in the "diagnosis-relative" nature of medical evidence. A given negative finding may serve as decisive exclusionary evidence for one condition while remaining completely benign or irrelevant to a competitor. Furthermore, overlapping clinical findings often exhibit redundancy rather than additive support, manifesting mathematically as negative or non-additive interactions. Existing interpretability methods either depend on fragile, post-hoc rationales and chain-of-thought tokens, or deploy generic feature-attribution and counterfactual perturbations that treat evidence as globally unwanted or irrelevant—failing to capture the nuanced logic of differential diagnosis.

This paper bypasses the challenge of recovering latent internal reasoning paths, adopting instead a rigorous, controlled behavioral auditing methodology. By decomposing clinical cases into discrete evidence units and observing observable diagnostic margin shifts across controlled subset interventions, the authors systematically mine and characterize low-order interactions among evidence units beyond individual feature contributions. Core idea: quantify candidate diagnostic margin changes under controlled subset interventions, decouple interaction discovery from error assignment via diagnosis-relative evidence roles, and confirm true clinical evidence misuse through multi-dimensional stability filtering and targeted counterfactuals.

Method

Overall Architecture

The proposed evidence-use auditing framework operates through four sequential stages: case evidence decomposition and controlled intervention generation, option log-probability diagnostic margin scoring, low-order evidence interaction mining, and role-aware mechanism attribution coupled with multi-axis stability validation. Taking a patient case and candidate differential diagnoses as input, the framework generates a structured diagnostic evidence-use profile that rigorously distinguishes clinically coherent differential reasoning from genuine shortcut exploitation and evidence misuse.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Clinical Case Input<br/>Patient findings E and candidate diagnoses C"] --> B["Controlled Evidence Interventions & Margin Scoring<br/>Subset interventions T and option log-probability margins"]
    B --> C["Low-Order Interaction Mining<br/>Calculate joint interaction effect I_d(S) for subset S"]
    C --> D["Diagnosis-Relative Role Assignment & Mechanism Labeling<br/>Classify roles: supporting, competitor, excluding, local"]
    D --> E["Multi-Dimensional Stability Filtering & Counterfactuals<br/>Prompt/order variants + neutralized perturbation + clinical review"]
    E --> F["Audited Diagnostic Profile<br/>Distinguish valid differentials from confirmed failures"]

Key Designs

1. Controlled Evidence Interventions & Diagnostic Margin Scoring: Quantifying Preference Shifts via Option Margin Functions

Recognizing that free-form generated explanations suffer from post-hoc rationalization and option probabilities are prone to global letter biases, the audit decomposes patient cases into evidence unit sets \(E = \{e_1, \dots, e_n\}\) and candidate sets \(C = \{c_1, \dots, c_m\}\). Under controlled interventions, the model is prompted with a selected subset of visible evidence \(T \subseteq E\) in a standardized multiple-choice format. Rather than sampling generated text, the audit extracts next-token log-probabilities \(s_T(c_j)\) for each candidate option letter \(c_j\). For any audited diagnosis \(d \in C\), its diagnostic margin under subset \(T\) is formulated as the relative preference over its closest competing diagnosis:

\[v_d(T) = s_T(d) - \max_{c_j \in C, c_j \ne d} s_T(c_j)\]

This margin formulation directly isolates within-prompt relative confidence, dampening token-prior biases and providing a coherent cooperative-game value function for combinatorial interaction analysis.

2. Low-Order Evidence Interaction Mining: Measuring Combinatorial Joint Effects Beyond Singletons

In clinical practice, isolated symptoms are rarely definitive; rather, disease syndromes emerge from complex combinations of clinical findings. To capture non-additive interactions among features, the audit adopts interaction indices from cooperative game theory, computing the joint interaction effect for any non-empty subset \(S \subseteq E\) relative to diagnosis \(d\):

\[I_d(S) = \sum_{T \subseteq S} (-1)^{|S| - |T|} v_d(T)\]

When \(|S| = 1\), the index reduces to the singleton marginal effect relative to the empty baseline. For \(|S| \ge 2\), positive values represent synergistic reinforcement towards diagnosis \(d\), whereas negative values indicate mutual cancellation, redundancy, or shifts favoring a competing differential. Auditing focuses primarily on low-order combinations (\(|S| \le 3\)), which effectively balances the combinatorial evaluation cost while preserving human-inspectable cognitive granularity for clinical review.

3. Diagnosis-Relative Evidence Roles & Mechanism Attribution: Decoupling Discovery from Failure Assignment

Standard feature attribution pipelines routinely misinterpret large negative interactions or non-target feature influences as model errors, ignoring that clinical differential diagnosis fundamentally requires competitor evidence to suppress target disease margins. The framework resolves this by establishing five diagnosis-relative role families: supporting evidence, competitor-supporting evidence, excluding/negated evidence, background/risk factors, and likely irrelevant or clinically local evidence.

Mechanism attribution applies a deterministic priority order: it first evaluates candidate-misuse triggers, specifically checking whether explicitly negated/absent findings positively increase target margins, or whether anatomically local evidence exerts unwarranted influence without clinical relevance. Interactions bypassing misuse criteria fall through to legitimate clinical mechanisms: competitor-driven margin reductions are labeled differential conflict/cancellation, saturating target interactions are marked target redundancy, and valid history influences are classified as background/history effects.

4. Multi-Dimensional Stability Filtering & Targeted Counterfactuals: Eliminating Artifacts to Isolate Confirmed Failures

Because option-scored language models are sensitive to prompt phrasing and candidate order, raw suspicious interactions are heavily contaminated with procedural artifacts. The framework applies a three-stage stability funnel: prompt wording variants (clinical intake summary vs. questionnaire dialogue), option letter permutations (shuffling candidate orders across multiple seeds to verify margin correlation), and perturbation neutralization (replacing hidden units with an explicit "Not specified" string rather than silently removing them, preventing implicit absence inferences).

Surviving high-confidence candidates undergo targeted counterfactual validation: specific suspicious evidence clusters are excised or rewritten within the full case context to observe subsequent margin deltas and candidate rank shifts. This isolates global model reliance from benign local noise, proving that while rash findings in inguinal hernia cases represent local incoherence that does not flip top predictions, disease-name family histories in myasthenia gravis induce severe, shortcut-like confidence over-amplification.

Key Experimental Results

Main Results

The audit was evaluated across five open-weight instruction-tuned LLMs (Qwen2.5-7B-Instruct, OpenBioLLM-8B, MedGemma-4B, BioMistral-7B, Llama3-Med42-8B) on the structured DDXPlus benchmark (500 validation cases) alongside two narrative case benchmarks, CupCase (200 test cases) and MedCase (200 test cases).

Model DDXPlus Acc (%) CupCase Acc (%) MedCase Acc (%) Avg. Acc (%)
Qwen2.5-7B-Instruct 65.8 87.0 40.5 64.4
OpenBioLLM-8B 74.0 87.5 55.0 72.2
MedGemma-4B 64.0 70.5 43.5 59.3
BioMistral-7B 54.6 80.0 44.5 59.7
Llama3-Med42-8B 73.4 77.0 49.0 66.5

Ablation & Mechanism Distribution

Fine-grained mechanism shares and interaction strengths on DDXPlus were evaluated alongside blinded clinical-coherence adjudication by five senior PhD-trained medical reviewers across an enriched 130-item high-strength sample (Fleiss' \(\kappa = 0.82\), with 77% initial full-reviewer agreement).

Mechanism Bucket Count Share (%) Strength Share (%) Mean Absolute Effect Adjudicated: Valid Adjudicated: Questionable Adjudicated: Invalid
Conflict/cancellation 40.0 42.7 2.51 36 4 0
Faithful target 37.7 36.6 2.29 29 1 0
Background/history 4.1 7.9 4.55 18 2 0
Weak/redundant 7.1 5.3 1.76 — — —
Target redundancy 3.0 4.4 3.49 — — —
Negated misuse 2.5 2.2 2.12 9 0 11
Control/unclear 5.6 0.9 — 19 1 0
Total / Overall 100.0 100.0 — 111 8 11

Ablation of stability filtering on the 300-item candidate failure review queue demonstrates progressive artifact pruning:

Filtering Stage Candidate Count \(n\) Human-Adjudicated Precision Questionable Rate False Positive Rate
Raw automatic 300 0.55 0.25 0.20
+ Prompt stable 220 0.65 0.20 0.15
+ Option order stable 180 0.75 0.15 0.10
+ Neutralize stable 120 0.80 0.12 0.08
Random control 100 0.10 0.15 0.75

Key Findings

  • Diagnostic accuracy does not mirror evidence-use fidelity: Model accuracy rankings on DDXPlus diverge noticeably from their ranking by faithful support strength or differential conflict strength. For instance, OpenBioLLM achieves the highest overall accuracy (74.0%) yet continues to exhibit non-trivial negated evidence misuse, demonstrating that high benchmark scores can mask clinical reasoning deficiencies.
  • Evidence interactions are ubiquitous and overwhelmingly valid: Across all datasets, differential conflict/cancellation and faithful target support account for 86%–94% of total interaction strength. This highlights that negative mathematical interactions primarily represent legitimate differential exclusion (e.g., pneumonia symptoms properly suppressing asthma margins) rather than model flaws.
  • True failure mechanisms concentrate in evidence polarity and clinical specificity: In the blinded clinical review, all 11 adjudicated invalid cases were strictly confined to the negated/absent evidence bucket (55.0% invalid rate within that bucket), including absent travel history spuriously boosting allergic sinusitis or Guillain-Barré syndrome, and unrelated rash findings distorting inguinal hernia evaluations.
  • Multi-axis stability filtering increases candidate failure precision from 55% to 80%: By eliminating surface prompt artifacts and option-position biases, the filtering funnel reduces false positives from 20% down to 8%, isolating genuinely pathological evidence mechanisms.

Highlights & Insights

  • Decoupling interaction discovery from error assignment: Rather than treating any non-linear feature interaction as an algorithmic anomaly, the paper models clinical differential diagnosis explicitly, allowing legitimate clinical conflict to be recognized as appropriate behavior.
  • The necessity of neutralized counterfactual perturbations: The insight that deleting negative evidence from a text prompt does not equal "absence of the finding" is critical. Neutralizing via "Not specified" provides a much cleaner causal counterfactual for language models.
  • Transferable audit paradigm for competitive multi-choice reasoning: The concept of margin-based interaction profiling translates naturally beyond clinical medicine to any high-stakes domain involving mutually exclusive hypotheses and diagnosis-relative evidence, such as legal judgment and complex hardware fault diagnosis.

Limitations & Future Work

  • Author-acknowledged limitations: The audit is inherently behavioral, inspecting output-level option margins rather than internal hidden states, attention pathways, or mechanistic circuits; exhaustive combinatorial evaluation becomes intractable beyond 6–8 evidence units, restricting audits to low-order combinations.
  • Observer-noted limitations: Structured role assignment relies heavily on rich ontological symptom annotations (such as those in DDXPlus). On unstructured clinical text narratives (like MedCase), extracting reliable evidence units and roles remains dependent on heuristic parsing.
  • Future directions: Scaling the intervention budget using greedy combinatorial approximations or active importance sampling; integrating behavioral audit margin penalties directly into preference alignment objectives (RLHF/DPO) to suppress shortcut learning.
  • vs. Feature Attribution & Saliency Methods (Integrated Gradients / SHAP): Conventional attribution assigns flat, global importance scores to individual tokens without modeling competitor dynamics. This work explicitly models non-additive interactions within a multi-hypothesis differential space.
  • vs. Chain-of-Thought Faithfulness Audits: Evaluating generated rationales is confounded by plausible-sounding hallucinations and unfaithful post-hoc justifications. This framework measures hard, observable decision preferences under controlled inputs.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering behavioral evidence auditing framework tailored to medical differential diagnosis and relative evidence roles.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 5 LLMs, 3 benchmarks, 130-item blinded expert clinician review, and extensive counterfactual ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally rigorous conceptualization, seamless integration of clinical principles and game-theoretic mathematics, and clear empirical exposition.
  • Value: ⭐⭐⭐⭐⭐ Sets a new standard for evaluating safety, clinical validity, and evidence fidelity in medical artificial intelligence.