Skip to content

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Healthcare

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/SpidermanGUAN/MedPriv-Bench
Area: Medical NLP
Keywords: Medical LLMs, Contextual Privacy Leakage, Privacy-Utility Trade-off, Retrieval-Augmented Generation (RAG), Natural Language Inference (NLI)

TL;DR

MedPriv-Bench is the first benchmark designed to jointly evaluate privacy preservation and clinical utility in medical open-ended question answering, establishing a multi-agent synthetic pipeline and a fine-tuned RoBERTa-NLI judge to reveal pervasive contextual leakage risks across nine major LLMs and demonstrating the superiority of supervised fine-tuning and brevity constraints.

Background & Motivation

The deployment of large language models (LLMs) in high-stakes domains such as healthcare has accelerated rapidly, yet concerns regarding hallucinations—generating medically incorrect or nonsensical claims—pose grave threats that can lead to delayed treatments or compromised patient safety. To anchor generative outputs in verifiable clinical evidence, retrieval-augmented generation (RAG) is increasingly embraced, allowing models to query proprietary hospital databases, electronic health records (EHRs), and clinical progress notes. However, connecting generative models to sensitive longitudinal patient records introduces severe and under-explored privacy vulnerabilities. While standard de-identification protocols like Named Entity Recognition (NER) successfully scrub explicit Protected Health Information (PHI) such as patient names, social security numbers, and hospital record identifiers prior to vector indexing, they remain fundamentally blind to semantic interdependencies woven into clinical narratives.

This failure mode precipitates "contextual leakage," a subtle threat grounded in the theory of contextual integrity. Even without mentioning explicit personal identifiers, an LLM synthesizing an answer from a patient's rich background may unnecessarily incorporate a unique constellation of clinical attributes—such as a rare diagnosis combined with stigmatized behaviors, psychiatric history, or housing instability. In a localized clinical community, this distinctive combination functions as a semantic fingerprint that enables malicious adversaries or unauthorized readers to re-identify the patient through linkage attacks. Crucially, such gratuitous disclosure directly violates HIPAA’s "Minimum Necessary Standard" (45 CFR § 164.502(b)), which mandates that disclosures of health information be confined strictly to what is necessary to accomplish the intended clinical purpose, alongside GDPR obligations.

Existing LLM privacy evaluations (such as LLM-PBE and PrivaCI-Bench) predominantly focus on general-domain personal data or agentic social scenarios, while medical benchmarks (like PubMedQA, MedQA, and MedSafetyBench) either rely on multiple-choice questions or measure ethical safety refusals rather than contextual data disclosure. Crucially, healthcare QA operates under an intrinsic tension between privacy and clinical utility: over-aggressive suppression produces evasive non-answers that degrade therapeutic efficacy, whereas maximizing exhaustive clinical helpfulness inadvertently leaks confidential patient context. To benchmark this Pareto frontier rigorously without violating patient privacy laws, researchers require a controlled, standardized evaluation framework. Core idea: develop MedPriv-Bench, a multi-agent human-in-the-loop benchmark constructed from peer-reviewed clinical summaries that forces models to confront realistic privacy pressure through white-box attack queries, evaluated via a fine-tuned RoBERTa-NLI model that achieves high-sensitivity automated detection of contextual privacy breaches alongside a 0–5 clinical utility rubric.

Method

Overall Architecture

The construction of MedPriv-Bench is structured into four sequential phases: context-aware PHI injection (Phase 1), white-box attack question generation (Phase 2), privacy-preserving reference answer synthesis (Phase 3), and dual-LLM automated filtering coupled with expert-in-the-loop validation (Phase 4). Building upon the de-identified PMC-Patients case summaries as an ethical knowledge base, a specialized Injection Agent embeds coherent sensitive facts categorized under a four-part taxonomy derived from HIPAA standards. A Question Agent acts as a white-box adversary, formulating clinical queries that implicitly benefit from the injected secrets to induce privacy pressure. An Answer Agent generates a gold-standard response accompanied by a 5-point clinical grading rubric. During evaluation, model responses are scored for clinical utility using GPT-5-mini against the rubric, while an automated judge powered by a domain-finetuned RoBERTa-NLI classifier evaluates fact-level and instance-level privacy leakage via sliding-window entailment testing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["PMC-Patients Clinical Narratives<br/>De-identified Case Summaries"] --> B["Context-Aware PHI Injection & White-Box Attack Query<br/>Four-Category PHI + Implicitly Dependent Query"]
    B --> C["Dual-LLM Arbitration & Expert-in-the-Loop Validation<br/>Correctness Verification + Minimum Utility Auditing"]
    C --> D["Fine-Tuned RoBERTa-NLI Automated Privacy Judging<br/>Sliding-Window Entailment + 0-5 Utility Rubric"]
    D --> E["Cross-Defense Benchmarking & Generation Word Limits<br/>Prompt / SFT / LDP Evaluation + 100-Word Constraint"]
    E --> F["MedPriv-Bench Benchmark Suite<br/>Pareto Frontier & Deployment Guidelines"]

Key Designs

1. Context-Aware PHI Injection & White-Box Attack Query: Synthetic De-identification Failure -> Multi-Category PHI Embedding with Utility-Coupled Adversarial Probing -> Generating Genuine Privacy Pressure in Medical QA

Standard synthetic leakage benchmarks rely on isolated keyword insertion, which fails to simulate natural medical narratives. MedPriv-Bench addresses this by defining a four-category PHI taxonomy \(T\) grounded in HIPAA Expert Determination (45 CFR § 164.514(b)(1)), 42 CFR Part 2 (substance use confidentiality), and ICD-10 F/Z codes: Stigmatized Behaviors (SB), Mental Health & Cognitive Status (MH&CS), Socioeconomic Determinants of Health (SDOH), and Quasi-Identifiers / VIP Status (QIV). The Injection Agent \(A_{inj}\) samples appropriate categories and embeds plausible sensitive facts \(E_\phi\) (such as untreated major depression with episodic suicidal ideation, or living in a car lacking biologics refrigeration) seamlessly into clinical summary chunks \(C_i\) to form sensitive context \(C_i^*\). The Question Agent \(A_{quest}\) then acts as a white-box adversary possessing full knowledge of \(C_i^*\) and \(E_\phi\). Rather than asking overt intrusive questions (e.g., "Does the patient use illicit drugs?"), it synthesizes natural clinical questions \(q\) regarding discharge planning, pain regimens, or infection control that implicitly require reasoning over the latent PHI to yield optimal care. The agent also records a reasoning path \(r_q\), establishing an authentic tension where answering thoroughly tempts the LLM to disclose protected details.

2. Dual-LLM Arbitration & Expert-in-the-Loop Validation: Synthetic Hallucinations and Non-Clinical Phrasing -> Consensus Filtering by Distinct Model Architectures with Clinical Nurse Auditing -> Guaranteeing Zero-Hallucination and Non-Evasive Utility

Because automated multi-agent synthesis can produce factual drift or unrealistic phrasing, the framework institutes a multi-tiered validation filter. First, candidate generation tuples pass through a dual-LLM audit comprising GPT-5.2 and Qwen 3.6 Plus. A tuple receives a "Pass" classification only when both heterogeneous models agree, substantially mitigating single-model inductive biases. The audit enforces two strict criteria: Correctness (Zero-Hallucination), which bans introducing ungrounded clinical lab values, dosages, or contradictory statements while permitting generalized clinical summarization; and Completeness (Minimum Viable Utility), which rejects evasive non-answers (e.g., "Consult your physician") that dodge clinical guidance. Subsequently, registered nurses and nursing graduate researchers perform a 20% random manual audit of the filtered dataset. The human annotators undergo joint calibration trials on sample cases before labeling, ensuring that the benchmark satisfies clinical realism and legal fidelity.

3. Fine-Tuned RoBERTa-NLI Automated Privacy Judging: High Latency of LLM Judges and Generic Entailment Blind Spots -> Domain-Adapted NLI with Overlapping Sliding Windows -> Delivering 90.7% Sensitivity and 0.056s Sub-Second Inference

Employing general-purpose LLMs as privacy judges introduces excessive computational cost and latency that precludes large-scale experimentation, whereas off-the-shelf NLI models lack sensitivity to subtle medical inferences. Crucially, in patient privacy informatics, failing to detect an actual breach (false negative) carries far greater ethical and legal risk than over-flagging an ambiguous generalization (false positive). To prioritize recall, the authors fine-tuned roberta-large-mnli on 1,392 curated medical QA pairs (1,113 training, 279 testing) balanced with leakage counterexamples. To overcome the 512-token architectural sequence limit, an overlapping sliding-window chunking strategy with a stride of 128 tokens partitions the model response \(\hat{y}_i = \{c_{i,1}, \dots, c_{i,k}\}\). Each chunk is evaluated independently against the sensitive fact hypothesis \(H_{i,j}\); if any chunk entails the hypothesis, the fact is classified as leaked. This fine-tuned NLI judge elevates instance-level sensitivity from 58.7% (pre-trained) to 90.7% and achieves an instance F1-score of 75.3% against human clinical raters, while operating at an ultra-fast average inference speed of 0.056 seconds per sample.

4. Cross-Defense Benchmarking & Generation Word Limits: Trade-off Discrepancies Across Defenses -> Systematic Comparison of Prompt, SFT, and LDP with Response Length Suppression -> Achieving Minimum Necessary Compliance Without Parameter Retraining

To explore viable mitigations, the authors formalize and benchmark four defense regimes: unprotected baseline (w/o), prompt-based privacy guardrails (w/), supervised instruction fine-tuning via LoRA (SFT on 885 instances), and inference-time Local Differential Privacy (LDP). The LDP defense applies Generalized Randomized Response (GRR) over the model’s top-\(K\) candidate tokens at each decoding step \(t\) with privacy budget \(\epsilon\): $\(\Pr(\tilde{y}_t = b_t) = \frac{e^\epsilon}{e^\epsilon + K - 1}, \quad \Pr(\tilde{y}_t = y) = \frac{1}{e^\epsilon + K - 1} \quad \forall y \in C_t \setminus \{b_t\}\)$ Beyond algorithmic defenses, the study isolates the mechanism of leakage via sensitivity analysis, demonstrating that leakage is predominantly driven by generation verbosity rather than retrieval volume. Enforcing a strict 100-word generation limit at inference time drastically suppresses instance leakage by 39.8% while maintaining a high clinical utility of 4.53, providing a lightweight, deployable implementation of HIPAA's Minimum Necessary Standard without degrading factual efficacy.

Benchmark & Evaluation Protocol

MedPriv-Bench comprises \(N=2,938\) verified instances, exhibiting balanced gender representation (52.76% male, 47.24% female) and comprehensive age stratifications: Pediatric 0–18 (15.04%), Adult 19–40 (25.63%), Middle Age 41–65 (37.07%), and Senior 65+ (22.26%). A held-out test split of \(n=738\) instances is designated for empirical evaluation. - Response Utility (0–5): Evaluated via GPT-5-mini using an instance-specific rubric generated alongside the gold-standard answer, assigning integer points that total exactly 5. - Fact-Level Leakage Rate (\(LR_{fact}\)): The proportion of total sensitive facts entailed by model responses relative to all injected secrets across the dataset: $\(LR_{fact} = \frac{\sum_{i=1}^N \sum_{j=1}^{m_i} I(\hat{y}_i, e_{i,j})}{\sum_{i=1}^N |E_\phi^{(i)}|}\)$ - Instance-Level Leakage Rate (\(LR_{inst}\)): An uncompromising any-leak safety metric indicating whether a response exposes at least one injected PHI item: $\(LR_{inst} = \frac{1}{N} \sum_{i=1}^N \max_j I(\hat{y}_i, e_{i,j})\)$

Evaluated models encompass closed-source proprietary systems (GPT-5-nano, Claude Haiku 4.5, Gemini 3 Flash), open-weight general architectures (Llama3-70B, Qwen 3.6, DeepSeek-V3.2), and open-source medical specialists (Med42-v2-8B, Med42-v2-70B).

Key Experimental Results

Main Results

The primary benchmark evaluation on the test set (\(n=738\)) across nine models under varying defenses is detailed below.

Model Category / Evaluated Model Defense Setting Response Utility (0–5) ↑ \(LR_{inst}\) (%) ↓ \(LR_{fact}\) (%) ↓ SB Leak (%) ↓ MH&CS Leak (%) ↓ SDOH Leak (%) ↓ QIV Leak (%) ↓
Closed-Source Models
GPT-5-nano w/o Defense 4.89 70.2 57.1 32.2 58.0 65.8 38.1
GPT-5-nano Prompt Defense 4.88 54.6 42.7 16.9 43.5 52.7 19.6
Claude Haiku 4.5 w/o Defense 4.71 89.0 80.0 75.4 88.3 80.5 36.1
Claude Haiku 4.5 Prompt Defense 4.60 61.4 48.6 33.1 61.1 46.9 6.2
Gemini 3 Flash w/o Defense 4.14 98.4 97.0 94.1 99.4 98.8 77.3
Gemini 3 Flash Prompt Defense 4.32 93.2 86.5 70.3 96.7 92.6 18.6
Open-Source General Models
Llama3-70B w/o Defense 4.19 86.3 72.9 59.3 75.8 77.0 52.6
Llama3-70B Prompt Defense 4.06 74.1 58.9 41.5 62.4 64.3 33.0
Qwen 3.6 w/o Defense 4.54 85.5 74.8 55.9 89.5 71.1 34.0
Qwen 3.6 Prompt Defense 4.48 85.6 74.0 58.5 89.7 69.5 26.8
DeepSeek-V3.2 w/o Defense 4.72 98.4 94.2 86.4 96.7 97.3 74.2
DeepSeek-V3.2 Prompt Defense 4.63 80.2 68.3 58.5 81.1 67.8 10.3
Medical-Specific Models
Med42-v2-70B w/o Defense 4.54 89.2 77.6 74.6 76.3 83.2 60.8
Med42-v2-70B Prompt Defense 4.60 83.6 71.8 77.1 69.4 78.7 44.3
Med42-v2-8B w/o Defense 3.87 85.5 72.8 53.4 81.3 74.8 38.1
Med42-v2-8B Prompt Defense 3.71 81.3 67.6 47.5 77.4 68.9 30.9
Privacy-Preserving Methods
Med42-v2-8B + LDP 3.03 31.3 20.5 23.7 10.3 30.7 22.7
Med42-v2-8B + SFT 4.25 50.5 38.9 27.1 48.6 36.9 9.3
Llama3-70B + LDP 2.96 30.5 20.8 15.3 15.6 28.1 19.6
Llama3-70B + SFT 4.33 56.1 42.0 52.8 17.5 37.7 29.4

Ablation & Sensitivity Analysis

Ablations covering SFT sample efficiency, seed sensitivity, retrieval volume, and response length constraints are summarized below.

Experimental Dimension Configuration / Subset Mean Utility (0–5) [95% CI] High Utility Rate (≥4) \(LR_{inst}\) (%) ↓ \(LR_{fact}\) (%) ↓ Empirical Insights & Mechanics
SFT Training Size (Med42-8B) 25% Split (221 rows) 4.05 [3.99, 4.12] 74.0% 62.3 51.4 Early compliance gains but higher residual leakage
50% Split (442 rows) 4.07 [4.01, 4.13] 74.7% 67.5 54.1 Plateau in privacy suppression at mid-scale
75% Split (664 rows) 4.10 [4.04, 4.17] 77.2% 67.6 56.1 Continued utility gains; privacy non-monotonic
100% Full (885 rows) 4.25 [4.19, 4.31] 82.0% 50.5 38.9 Optimal Pareto frontier; lowest leakage rate
SFT Seed Sensitivity (Med42-8B) 3 Random Seeds 4.12 ± 0.12 — 57.8 ± 6.5 46.5 ± 6.6 Stable utility but moderate variance in privacy metrics
Context Chunk Size (GPT-5-nano) 1 Retrieved Chunk 4.30 — ~48.0 ~36.0 Reduced context bounds utility with moderate leakage
3 Retrieved Chunks (Full) 4.88 — 54.6 42.7 Rich evidence unlocks 4.88 utility with slight leak increase
Response Word Limit (GPT-5-nano) Max 100 Words 4.53 — 14.8 11.2 Brevity constraint cuts \(LR_{inst}\) by 39.8% while preserving 4.53 utility
Max 300 Words 4.75 — 38.5 29.4 Intermediate expansion brings gradual privacy erosion
Max 500 Words 4.86 — 52.1 40.8 Output verbosity approaches unconstrained leakage levels

Key Findings

  • Clinical Competence Does Not Imply Privacy Robustness: Several high-utility models exhibit catastrophic data leakage in unconstrained settings (e.g., DeepSeek-V3.2 achieves 4.72 utility but leaks 94.2% of sensitive facts; Gemini 3 Flash leaks 97.0%). Notably, the domain-adapted Med42-v2-70B model with prompt defense retains an \(LR_{fact}\) of 71.8%—substantially leakier than many defended general-purpose models. Clinical domain pre-training unintentionally empowers models to identify and articulate latent patient vulnerabilities, demonstrating that privacy safety must be engineered separately from medical accuracy.
  • Prompt Defenses Are Lexically Superficial and Highly Architecture-Dependent: System-prompt defenses reliably scrub explicit quasi-identifiers (e.g., Claude Haiku 4.5 cuts QIV leakage from 36.1% to 6.2%), but fail against complex inferential categories such as Mental Health & Cognitive Status (MH&CS) and SDOH. Furthermore, prompt responsiveness varies drastically across model backbones: GPT-5-nano experiences noticeable leakage drops, whereas Qwen 3.6 exhibits virtually zero behavioral shift under defense prompts.
  • SFT Provides a "Win-Win" Frontier, While LDP Collapses Utility: Supervised instruction fine-tuning on as few as 885 curated pairs fundamentally alters generation dynamics, boosting Med42-v2-8B utility from 3.87 to 4.25 while reducing \(LR_{fact}\) from 72.8% to 38.9%. In contrast, Local Differential Privacy (LDP) successfully suppresses leakage to 20.5% but shatters clinical utility down to 3.03, generating nonsensical tokens that render responses clinically unsafe.
  • Contextual Leakage Is Driven by Over-Generation Rather Than Retrieval: Ablating retrieval chunk volume indicates that expanding retrieved context from 1 to 3 chunks elevates utility by +0.58 while increasing leakage only moderately. Conversely, imposing a 100-word generation constraint suppresses instance leakage from 54.6% to 14.8% while maintaining strong utility (4.53), demonstrating that output verbosity is the primary operational vector for contextual privacy violations.

Highlights & Insights

  • From Entity Redaction to Contextual Integrity: The paper shifts clinical privacy evaluation from superficial NER scrubbing to assessing complex relational inferences, providing the first benchmark operationalization of HIPAA’s Minimum Necessary Standard in generative AI.
  • White-Box Red-Teaming Coupled with High-Recall NLI: Framing privacy auditing as an NLI problem using a domain-tuned RoBERTa judge delivers an exceptionally high sensitivity of 90.7% with sub-second latency (0.056s per query), offering an open, reproducible alternative to costly LLM-as-a-judge pipelines.
  • Length Constraint as a Frictionless Compliance Guardrail: Validating that brevity constraints suppress nearly 40% of contextual leakage provides healthcare system architects with an immediately deployable inference-time guardrail that requires zero retraining.

Limitations & Future Work

  • Single-Turn Evaluation Scope: MedPriv-Bench exclusively measures single-turn question answering, omitting multi-turn clinical consultations where an adversary can iteratively probe, triangulate, and accumulate dispersed clues to exacerbate contextual leakage across dialogue turns.
  • Surrogate Knowledge Base vs. Noisy Hospital EHRs: Utilizing peer-reviewed PMC-Patients summaries guarantees ethical distribution and high data cleanliness, but lacks the messy abbreviations, typographical errors, and fragmented telemetry characteristic of raw hospital EHRs.
  • Sliding-Window NLI Blind Spots on Distributed Cues: The 512-token sequence limit of RoBERTa requires chunking that struggles to detect subtle inferences distributed across distant sentences (such as mentioning parole support early on and correctional health later), contributing to false negatives in lengthy outputs like GPT-5-nano.
  • Monolingual Focus: The benchmark evaluates English clinical narratives only, leaving privacy dynamics across multilingual and cross-border regulatory environments unexplored.
  • Comparison with MedSafetyBench: MedSafetyBench focuses on refusing harmful medical actions (e.g., self-harm or lethal dosing advice), whereas MedPriv-Bench tackles legitimate clinical inquiries where the danger lies in gratuitously revealing confidential patient history.
  • Comparison with Contextual Privacy Benchmarks (CONFAIDE, PrivaCI-Bench): Prior benchmarks investigate contextual integrity in generic office, legal, or social interactions without evaluating domain-specific clinical utility trade-offs. MedPriv-Bench pioneers joint utility-privacy optimization specifically for medicine.
  • System Architecture Implications: Future clinical RAG systems should abandon the naive assumption that input de-identification is sufficient, moving toward multi-layered defenses combining intent-specific retrieval filtering, SFT-adapted generation models, and strict output length budgets.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the first formal benchmark evaluating the privacy-utility trade-off in medical open-ended QA, introducing white-box adversarial injection into clinical narratives.
  • Experimental Rigor: ⭐⭐⭐⭐⭐ Evaluates 9 leading LLMs across 3 defense paradigms with extensive ablations, human clinical validation, and rigorous inter-annotator statistics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity, structured narrative progression, and thorough legal and clinical contextualization.
  • Impact & Practical Utility: ⭐⭐⭐⭐⭐ Fully open-sourced dataset, code, and fine-tuned NLI weights addressing an urgent, high-stakes bottleneck for real-world medical AI deployments.