Skip to content

Dementia-R1: Reinforced Pretraining and Reasoning from Unstructured Clinical Notes for Real-World Dementia Prognosis

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/hangeol/dementiar1
Area: Medical LLM
Keywords: Clinical Prognosis, Unstructured Notes, Reinforcement Learning, Verifiable Rewards, Longitudinal Reasoning

TL;DR

Dementia-R1 introduces a two-stage reinforcement learning framework that leverages Cold-Start RL with tolerance-aware verifiable clinical rewards (MMSE, GDS, CDR) to master intermediate disease trajectory forecasting before task fine-tuning, achieving 84.02% AUROC on real-world unstructured notes and outperforming models up to 10× larger.

Background & Motivation

The widespread adoption of electronic health records (EHRs) has accumulated vast repositories of longitudinal patient encounters spanning months or years. However, approximately 80% of clinical data remains unstructured free text, such as physician progress notes and outpatient SOAP narratives, where progressive behavioral shifts, functional decline, and non-monotonic symptom trajectories are documented implicitly. Conventional longitudinal deep learning models (e.g., RNNs or time-to-event transformers like MOTOR) are designed primarily for structured tabular records, largely failing to ingest unstructured narratives. Conversely, modern medical large language models (such as HuatuoGPT-o1 and MedGemma) predominantly evaluate static, snapshot-style medical question answering benchmarks, leaving long-term temporal and causal disease progression modeling an open challenge.

Predicting long-term dementia progression from longitudinal narratives is particularly arduous because real-world clinical trajectories are inherently fluctuating and non-monotonic: cognitive scores and daily living assessments often experience transient dips or spikes due to acute concurrent illnesses, emotional distress, or measurement noise. Consequently, the ultimate clinical outcome—whether a patient converts to full dementia over months or years—is delayed and sparse. Standard supervised fine-tuning (SFT) requires cost-prohibitive, temporally consistent expert-annotated Chain-of-Thought rationales that are difficult to standardize across multi-year trajectories. On the other hand, directly applying outcome-level reinforcement learning with binary terminal rewards suffers severely from reward sparsity and exploration collapse in the vast latent reasoning space.

This paper tackles this bottleneck by recognizing that while the terminal prognosis label is sparse, a patient's historical medical records are punctuated by objective, standardized neurocognitive test scores (e.g., MMSE, GDS, CDR) recorded during routine clinical visits. The core idea is to bypass rationale-based supervised cold starts by pre-training the model via Cold-Start RL on verifiable intermediate clinical indices using Group Relative Policy Optimization (GRPO), incentivizing the autonomous emergence of temporal reasoning primitives before fine-tuning on the terminal prognosis under sparse rewards.

Method

Overall Architecture

Dementia-R1 models longitudinal prognosis as mapping a patient's chronological, unstructured clinical note history \(H_{<T} = \{x_1, \dots, x_t\}\) to a future clinical status at a target anchor point \(T_{\text{anchor}}\). The architecture comprises two synergistic training stages:

  1. Longitudinal History Construction & Verifiable Data Pipeline: Historical multi-visit SOAP clinical notes are concatenated chronologically into a contextual document. An on-premise auxiliary extractor identifies objective cognitive assessment indices at subsequent visits to serve as verifiable targets.
  2. Stage 1 (Cold-Start Pre-training): Conditioned on preceding text history, the model generates an explicit temporal reasoning path within <think> tags and forecasts intermediate clinical index scores within <answer> tags, trained via GRPO using a tolerance-aware verifiable reward.
  3. Stage 2 (Task Fine-tuning): Building upon the temporal trajectory reasoning primitives acquired in Stage 1, the model is fine-tuned to predict the binary dementia prognosis label (Dementia vs. Non-Dementia) using GRPO with terminal outcome rewards.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Patient Longitudinal Notes<br/>H&lt;t = [x1, x2, ..., xt]"] --> B["Stage 1: Cold-Start Pre-training<br/>Forecast Verifiable Clinical Indices"]
    B --> C["Stage 2: Task Fine-tuning<br/>Predict Final Dementia Prognosis"]
    C --> D["Downstream Clinical Prognosis<br/>Binary Decision & Grounded Rationale"]

Key Designs

1. Stage 1: Cold-Start Pre-training: Self-Supervised Clinical Index Forecasting Standard RL with verifiable rewards (RLVR) in reasoning models requires ground-truth verification at test time, which longitudinal prognosis inherently lacks. Dementia-R1 circumvents this by pre-training the policy to forecast intermediate clinical scores \(s_t\) already present in the subsequent notes of the patient's record. This forces the model to synthesize symptom evolution, medication switches, and functional descriptions into structured numeric trajectories.

To accommodate different granularities and measurement noise across medical scales, a tolerance-aware step reward \(R_{\text{cold}}\) is employed. For predicted index \(\hat{s}_t\), ground truth \(s_t\), and error margin \(\delta\): $\(R_{\text{cold}} = \mathbb{I}(|\hat{s}_t - s_t| \le \delta)\)$ For the 30-point Mini-Mental State Examination (MMSE), the tolerance is set to \(\delta = 2\), matching the established clinical Reliable Change Index for MMSE. For coarser staging scales such as the Global Deterioration Scale (GDS, range 1–7) and Clinical Dementia Rating (CDR, range 0–3), strict exact matching is enforced (\(\delta = 0\)).

2. Stage 2: Task Fine-tuning: Transferring Trajectory Reasoning to Sparse Binary Prognosis After Stage 1 equips the model with robust longitudinal reasoning primitives, Stage 2 transitions to optimizing for the ultimate clinical outcome \(y \in \{0, 1\}\) (Dementia vs. Non-Dementia, defined as GDS \(\ge 4\) in the Asan cohort; or Parkinson's Disease Dementia [PDD] in the HPH cohort). Because the terminal label is high-level and binary, Stage 2 employs an outcome verification reward \(R_{\text{task}}\): $\(R_{\text{task}} = \begin{cases} 1, & \text{if predicted label matches gold diagnosis} \\ 0, & \text{otherwise} \end{cases}\)$ Because the policy has already internalized the dynamics of cognitive decline and trajectory interpretation during Stage 1, it successfully avoids shortcut learning and converges smoothly despite the extreme sparsity of the delayed reward.

3. Group Relative Policy Optimization (GRPO) with Inherent Gradient Auto-Filtering Both stages optimize the policy using GRPO, which estimates baseline advantages across a group of \(G\) sampled completions \(\{o_1, \dots, o_G\}\) without requiring a separate value/critic network: $\(A_i = \frac{R(o_i) - \text{mean}(\{R(o_j)\}_{j=1}^G)}{\text{std}(\{R(o_j)\}_{j=1}^G)}\)$ A crucial theoretical and practical discovery of this design is its inherent resilience to extraction noise in Stage 1. When an auxiliary extractor produces an incorrect or clinically implausible target label, the model's language prior steers all \(G\) completions away from the erroneous score. Consequently, all completions receive \(R(o_i) = 0\), leading directly to \(\text{mean} = 0\), \(\text{std} = 0\), and advantage \(A_i = 0\). The gradient contribution from noisy prompts completely vanishes, effectively auto-filtering corrupted data without degrading policy parameters. Empirical stress tests confirm that injecting up to 70% above-tolerance label noise into Stage 1 reduces downstream dementia AUROC by less than 0.5 percentage points.

Loss & Training

The overall policy training objective optimizes the standard clipped surrogate objective with KL-divergence regularization: $\(\mathcal{L}(\theta) = \mathbb{E}_{q_t, \{o_i\}} \left[ \frac{1}{G} \sum_{i=1}^G \min \left( \frac{\pi_\theta(o_i|q_t)}{\pi_{\theta_{\text{old}}}(o_i|q_t)} A_i, \; \text{clip}\left(\frac{\pi_\theta(o_i|q_t)}{\pi_{\theta_{\text{old}}}(o_i|q_t)}, 1-\epsilon, 1+\epsilon\right) A_i \right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right]\)$ Experiments were conducted using PyTorch and the Open-R1 framework across 4 NVIDIA H100 (80GB) GPUs using DeepSpeed ZeRO-3 and vLLM in colocate mode. Training runs with per-device batch size 1, gradient accumulation 2, group size \(G = 8\) (effective batch size 8), a 2,000-token completion limit, and Bfloat16 precision over 5,000 steps.

Key Experimental Results

Main Results

On the real-world unstructured clinical cohort from Asan Medical Center (AMC, ~3,000 neurocognitive disorder patients), Dementia-R1 (7B backbone) outperforms specialized medical models and general open-weights models up to 10× larger across all evaluation metrics.

Method Size Accuracy (%) Precision (%) Recall (%) F1 score (%) AUROC (%) AUPRC (%)
HuatuoGPT-o1 8B 67.53 71.33 60.63 65.53 73.89 74.81
MedGemma 27B 67.27 62.42 89.79 73.64 74.14 71.06
Qwen2.5-7B-Inst 7B 72.81 74.37 71.13 72.71 78.59 79.24
Qwen2.5-32B-Inst 32B 62.03 57.67 95.63 71.95 79.93 79.33
Qwen2.5-72B-Inst 72B 74.46 70.60 85.42 77.30 83.41 83.44
Llama-3.3-70B-Inst 70B 67.27 62.14 91.39 73.98 80.70 81.11
SFT w/o Stage 1 7B 64.68 60.97 85.13 71.05 71.65 69.82
GRPO w/o Stage 1 7B 72.15 70.19 78.74 74.21 79.88 80.46
SFT \(\to\) SFT 7B 74.97 73.65 79.16 76.30 77.15 72.41
SFT \(\to\) GRPO 7B 73.60 71.00 81.58 75.92 77.75 72.69
Dementia-R1 (Ours) 7B 77.45 79.26 75.46 77.31 84.02 85.73

On the independent Haeundae Paik Hospital (HPH) cross-disease Parkinson's disease cohort, Dementia-R1 achieves the top AUROC (78.37%) and AUPRC (51.03%) for PDD prediction, surpassing Qwen2.5-72B-Instruct (72.37% AUROC) and the strongest 7B baseline SFT \(\to\) GRPO (74.75% AUROC). On the structured ADNI benchmark, Dementia-R1 attains 83.17% AUROC, the highest among all LLM baselines.

Ablation Study

Ablations on the AMC cohort highlight the indispensability of Stage 1 pre-training and analyze input modality contributions:

Configuration (AMC Cohort) Time Window AUROC (%) AUPRC (%) Key Insight / Finding
Dementia-R1 (Full Model) Overall 84.02 85.73 Full two-stage GRPO \(\to\) GRPO framework
w/o Stage 1 (Direct GRPO) Overall 79.88 80.46 Eliminating Stage 1 index forecasting drops AUROC by 4.14%
SFT \(\to\) GRPO (Teacher CoT Cold-Start) Overall 77.75 72.69 Distilled CoT rationales memorize rather than generalize
Note Only (Raw Free Text) 18–24m 76.20 83.10 Effective trajectory tracking even without explicit scores
Index Only (Structured Scores Only) 18–24m 71.50 78.20 Omitting clinical narratives sharply degrades long-range prognosis
Note + Index (Combined) 18–24m 78.51 86.15 Optimal synergy between narrative context and objective tests
Inference Masking: Mask MMSE (-M) Overall 79.41 80.65 MMSE carries highest prognostic signal among AMC indices
Inference Masking: Mask All Indices (-ALL) Overall 78.07 78.25 Retains substantial performance, confirming internalized reasoning

Key Findings

  • Robustness Across Fluctuating Trajectories: Evaluating on a specialized subset of 209 AMC patients whose GDS trajectories crossed the dementia diagnostic threshold (\(3 \to 4 \to 3\)), Dementia-R1 led all nine baselines with 77.72% AUROC, widening its advantage over the single-stage ablation (70.37%) to 7.35 points. Qualitative inspection revealed that Dementia-R1 grounded its prediction on caregiver-reported improvements and medication changes, while 32B/72B baselines over-called based on single temporary score spikes.
  • Biomarker Informativeness: Index masking reveals cohort-dependent prognostic value: MMSE-family signals contribute the largest prognostic margin in hospital EHRs (AMC and HPH), whereas CDR-SB dominates predictive power in the ADNI cohort.
  • Blinded Neurologist Evaluation: In a pairwise blind study conducted by two board-certified neurologists evaluating 100 judgment pairs across AMC and HPH, Dementia-R1 scored a 68% win rate in Overall Clinical Utility, a 72% win rate in Evidence Grounding, and a 95% Win/Tie rate in Temporal Reasoning Accuracy against Qwen2.5-32B-Instruct. Notably, the baseline frequently suffered from alarming hallucinations—such as confusing Ebixa (memantine) with donepezil and misinterpreting GDS dementia staging as a depression inventory.

Highlights & Insights

  • Turning Clinical Routine Measurements into Self-Supervised RL Anchors: Bypasses the bottleneck of human CoT annotation by exploiting existing clinical scores as dense, verifiable intermediate rewards.
  • Auto-Filtering of Corrupted Data under GRPO: Establishes that policy exploration under group advantage normalization naturally drops invalid or noisy labels with zero gradient updates, rendering the framework resilient to up to 70% label noise.
  • Superior Long-Horizon Reasoning: Performance gains over larger models widen at the longest prediction windows (18–24m for AMC, 36m+ for ADNI), demonstrating that Stage 1 grounds genuine temporal dynamics rather than superficial keyword matching.

Limitations & Future Work

  • Cohort & Demographic Scope: Unstructured datasets originate from two South Korean tertiary hospitals; validation on international cohorts (e.g., US/UK EHR systems) is necessary to confirm cross-lingual and cross-demographic robustness.
  • Binary Outcome Simplification: The clinical task is currently formulated as binary dementia versus non-dementia, collapsing intermediate stages such as Mild Cognitive Impairment (MCI) and omitting dementia etiology subtyping (AD, VaD, DLB).
  • Reliance on Clinical Staging rather than Biomarker Confirmation: Diagnostic outcomes are established based on physician documentation, ICD codes, and medication records rather than neuropathologically confirmed post-mortem or amyloid-PET/CSF gold standards.
  • vs. HuatuoGPT-o1 / MedGemma: While previous medical models excel in static board-style question answering, they fail on fluctuating multi-visit disease trajectories; Dementia-R1 demonstrates how verifiable intermediate rewards unlock genuine longitudinal clinical reasoning in compact 7B models.
  • vs. CARE-AD / MOTOR: Unlike traditional machine learning frameworks restricted to tabular features or LLMs optimizing only final outcomes, Dementia-R1 seamlessly integrates unstructured clinical narratives and quantifiable indices into an end-to-end RL curriculum.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant reformulation of clinical index forecasting as a Cold-Start RL milestone.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across three distinct cohorts, extensive ablations, noise stress tests, and clinician blinded audits.
  • Writing Quality: ⭐⭐⭐⭐⭐ Clear, rigorous, and cohesive presentation linking clinical necessity with reinforcement learning mechanics.
  • Value: ⭐⭐⭐⭐⭐ Establishes a practical, highly generalizable blueprint for longitudinal reasoning from unstructured clinical narratives across chronic progressive illnesses.