CARE: Causally-Aligned Reasoning Exploration for Medical Large Language Models

Conference: ECCV2026
Paper: Official page · PDF
Area: Medical Multimodal Reasoning
Keywords: Causal sufficiency, proximal learnability, group-relative policy optimization, self-verification, experience replay

TL;DR

Instead of rewarding every medical reasoning trajectory that reaches the correct answer, CARE checks correctness, learnable difficulty, and whether the rationale independently reproduces the answer, then combines online exploration with difficult-experience replay to raise Hulu-Med-7B's average score on PMC-VQA, MedQA, and MMMU-Med from 63.9 to 65.4 and achieve 86.2% expert-rated valid reasoning on a MedQA subset answered correctly by both compared methods.

Background & Motivation

Medical models need to turn images or case descriptions into evidence-based decisions, but expert-written reasoning traces are expensive and difficult to collect at scale. Reinforcement learning on existing question-answer labels offers a less costly route: reward the generated explanation whenever its final answer is correct. An autoregressive model, however, can either derive that answer from its explanation or bypass the explanation and guess from keywords in the original question or knowledge encoded during pretraining. An outcome reward does not distinguish these routes, so an invalid explanation can be reinforced alongside a correct answer.

Reward granularity is only part of the problem. A medical training mixture can contain recognition questions the model has already mastered and diagnostic questions about which it remains highly uncertain. Repeatedly training on the former may add little, while the latter may provide unstable learning signals even when an occasional rollout happens to be correct. The authors therefore shift attention from simply improving answer accuracy to deciding which self-generated experiences deserve to enter the next training update, considering both explanation quality and the model's present competence.

CARE avoids training an additional expert verifier. It asks the model to answer again from its own rationale without seeing the original question, while recent trajectory negative log-likelihood quantiles remove overly easy and overly difficult experiences. Core idea: only trajectories that are correct, lie within the current learnability window, and independently reproduce their answer from the rationale receive a positive admission reward and enter experience replay.

Method

Overall Architecture

Training inputs are medical questions with reference answers; they may contain images and text or text alone, and do not require expert-authored rationales. The policy generates a group of rationale-answer pairs for each question. Outcome and difficulty filtering followed by rationale self-verification produces a binary admission signal, which both determines relative advantages over the complete current candidate group and selects historical experiences for replay. The two streams then update the same policy.

This is a training framework, not a deployment pipeline that must consult several external medical agents. Self-verification uses the current model in inference mode. Online and replay trajectories are two sources of training data, not two independently answering models.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Medical question<br/>and reference answer"] --> Rollouts["Policy generates<br/>8 candidates per question"]
    Rollouts --> Window["Outcome and<br/>difficulty filtering"]
    Window -->|Pass| Verify["Rationale<br/>self-verification"]
    Window -->|Fail: admission 0| Online["Dual-stream optimization<br/>Online group-relative update"]
    Verify -->|Agreement: 1; otherwise: 0| Online
    Verify -->|Only agreeing trajectories enter buffer| Replay["Dual-stream optimization<br/>Difficult-experience replay"]
    Online --> Update["Combine losses<br/>and update policy"]
    Replay --> Update

Key Designs

1. Outcome and difficulty filtering: keep correct experiences within the model's effective learning range

CARE first extracts the final answer using a rule-based function and compares it with the reference, so this is not label-free self-learning. Incorrect candidates immediately fail admission. It then examines the length-normalized negative log-likelihood of the complete response: the average surprise per token when the model generates that trajectory. Length normalization prevents an explanation from appearing harder simply because it is longer. This measures the model's uncertainty, not physician-defined disease complexity, and it is not a direct computation of entropy over the entire output distribution.

\[ \mathcal L_{\mathrm{seq}}(y\mid x;\theta) =-\frac{1}{|y|}\sum_{t=1}^{|y|}\log\pi_\theta(y_t\mid x,y_{<t}). \]

A FIFO history buffer stores the most recent 2,000 NLL values. Its 0.2 and 0.9 quantiles define the lower and upper bounds, and only candidates inside this window proceed. Values below the lower bound indicate familiar trajectories, whereas values above the upper bound indicate excessive uncertainty. Unlike fixed thresholds, these quantiles move with policy competence and the recent data distribution. A task may be too difficult early in training, enter the useful learning region later, and eventually become too familiar to warrant repeated emphasis.

The window is first and foremost an experience selection rule. The paper motivates it through a gradient variance bound and a non-vanishing learning signal, but bounding NLL alone does not generally bound every parameter gradient without additional assumptions; easy examples are not necessarily shortcuts either. A cautious interpretation is that the authors introduce an executable dynamic difficulty proxy and test its usefulness through isolated-region experiments, rather than establish that every admitted experience has high clinical information value.

2. Rationale self-verification: hide the original question and test whether the explanation supports the same answer

A correct, moderately difficult response can still contain an explanation that merely accompanies the answer. CARE extracts the rationale and places it in a verification prompt asking for a diagnosis based only on that reasoning, hiding the original image and question. The current model then predicts a new answer in inference mode. The candidate passes only if that answer matches its original answer. Because the original candidate has already passed the correctness check, this is not a case of two matching incorrect answers endorsing each other. The algorithm performs correctness and difficulty checks first, avoiding the additional verification call for candidates that already fail either prerequisite.

Let \(\hat a\) denote the answer obtained from the rationale alone and \(\mathcal M\) the rule-based answer extractor. The operational admission rule can be written as:

\[ \Phi(x,y)= \mathbb I[\mathcal M(y)=a^*]\, \mathbb I[\tau_{\mathrm{low}}\leq\mathcal L_{\mathrm{seq}}\leq\tau_{\mathrm{high}}]\, \mathbb I[\hat a=a]. \]

The product makes the conjunction explicit: the three conditions cannot compensate for each other through a weighted score. It rejects experiences whose answer is correct but whose explanation cannot support that answer without the original input. The paper calls this causal sufficiency and compares hiding the input to a do-intervention. Operationally, however, it is an input-ablation consistency check. A rationale may reveal the answer directly or contain consistent but medically false statements, and the same model can share biases across generation and verification. Passing therefore establishes operational agreement, not independently verified clinical causality.

3. Dual-stream optimization: learn relative preferences from current candidates and consolidate difficult historical successes

The online stream retains each question's complete candidate group and standardizes admission scores rather than answer correctness alone. Admitted trajectories receive a relatively positive signal, while rejected trajectories can receive a negative one. Being excluded from replay therefore does not mean disappearing from online optimization. This differs from simply collecting successful samples for supervised learning: the policy learns which responses to the same question are relatively preferable, without requiring a separate value network.

\[ A^{(i)}=\frac{\Phi^{(i)}-\mu(\Phi)}{\sigma(\Phi)+\epsilon}, \qquad \mathcal L_{\mathrm{CARE}}=\mathcal L_{\mathrm{on}}+\lambda\mathcal L_{\mathrm{rep}}. \]

The online objective also includes a KL penalty against a reference policy to constrain excessive deviation. The replay stream samples only historical trajectories that passed every check, reinforces their generation probability, and uses a within-batch Softmax over length-normalized NLL to emphasize less familiar admitted experiences. The methodological overview describes weights as proportional to NLL, whereas the implementation details explicitly specify Softmax normalization; this note follows the implementation description. Replay does not blindly pursue the hardest samples, because excessively difficult trajectories were filtered before entering the buffer.

The streams also complement one another when the current candidate group is uninformative. If every candidate passes or every candidate fails, all admission scores are identical and the displayed advantage is zero, leaving no within-group preference signal; historical replay may still supply a learning signal. This is an implication of the advantage formula, not a separately measured ablation result. Equations (14) and (15) are damaged in the cached PDF text extraction, so their complete layout and signs cannot be reliably recovered. This note does not invent expanded losses, PPO clipping terms, or missing implementation details.

A Worked Example

Consider an image-based medical multiple-choice question whose reference answer is option A. The policy generates 8 candidates, following the paper's setting. The following is an illustrative control-flow example, not a case reported by the authors: one candidate answers A but has NLL above the upper threshold, so its admission is 0. Another answers A and lies inside the window, but the model predicts B when given only its rationale, so it is also rejected. Only a candidate answering A, lying inside the window, and producing A again under self-verification receives admission 1 and enters replay.

All 8 admission values participate in the current online advantage calculation; the mean is not computed over admitted candidates alone. In a later replay update, historical admitted trajectories receive within-batch weights based on their current NLL. No specific NLL values or numbers of passing candidates are assigned here because the paper does not provide such a case-level trace. The example highlights that answer correctness is only CARE's first gate.

Loss & Training

The main backbone is Hulu-Med-7B, with transfer also evaluated on HuatuoGPT-V 7B. Training mixes PMC-VQA, SLAKE, PathVQA, MedMCQA, and PubMedQA; the authors state that training examples do not overlap with benchmark test sets. Training uses 8 A100 GPUs, a rollout batch size of 128, an update batch size of 64, and 8 rollouts per question. These batch-size terms follow the paper without assuming whether their implementation counts prompts or individual sequences.

The replay loss weight is \(\lambda=1.0\), the fixed replay ratio is 50%, and the KL coefficient is 0.04. Experience-based optimization and the replay buffer activate only after batch Pass@1 reaches 35%. The paper reuses \(\beta\) for both the upper quantile 0.9 and the KL coefficient 0.04; this note distinguishes them by name rather than treating them as the same hyperparameter. The warm-start threshold also means that CARE depends on a backbone with some existing task competence, rather than starting from a model entirely unable to answer.

Key Experimental Results

Main Results

The following selection comes from Tables 1 and 2. Scores retain the paper's percentage scale, with higher being better; gains are percentage-point differences against the same Hulu-Med-7B backbone, not relative percentages. MedXQA appears separately in the multimodal and text tables and must not be conflated into one result.

Evaluation	Hulu-Med-7B	CARE-7B	Gain (percentage points)
OmniVQA	84.2	85.6	+1.4
PMC-VQA	66.8	68.2	+1.4
VQA-RAD	78.0	79.3	+1.3
SLAKE	86.8	88.1	+1.3
PathVQA	65.6	67.1	+1.5
MedXQA (multimodal)	29.0	31.2	+2.2
MMMU-Med	51.4	53.0	+1.6
MMLU-Pro-Med	60.6	62.4	+1.8
MedXQA (text)	19.6	22.1	+2.5
PubMedQA	77.4	78.6	+1.2
MedMCQA	67.6	69.1	+1.5
MedQA	73.5	75.0	+1.5
MMLU-Med	79.5	81.1	+1.6

These results support consistent improvement over the same backbone, not universal superiority over every model on every task. For example, Table 1 reports 76.9 on MMMU-Med for Gemini-2.5-Flash, above CARE's 53.0. On HuatuoGPT-V 7B, PMC-VQA rises from 53.1 to 54.6 and MedQA from 52.9 to 54.6, suggesting the gains are not unique to Hulu. Nevertheless, demonstrated backbone transfer remains limited to these two 7B medical VLMs.

Ablation Study

Every configuration in Table 3 starts from Hulu-Med-7B. Averages retain the paper's one-decimal precision.

Configuration	PMC-VQA	MedQA	MMMU-Med	Average
Base SFT	66.8	73.5	51.4	63.9
Standard GRPO, outcome-only	67.2	73.8	51.8	64.3
CARE without causal sufficiency	67.5	74.2	52.1	64.6
CARE without learnability window	67.8	74.4	52.4	64.9
CARE without difficult-experience replay	68.0	74.7	52.6	65.1
Full CARE	68.2	75.0	53.0	65.4

Relative to full CARE, removing the three components lowers the average by 0.8, 0.5, and 0.3 percentage points, respectively; self-verification has the largest measured effect in this evaluation. CARE exceeds standard GRPO by 1.1 points and SFT by 1.5 points. Setting \(\lambda=0\) removes the entire replay stream, so this ablation does not isolate the benefit of Softmax difficulty weighting over uniform replay.

Key Findings

The distractor experiment in Figure 3 injects irrelevant symptoms and noise into MedQA questions, exposing a larger difference than clean-set scores alone.

MedQA configuration	Original accuracy (%)	Noisy accuracy (%)	Drop (percentage points)
SFT base	73.5	60.2	13.3
Standard GRPO	73.8	58.5	15.3
CARE	75.0	71.2	3.8

The human analysis randomly samples 500 MedQA test questions answered correctly by both methods, with majority voting by 3 experts. For standard GRPO, 45.0% of rationales are classified as spurious shortcuts; for CARE, 86.2% are classified as valid causal reasoning. These are different categories and cannot be subtracted as an improvement in the same metric. The conditional-subset figure of 86.2% is also not a whole-test-set reliability estimate.

Self-verification rejection among correct trajectories declines from 48.5% to 15%, alongside an increase in valid reasoning. In isolated training on different NLL regions, the paper reports gains of +0.4% for trivial experiences, +5.2% for the proximal window, and -1.2% for extremely difficult experiences. The cache does not clearly identify the exact benchmark aggregation behind these gains, so the authors' notation is retained without presenting +5.2% as a MedQA or main-table average improvement. Gradient-norm and training curves support improved stability, but reliable point-by-point curve values are not available from the cached text.

Highlights & Insights

• The reward targets usable experiences, not just answers. A shared admission signal controls both online rewards and replay entry, so whether a rationale supports its answer affects parameter updates rather than remaining a post hoc explanation score.
• Dynamic difficulty operates at two levels. The window removes unsuitable extremes, while replay emphasizes comparatively difficult admitted experiences. Filtering determines the acceptable region; weighting determines attention within it.
• Distractor robustness is more revealing than a small average-score gain. Standard GRPO slightly improves clean MedQA performance yet falls below SFT under noise, showing why answer accuracy alone is an inadequate measure of reasoning quality.

Limitations & Future Work

• Agreement does not establish causal correctness. This is the note's critical assessment: answer leakage, preserved question shortcuts, and shared model biases can all allow self-verification to pass. Answer masking, factual perturbations, and independent expert blind review would help distinguish reproduction from medical deduction.
• Theoretical guarantees require fuller assumptions. The cached proofs are brief and do not adequately establish the general claims that agreement filtering eliminates every shortcut or that an NLL window necessarily bounds gradient variance. Evidence that the mechanism helps is not verification of an unconditional theorem.
• Evaluation scope and statistical uncertainty remain limited. The 500 human-reviewed examples cover only the jointly correct subset; the cache does not report expert agreement coefficients, repeated-run variance, or confidence intervals. Noisy MedQA is not a substitute for prospective validation across hospitals and populations.
• Reproduction and cost details are incomplete. The cache omits complete training time, verification overhead, replay capacity, and historical-trajectory reverification policy, and does not separately ablate uniform replay. Equal-compute comparisons and checks of whether old experiences remain learnable and self-consistent as the policy changes would strengthen the evidence.

Related Work & Insights

• Compared with standard GRPO: Both use within-group relative signals rather than a separate value network. CARE changes the reward criterion and adds replay of admitted historical experiences, so its gains should not be attributed solely to using RL.
• Compared with Hulu-Med and HuatuoGPT-V: These provide medical vision-language backbones, while CARE supplies a training method on top. Same-backbone comparisons reveal the incremental value of experience curation more clearly than comparisons across model scales.
• Compared with Med-R1 and MedVLM-R1: These also investigate reinforcement learning for medical VLMs. CARE focuses on experience admission and rationale consistency beyond final-answer correctness, rather than introducing another image-encoding module.
• Transferable lesson: In tasks with reference answers but no process annotations, one can test whether an intermediate artifact supports solving the task after removing the original input. The artifact must also be checked for answer leakage, or the test may collapse into information copying.

Rating

• Novelty: 4/5. Rationale self-verification, a dynamic difficulty window, and dual-stream training form a targeted admission mechanism, although the causal framing exceeds what the operational test establishes.
• Experimental Thoroughness: 3/5. Multimodal and text evaluations, a second backbone, component ablations, and human analysis are included, but variance estimates, compute fairness, and external clinical validation are missing.
• Writing Quality: 3/5. The pipeline and hyperparameters are reasonably clear, but the boundary between theoretical claims and verifiable evidence needs more care; damaged formula extraction is separately a reproduction limitation of this note.
• Value: 4/5. The experience-selection approach is reusable and its distractor-robustness results are notable, but these findings do not establish readiness for clinical deployment.