Skip to content

Criterion-Activated Preference Synthesis: Attributed Alignment for Medical Dialogue Agents

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Medical NLP
Keywords: Preference data synthesis, Attribution ambiguity, Direct preference optimization, Dual-signal scoring, Clinical dialogue alignment

TL;DR

This paper introduces Criterion-Activated Preference Synthesis (CAPS), a criterion-first preference synthesis framework that seeds context generation with explicit quality standards, decomposes evaluations into rubric-bound and general blind judges, and applies group-level Empirical Activation Tests (EAT) and pair-level Attribution Ratio (AR) filters to isolate clean, causal preference signals—outperforming a 22k SFT baseline and matching 35k human preference pairs on HealthBench with only 2k synthesized pairs.

Background & Motivation

Deploying large language models as medical dialogue and clinical decision-support agents demands rigorous adherence to clinical guidelines, strict factual correctness, and patient-centered communication safety. Direct Preference Optimization (DPO) has emerged as the default objective for post-training alignment; however, it optimizes solely over binary chosen-versus-rejected labels. In high-stakes medical dialogues, a preferred clinical response frequently exhibits multiple confounded advantages over a rejected one simultaneously—such as citing primary clinical guidelines, providing empathetic explanations, using clearer formatting, and offering exhaustive detail. Under standard pairwise loss formulations, the optimization gradient cannot differentiate whether the preference signal rewards life-saving pharmacological discrimination or merely penalizes shorter length and terse tone, an inherent defect termed attribution ambiguity.

Scaling up preference data through multi-model response harvesting does not resolve this attribution dilemma and frequently exacerbates reward hacking. When preferences are learned across diverse model families and scales, stylistic idiosyncrasies, verbosity biases, and superficial conversational markers systematically inflate holistic evaluation gaps. Consequently, the downstream policy tends to overfit to rhetorical fluency and uncritical refusals rather than mastering foundational diagnostic and contraindication reasoning. Furthermore, when evaluation rubrics are assigned post hoc to pre-existing dialogue corpora, many criteria fail to activate meaningfully—enforcing multi-step pharmacokinetic reasoning on a straightforward triage inquiry introduces irrelevant token overhead and injects high-variance statistical noise into preference optimization.

Resolving this tension requires replacing the conventional "collect responses first, annotate criteria post hoc" pipeline with a criterion-first generation and dual-signal verification paradigm. Instead of hoping that models disentangle medical norms from uncontrolled multi-model competitions, practitioners should synthesize clinical contexts directly from explicit standards and deploy prompt-separated judges to measure both criteria compliance and holistic quality. Core idea: Introduce Criterion-Activated Preference Synthesis (CAPS), which anchors query generation and response rollouts in explicit quality standards, evaluates candidates via decoupled rubric-bound and blind judges, filters uninformative contexts via the group-level Empirical Activation Test (EAT), and audits pair-level preference purity via the Attribution Ratio (AR) to guarantee that the learned preference is strictly caused by the target standard rather than surface confounders.

Method

Overall Architecture

The CAPS pipeline is organized into five sequential, modular stages: target criterion instantiation from domain standards, criterion-activated context synthesis with multi-source candidate rollouts, dual-signal scoring under prompt-level information asymmetry, group-level discriminative verification via the Empirical Activation Test (EAT), and pair-level causal auditing via the Attribution Ratio (AR) coupled with geometric quality constraints. The resulting verified preference triplets \((x, y^+, y^-)\) are fed directly into standard DPO without exposing the criterion text to downstream training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Explicit Standard Bank<br/>(Clinical Guidelines / Statutory Rules / General Rubrics)"] --> GEN["Context Synthesis & Multi-Source Rollout<br/>(Generate x & M multi-temperature candidates)"]
    GEN --> D1["Dual-Signal Scoring & Partitioning<br/>Signal A Rubric Judge vs. Signal B Blind Judge"]
    D1 --> D2["Empirical Activation Test (EAT)<br/>Calculate blind score Cohen's d to filter contexts"]
    D2 -->|Retain d in [0.5, 2.0]| D3["Attribution Ratio (AR) Audit & Selection<br/>Calculate AR & margin, enforce directional constraints"]
    D3 --> DPO["Downstream Policy Optimization<br/>(Train policy π_θ via standard DPO)"]

Key Designs

1. Dual-Signal Scoring & Partitioning: Decoupling criteria compliance from holistic perceived quality -> Implementing prompt-isolated rubric and blind evaluators with double-sided thresholding -> Eliminating confirmation bias and providing clean response groups

Single-agent LLM evaluators charged with judging both rule compliance and overall response quality suffer from severe confirmation bias: observing that a candidate adheres to a guideline artificially inflates the judge's holistic rating. CAPS physically separates evaluation into two API calls with disjoint prompt contexts. Judge \(J^{(A)}\) (Signal A, rubric-bound) receives the query, response, and target criterion \(c\), outputting a compliance score \(s^{(A)} \in [1, 10]\). Judge \(J^{(B)}\) (Signal B, general blind) is entirely blinded to \(c\), evaluating the response strictly on general clinical relevance, factuality, coherence, and safety on a \(1\text{--}10\) scale. To curb stochastic decoding variance, each judge is sampled \(K_{vote} = 3\) times, and the scores are aggregated via trimmed mean (the median at \(K_{vote}=3\)): $\(s_k = \text{TrimmedMean}\left(\{s_{k, t}\}_{t=1}^{K_{vote}}\right)\)$ Responses are then partitioned via a double-sided threshold on Signal A: candidates with \(s_k^{(A)} > \tau^+\) form the satisfy group \(Y^+\), while those with \(s_k^{(A)} < \tau^-\) form the violate group \(Y^-\) (default \(\tau^+ = \tau^- = 7\)). Ambiguous responses in the middle tier (\(v_k = \bot\)) are discarded immediately. This ensures clean separation on the target medical norm while feeding unbiased scores into downstream statistical testing.

2. Empirical Activation Test (EAT): Preventing uninformative standards from polluting the preference distribution -> Computing Cohen's d over blind scores with double-sided truncation -> Discarding degenerate contexts with negligible activation or collapsed variance

Not every clinical standard produces a meaningful preference signal in every scenario. If satisfying a criterion does not shift the perceived response quality from a blind perspective, the context is clinically uninformative and contributes only noise. EAT operates at the context group level by computing the standardized effect size Cohen's \(d\) on the blind Signal B scores between \(Y^+\) and \(Y^-\): $\(d = \frac{\bar{s}^{(B), +} - \bar{s}^{(B), -}}{s_p}, \quad s_p = \sqrt{\frac{(n_+ - 1)\sigma_+^2 + (n_- - 1)\sigma_-^2}{n_+ + n_- - 2}}\)$ requiring \(|Y^+|, |Y^-| \ge 2\). CAPS enforces a double-sided activation band \(d \in [\tau_{min}, \tau_{max}] = [0.5, 2.0]\). When \(d < 0.5\), criterion satisfaction fails to distinguish quality, either due to true irrelevance in that patient vignette or excessive variance across heterogeneous rollout models. When \(d > 2.0\), the gap reflects either a trivial distinction (such as hallucinated off-topic tokens that invite surface reward hacking) or an estimation artifact where coarse ratings cause pooled standard deviation to collapse (\(s_p \to 0\)). Contexts outside \([0.5, 2.0]\) are pruned before pair construction.

3. Attribution Ratio (AR) Audit & Selection: Eliminating multi-model confounders and verbosity bias in individual pairs -> Normalizing criterion-specific margin against holistic margin under geometric bounds -> Isolating pure, criterion-driven preference gradients

Even within an EAT-passing context, an arbitrarily sampled pair \((y^+, y^-)\) may exhibit a quality gap driven primarily by extraneous factors like model capacity or text length rather than the target standard. To quantify attribution purity, CAPS introduces the Attribution Ratio (AR): $\(\Delta_A = s^{(A)}_{y^+} - s^{(A)}_{y^-}, \quad \Delta_B = s^{(B)}_{y^+} - s^{(B)}_{y^-}\)$ $\(AR = \frac{\Delta_A}{\max(\Delta_A, \Delta_B) + \epsilon} \in [0, 1]\)$ When the criterion-specific gap dominates (\(\Delta_A \ge \Delta_B\)), \(AR \approx 1\); when stylistic or superficial gaps overshadow the criterion (\(\Delta_B \gg \Delta_A\)), \(AR\) drops sharply. Pairs must satisfy \(AR \ge \tau_{AR} = 0.6\), supplemented by four empirical constraints: ① Directional consistency: \(\Delta_A > 0\) and \(\Delta_B > 0\), eliminating paradoxical instances where satisfying the guideline harmed blind quality; ② Chosen quality floor: \(s^{(B)}_{y^+} \ge 6.0\), ensuring chosen outputs are genuinely competent; ③ Margin bounds: \(\Delta_B \in [1.0, 5.0]\), avoiding imperceptible nuances and trivial extremes; ④ Diversity cap: each response appears in at most \(K_{rep} = 2\) pairs. Valid pairs are ranked by \(\text{score}(y^+, y^-) = AR \times \Delta_B\), retaining the top \(K_{pair} = 5\) pairs to maximize both attribution purity and gradient strength.

Loss & Training

CAPS keeps the training objective completely standard: policy \(\pi_\theta\) is aligned using Direct Preference Optimization (DPO) over the curated pairs, while reference model \(\pi_{ref}\) serves as the regularization anchor. The objective optimizes: $\(\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{CAPS}}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y^+ \mid x)}{\pi_{ref}(y^+ \mid x)} - \beta \log \frac{\pi_\theta(y^- \mid x)}{\pi_{ref}(y^- \mid x)}\right)\right]\)$ Hyperparameters follow standard DPO configurations: temperature \(\beta = 0.1\), global batch size 32, learning rate \(2 \times 10^{-6}\) with cosine decay schedule and 5% linear warmup, trained for 3 epochs with maximum sequence length capped at 4096 tokens. The criterion \(c\) is never presented during training, guaranteeing algorithmic compatibility with alternative objectives such as IPO or KTO.

Key Experimental Results

Main Results

On HealthBench (evaluating clinical dialogue, medical factuality, and communication safety) alongside general and statutory legal benchmarks, CAPS achieves exceptional data efficiency and downstream alignment quality. All reported numbers are transcribed directly from the paper:

Domain & Benchmark Base Model Baseline Method Baseline Data Size Baseline Score CAPS Setting CAPS Data Size CAPS Score Net Improvement / Delta
Medical HealthBench Qwen3-8B Base Model
HelpSteer3 DPO
RaR-Medicine SFT
–
35k
22k
41.53
45.19
41.79
CAPS-Medical 2k 45.30 +3.77 over Base; matches 35k human pairs
Medical HealthBench Qwen3-14B Base Model
HelpSteer3 DPO
RaR-Medicine SFT
–
35k
22k
45.34
47.59
44.07
CAPS-Medical 2k 47.91 +2.57 over Base; beats 22k SFT by +3.84
Instruction IFBS (Strict) Qwen3-8B HelpSteer3 DPO 35k 32.33 CAPS-General 20k 35.00 +2.67 over 35k human annotations
Creative Writing v3 Qwen3-8B HelpSteer3 DPO 35k 58.96 CAPS-General 20k 74.53 +15.57 over 35k human pairs (nears 32B)
Legal Reasoning PLaw Qwen3-8B Infinity-Instruct SFT 10k 26.39 CAPS-Legal 6k 47.89 +21.50 over SFT; +6.60 over Base
Legal Retrieval MSLR Qwen3-8B Base (55.58) / Inf-Inst 10k 31.43 CAPS-Legal 6k 61.62 +6.04 over Base; reverses SFT collapse

Ablation Study

The authors conducted systematic component-level ablations and sensitivity analyses in the medical domain (HealthBench with Qwen3-8B) and across general and legal benchmarks:

Ablation / Sensitivity Configuration Configuration Details HealthBench (8B) IFBench (Strict) PLawBench (Legal)
Full CAPS Pipeline \(d \in [0.5, 2.0], \tau_{AR} = 0.6, \tau = (7, 7)\) 45.30 35.00 47.89
Remove AR Ranking Retain AR-filtered pairs, sample randomly – 33.00 (-2.00) 46.95 (-0.94)
Remove AR Entirely Retain all EAT-passing pairs without AR audit 44.62 (-0.68) 31.33 (-3.67) 46.65 (-1.24)
Remove EAT Entirely All context-criterion pairs advance to selection 44.81 (-0.49) 28.67 (-6.33) 46.21 (-1.68)
Random Pairing Control Random chosen-rejected assignment from rollouts – 29.67 33.39 (-14.50)
Single Judge: Blind Only Select pairs purely by Signal B holistic margin – 31.67 42.49
Single Judge: Rubric Only Select pairs purely by Signal A rubric margin – 31.00 44.10
EAT Threshold: Wide \(d \in [0.2, 3.0]\) 45.22 – –
EAT Threshold: Narrow \(d \in [0.8, 1.5]\) 45.54 – –
AR Threshold: \(\tau_{AR} = 0.4\) Relaxed attribution purity 45.37 – –
AR Threshold: \(\tau_{AR} = 0.8\) Strict attribution purity 45.50 – –

Key Findings

  • Direct SFT on domain corpora risks degrading complex reasoning, whereas verified preference optimization recovers latent competence: On HealthBench, applying direct SFT on 22k RaR-Medicine examples degrades Qwen3-14B performance from 45.34 to 44.07. Similarly, Infinity-Instruct SFT causes PLawBench to collapse from 41.29 to 26.39. In sharp contrast, converting raw guidelines into verified preference pairs via CAPS unlocks structured reasoning without distributional collapse.
  • EAT and AR address orthogonal causal vulnerabilities: Removing EAT allows uninformative contexts to introduce evaluation noise, causing a severe 6.33-point plunge on IFBS. Removing AR allows superficial length and model-scale stylistic differences to drive preferences, dropping IFBS by 3.67 points and degrading HealthBench to 44.62.
  • Robustness across heterogeneous blind judges confirms objective attribution: In an audit replacing the blind judge with GPT-5.4-mini, Gemini-3.5, and Claude-4.6 across 50 stratified pairs (Table 12), Spearman rank correlation remained between 0.83 and 0.84, with a perfect 1.00 Jaccard overlap among top-30% retained pairs, proving that AR measures genuine cross-model factual superiority.

Highlights & Insights

  • Paradigm shift from post-hoc labeling to criterion-first generation: Instead of collecting noisy pairwise comparisons and struggling to explain them after the fact, CAPS operationalizes explicit guidelines as prospective generative seeds and retrospective audit boundaries.
  • Information-asymmetric dual judging neutralizes confirmation bias: Restricting the rubric judge to compliance scoring and blinding the general judge to the rubric creates an adversarial check-and-balance, enabling mathematical attribution bounding without requiring complex reward modeling.
  • Extreme sample efficiency for high-stakes specialization: CAPS achieves state-of-the-art alignment with merely 2k medical and 6k legal pairs, demonstrating that data purity and causal attribution vastly outweigh raw volume in expert domains.

Limitations & Future Work

  • High verification inference cost: The medical synthesis pipeline generated over 5.37 million LLM judge calls across rollouts and stochastic voting, totaling an estimated API expenditure of $1,600 to $1,900 for 2k retained pairs.
  • Absence of dynamic multi-criterion conflict resolution: The current implementation treats individual guidelines independently or in static combinations; it cannot yet arbitrate conflicting clinical directives in highly complex, multi-morbidity patient scenarios.
  • Judge backbone reliance: The core pipeline relies on DeepSeek-V3.2 for both prompts. Although small-scale evaluations with external proprietary judges showed high concordance, full structural independence between signals remains an open theoretical goal.
  • vs. rDPO & CLAIR: rDPO applies rubrics post hoc as soft weights over existing datasets without controlling generation contexts; CLAIR uses minimal-edit heuristics to isolate differences but achieves only 43% Jaccard alignment with intended targets. CAPS guarantees causal alignment end-to-end via active context synthesis and quantitative AR thresholding.
  • vs. PVar & Difficulty Selection: PVar evaluates prompt-level diversity but ignores whether criteria activate meaningful quality shifts; difficulty-based filtering evaluates margin magnitude without tracing its causal source. EAT and AR jointly resolve both gaps.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the preference attribution ambiguity problem with mathematical clarity and proposes the pioneering EAT/AR dual-signal framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-domain validation on medical (HealthBench), legal, and general benchmarks, enriched with distillation controls and judge robustness sweeps.
  • Writing Quality: ⭐⭐⭐⭐⭐ Outstanding technical rigor, precise terminology, disciplined mathematical notation, and compelling case analyses.
  • Value: ⭐⭐⭐⭐⭐ Delivers an actionable, high-efficiency alignment methodology for mission-critical medical dialogue systems and safety-sensitive AI applications.