Skip to content

Criterion-Activated Preference Synthesis: Attribution-Preserving Preference Data from Explicit Standards

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Preference Optimization, Direct Preference Optimization, Attribution Ambiguity, Synthetic Data, Dual-Signal Evaluation

TL;DR

To resolve preference attribution ambiguity where binary chosen-rejected pairs conflate style, length, and correctness, CAPS prospectively synthesizes contexts from explicit standards, decouples rubric adherence from holistic quality via dual-signal scoring, and purifies pairs through group-level Empirical Activation Tests (EAT) and pair-level Attribution Ratios (AR)—achieving superior performance across general, medical, and legal benchmarks with only 20k synthetic pairs compared to 35k human HelpSteer3 pairs.

Background & Motivation

Direct Preference Optimization (DPO) has emerged as an efficient and widely adopted paradigm for steering large language models, but its optimization relies entirely on unadorned chosen–rejected response pairs. In practice, a preferred response often differs from its rejected counterpart across multiple entangled dimensions simultaneously—it may cite primary sources more accurately, display greater stylistic polish, adhere to better markdown formatting, and exhibit longer verbosity. Because standard preference losses distill this multidimensional discrepancy into a single scalar ranking signal, the learning objective provides no structural indication of which specific behavioral shift drove the preference label. This fundamental ambiguity regarding which underlying behavior caused the quality differential is termed preference attribution ambiguity.

Scaling synthetic preference collection across heterogeneous model pools drastically exacerbates this issue rather than curing it. Aggregating multi-source rollouts inevitably introduces superficial confounders such as signature formatting styles, model-specific tones, and systematic length discrepancies across model sizes. Under naive contrastive loss formulations, the optimization policy frequently exploits these surface-level differences, inducing severe reward hacking and leading to degenerated, overly verbose, or stylized outputs. While recent post-hoc curation pipelines attempt to filter noisy or outlier margins after collection, they operate downstream of data generation and cannot reconstruct the lost causal link between explicit quality standards and observed preference margins.

The essential path forward lies in inverting the conventional pipeline: rather than collecting unconstrained response pairs and guessing their quality drivers post hoc, one should prospectively position explicit, fine-grained evaluation criteria as the generative seed and verification anchor of preference synthesis. Core idea: propose Criterion-Activated Preference Synthesis (CAPS), a criterion-first framework that prospectively synthesizes realistic contexts from explicit standards, evaluates rollouts via prompt-separated "rubric-bound (Signal A)" and "general blind (Signal B)" judges, and applies group-level Empirical Activation Tests (EAT) alongside pair-level Attribution Ratios (AR) to guarantee that downstream preference optimization learns clean, attribution-preserving behavioral shifts.

Method

Overall Architecture

The CAPS pipeline operationalizes explicit quality criteria into attribution-verified preference pairs through five sequential stages: prospective context synthesis from an explicit criterion bank, diverse multi-temperature response rollout from heterogeneous model families, prompt-isolated dual-signal scoring, group-level context filtering via Empirical Activation Tests (EAT), and pair-level confounding audit and ranking via Attribution Ratios (AR). The retained triplets \((x, y^+, y^-)\) are subsequently trained with standard DPO without exposing the explicit criterion text during policy optimization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Explicit Criterion Bank C<br/>(General Rubrics / Clinical Guidelines / Legal Statutes)"] --> B["Prospective Context Synthesis & Multi-Source Rollout<br/>Generate Contexts from Target Criterion & Sample Candidates"]
    B --> C["Prompt-Separated Dual-Signal Scoring<br/>Rubric-Bound J(A) vs. General Blind J(B) Independent Scoring"]
    C --> D["Empirical Activation Test (EAT)<br/>Group-Level Cohen's d Effect Size Filtering on Blind Scores"]
    D --> E["Attribution Ratio Audit & Pair Selection (AR)<br/>Pair-Level Confounder Disentanglement & Margin Joint Ranking"]
    E --> F["Standard DPO Policy Optimization<br/>Train Target Policy on Attribution-Preserved (x, y⁺, y⁻) Triplets"]

Key Designs

1. Prospective Context Synthesis & Multi-Source Rollout: Grounding Queries in Natural Criterion Sensitivity

Existing synthetic workflows typically sample arbitrary instructions and subsequently retro-fit evaluation rubrics, frequently resulting in uninformative prompts where the rubric never naturally activates. CAPS reverses this sequence: starting from an explicit quality criterion \(c \in \mathcal{C}\) (e.g., statutory provisions, clinical guidelines, or nuanced writing rules), a generator model \(G\) (such as DeepSeek-V3.2) synthesizes \(N\) diverse user contexts \(x\) where satisfying \(c\) naturally dictates response quality. The prompt enforces ecological validity and rich factual density while forbidding meta-language or verbatim quotation of the criterion in the query, preventing trivial lexical shortcuts. For each context \(x\), responses are gathered across a multi-model pool \(\mathcal{M}\) over varied decoding temperatures \(\{0.5, 0.7, 0.9, 1.0\}\), yielding \(K_{\text{roll}} = 20\) candidate rollouts spanning wide quality and stylistic spectra.

2. Prompt-Separated Dual-Signal Scoring: Eliminating Confirmation Bias via Information Asymmetry

No single scoring prompt can objectively determine whether a response satisfies a specific target standard while simultaneously judging whether doing so genuinely enhances overall perceived utility. CAPS addresses this cognitive conflict by splitting judgment into two strictly segregated channels: Signal A (rubric-bound) and Signal B (general blind). Judge \(J^{(A)}\) is provided with full access to criterion \(c\) and scores each candidate \((x, y_k)\) on a 1–10 scale, utilizing a double-sided threshold \((\tau^+, \tau^-) = (7.0, 7.0)\) to partition rollouts into a satisfy group \(Y^+\) and a violate group \(Y^-\) (discarding ambiguous intermediate scores). Conversely, judge \(J^{(B)}\) evaluates the identical candidate completely blind to \(c\), assigning a continuous 1–10 holistic quality score \(s^{(B)}\) covering fluency, factual accuracy, reasoning depth, and coherence. Executed via disjoint API requests with zero shared attention context and stabilized via \(K_{\text{vote}} = 3\) stochastic evaluations aggregated by trimmed mean (median), this asymmetry ensures that Signal B serves as an uncorrupted arbiter of true perceptual gain.

3. Empirical Activation Test (EAT): Discarding Contexts with Negligible or Trivial Signal at Group Level

Not every synthetic context legitimately activates a criterion—demanding legal statutory citations on a creative writing query simply injects spurious noise. To prune uninformative contexts before forming preference pairs, CAPS introduces the group-level Empirical Activation Test (EAT). For a given context \(x\), EAT evaluates Cohen's \(d\) effect size strictly on the Signal B blind scores between the satisfy group \(Y^+\) and violate group \(Y^-\):

\[d = \frac{\bar{s}^{(B), +} - \bar{s}^{(B), -}}{s_p}, \quad s_p = \sqrt{\frac{(n_+ - 1)\sigma_+^2 + (n_- - 1)\sigma_-^2}{n_+ + n_- - 2}}\]

Crucially, computing \(d\) on Signal B rather than Signal A prevents tautological validation; it confirms that satisfying \(c\) measurably improves blind quality rather than simply showing the evaluator can parse the rubric. CAPS applies a two-sided interval gate \(d \in [\tau_{\min}, \tau_{\max}] = [0.5, 2.0]\). Values below \(0.5\) indicate that the criterion fails to activate or is drowned by intra-group model variance, whereas values exceeding \(2.0\) indicate degenerate triviality (such as off-topic gibberish or zero-variance artifacts) that encourage surface reward hacking. Contexts outside this interval are discarded.

4. Attribution Ratio Audit & Pair Selection (AR): Disentangling Non-Causal Confounders at Pair Level

Even within an EAT-verified context, an arbitrary candidate pair \((y^+, y^-)\) may still exhibit a quality disparity driven primarily by stylistic quirks, model scale, or length rather than criterion compliance. To audit individual pairs, CAPS defines the Attribution Ratio (AR) by contrasting the criterion-specific margin \(\Delta_A\) against the holistic blind margin \(\Delta_B\):

\[\Delta_A = s^{(A)}_{y^+} - s^{(A)}_{y^-}, \quad \Delta_B = s^{(B)}_{y^+} - s^{(B)}_{y^-}\]
\[\text{AR} = \frac{\Delta_A}{\max(\Delta_A, \Delta_B) + \epsilon} \in [0, 1]\]

When non-target confounders (such as verbose padding or superficial elegance) dominate the preference gap, \(\Delta_B \gg \Delta_A\), driving \(\text{AR}\) toward zero and triggering exclusion. Pairs are retained only if \(\text{AR} \ge \tau_{\text{AR}} = 0.6\), supplemented by directional consistency (\(\Delta_A > 0, \Delta_B > 0\)), minimum chosen quality (\(s^{(B)}_{y^+} \ge 6.0\)), quality margin bounds (\(\Delta_B \in [1.0, 5.0]\)), and a response diversity cap (\(K_{\text{rep}} \le 2\)). Qualifying pairs are then ranked via a joint criterion-attribution objective:

\[\text{score}(y^+, y^-) = \text{AR} \times \Delta_B\]

The top \(K_{\text{pair}} = 5\) pairs per \((x, c)\) are selected, ensuring that downstream updates are driven by high signal magnitude without sacrificing attribution specificity.

Loss & Training

The filtered preference dataset \(\mathcal{D}_{\text{CAPS}} = \{(x, y^+, y^-)\}\) is optimized using standard Direct Preference Optimization (DPO):

\[\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\text{CAPS}}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y^+ \mid x)}{\pi_{\text{ref}}(y^+ \mid x)} - \beta \log \frac{\pi_\theta(y^- \mid x)}{\pi_{\text{ref}}(y^- \mid x)} \right) \right]\]

Policy models (Qwen3-8B and Qwen3-14B) are trained with temperature coefficient \(\beta = 0.1\), global batch size 32, learning rate \(2 \times 10^{-6}\) with cosine decay schedule, warmup ratio 0.05, and max sequence length 4096 over 3 epochs using LLaMA-Factory. Because the explicit criterion \(c\) is strictly omitted during training, performance gains stem entirely from the attribution purity of the data itself.

Key Experimental Results

Main Results

CAPS was extensively benchmarked across general-purpose alignment (IFEval, IFBench, AlpacaEval LC, Creative Writing v3), clinical healthcare reasoning (HealthBench), and Chinese legal reasoning and retrieval (PLawBench, MSLR), benchmarked against base models, the 35k human-annotated HelpSteer3 dataset, and domain-specific SFT sets.

Domain / Benchmark Metric Qwen3-8B Base HelpSteer3 (35k DPO) CAPS (Ours) Relative Gain / Contrast
General: IFEval Strict Acc / Loose Acc 82.26 / 85.95 84.47 / 87.43 86.32 / 89.46 +1.85 / +2.03 over 35k HelpSteer3
General: IFBench Strict Acc / Loose Acc 25.67 / 30.00 32.33 / 36.67 35.00 / 38.33 +2.67 / +1.66 over 35k HelpSteer3
General: AlpacaEval Length-Controlled WR 57.00 66.51 68.59 +2.08 over HelpSteer3
General: Creative Writing v3 Score 59.00 58.96 74.53 +15.57 over HelpSteer3 (approaching 32B)
Medical: HealthBench Clinical QA Acc (2k pairs) 41.53 45.19 45.30 2k synthetic pairs match 35k human pairs
Legal: PLawBench Legal Practice Acc (6k pairs) 41.29 44.94 47.89 +6.60 over Base, +2.95 over HelpSteer3
Legal: MSLR Multi-Step Reasoning Acc 55.58 45.03 61.62 +6.04 over Base, avoids baseline degradation

On the Qwen3-14B backbone, CAPS preserves identical superiority: reaching 38.33 IFBS, 79.66 Creative Writing v3 (surpassing Qwen3-32B at 77.04), 51.25 on PLawBench, and 65.77 on MSLR (surpassing Qwen3-32B at 65.08). Crucially, the 10k Infinity-Instruct legal SFT baseline severely degrades Qwen3-8B on PLawBench and MSLR down to 26.39 and 31.43, demonstrating the hazards of unverified domain instruction tuning.

Ablation Study

Ablation experiments conducted on Qwen3-8B isolate pipeline components on general instruction following (IFBench Strict) and professional statutory reasoning (PLawBench):

Configuration / Ablation Level IFBench (IFBS) PLawBench Mechanism & Analytical Note
Full CAPS (Dual-Signal EAT + AR) 35.00 47.89 Complete group-level activation and pair-level attribution
Single Judge Control: Blind Judge Only (\(J^{(B)}\)) 31.67 42.49 Lacks criterion focus; vulnerable to superficial style
Single Judge Control: Rubric Judge Only (\(J^{(A)}\)) 31.00 44.10 Lacks holistic check; permits unnatural rule-satisfying text
Random Sampling Baseline 29.67 33.39 Inverts legal doctrine direction; severe degradation (-7.90)
Component Ablation: Remove AR Ranking (\(\text{AR} \times \Delta_B\)) 33.00 46.95 Random selection within passing pairs weakens gradient signal
Component Ablation: Remove AR Filter Entirely 31.33 46.65 Multi-model stylistic confounders degrade preference updates
Component Ablation: Remove EAT Filter Entirely 28.67 46.21 Unactivated contexts introduce extreme noise into pair selection

Furthermore, distillation-control baselines in Table 2 demonstrate that SFT on rollouts from Qwen3-235B or Gemma-4-31B yields lower performance (24.33 / 23.33 IFBS) than the base model (25.67), establishing that CAPS's performance gains stem from contrastive attribution preservation rather than teacher distillation.

Key Findings

  • Complementary Dual Safety Gates: EAT and AR target orthogonal failure modes. Removing EAT triggers the sharpest drop on IFBS (-6.33 points), proving that attempting to audit pairs in contexts where the criterion never naturally activated produces spurious preferences from judge noise. Conversely, removing AR steadily degrades both general and legal benchmarks, confirming the necessity of penalizing \(\Delta_B \gg \Delta_A\) stylistic confounders.
  • Sensitivity of Domain Reasoning to Inverted Preferences: While random pair selection retains modest performance on general benchmarks (29.67 IFBS) due to broad lexical overlap, it drops catastrophic points on PLawBench (33.39 vs. 41.29 base). Because legal answers can diverge in doctrinal validity while maintaining equal fluency, random direction reinforces legally incorrect logic.
  • Data-Efficiency Superiority: Under a constrained 5k HelpSteer3 rubric budget, CAPS matches the 5k original DPO baseline with only 3k synthesized pairs and surpasses the 10k original DPO baseline at 6.5k pairs (32.33 IFBS / 37.00 IFBL vs. 31.33 / 36.67), proving superior sample efficiency.

Highlights & Insights

  • Dual-Signal Prompt Asymmetry: By decoupling rubric verification from holistic blind evaluation through separate API calls, CAPS eliminates judge circularity without requiring specialized, multi-head reward models. The mathematical gap between the two signals directly exposes non-target confounders.
  • Hypothesis-Driven Context Admission via Cohen's \(d\): Applying statistical effect size thresholds to context screening bridges empirical social science methodology with synthetic data curation, systematically rejecting uninformative or trivially over-optimized prompts.
  • Creative Open-Ended Enhancement alongside Strict Compliance: Unlike conventional DPO pipelines that constrain model creativity to boost instruction adherence, CAPS achieves a +15.5 point surge on Creative Writing v3 while concurrently improving IFEval and IFBench, proving that targeted criterion learning reinforces structured expressiveness.

Limitations & Future Work

  • High Upstream Inference Cost: The dual-signal verification pipeline involves multiple stochastic samples across large rollout pools. In the medical domain, synthesizing 2k pairs required over 5.37 million judge calls and ~\(1,600–\)1,900 in API costs.
  • Correlated Judge Model Biases: Both Signal A and Signal B rely on DeepSeek-V3.2 backbones. While empirical cross-judge checks using GPT-5.4-mini, Gemini-3.5-flash, and Claude-4.6 show strong rank stability (Top-30% Jaccard = 1.00), prompt isolation does not completely eliminate underlying statistical architecture biases.
  • Multi-Criterion Conflict Resolution: The current formulation optimizes single or unified composite criteria. Resolving real-world trade-offs where independent criteria inherently conflict (e.g., extreme conciseness vs. comprehensive safety disclaimers) remains an open challenge.
  • vs. rDPO / CLAIR: rDPO uses rubrics merely as soft weights on pre-collected pairs, while CLAIR relies on minimal-edit heuristics that achieve only 43% Jaccard similarity between intended and realized attributions. CAPS operates prospectively from the criterion seed through context synthesis and attribution filtering.
  • vs. PVar / Difficulty-Based Selection: PVar verifies prompt-level response variance without assessing criterion relevance; difficulty selection filters margin magnitude without identifying its causal source. CAPS's EAT+AR dual gate fills both voids simultaneously.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Inverts the standard post-hoc alignment pipeline by grounding preference data synthesis in explicit criteria with rigorous causal attribution.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across general, medical, and legal domains with comprehensive ablation, distillation controls, and judge robustness sweeps.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant mathematical formulations, and thorough diagnostic appendix.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, data-efficient blueprint for post-training alignment without expensive manual annotation.