Skip to content

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ckarouzos/prefadap
Area: Large Language Model Alignment
Keywords: Preference Tuning, Domain Adaptation, Generalization Gap, Output Diversity, Mode Collapse

TL;DR

This paper conducts the first systematic investigation of five preference alignment objectives and various domain adaptation strategies under domain shift across summarization, QA helpfulness, and safety, proving that adaptation strategy dominates alignment objective and revealing that pseudo-labeling achieves high target win rates at the expense of severe linguistic mode collapse.

Background & Motivation

Post-training through preference optimization—spanning offline methods such as Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO), and Odds-Ratio Preference Optimization (ORPO), as well as online reinforcement learning like PPO and Group Relative Policy Optimization (GRPO)—has become the de facto standard for aligning large language models with human values. However, existing alignment recipes almost universally operate under the assumption of identically distributed data. In real-world deployments, language models invariably encounter distribution shifts across genres, discourse registers, and semantic boundaries, such as transitioning from informal Reddit social media threads to formal journalistic news, or shifting between disparate harm domains in safety guardrailing. Under such domain shifts, aligned models frequently suffer catastrophic out-of-domain degradation and behavioral misalignment.

The fundamental tension behind this breakdown lies in the objective of preference tuning: loss functions encourage policies to heavily exploit discriminative statistical signals specific to the source preference annotations. This leads to acute over-specialization to source stylistic artifacts and surface-level shortcuts, while obtaining dense, high-quality human preference annotations in each novel target domain is economically and logistically prohibitive. Prior studies on out-of-domain generalization have predominantly evaluated isolated objectives—such as analyzing DPO alone—leaving open the critical question of whether out-of-domain vulnerability stems from the alignment loss itself or from the absence of targeted domain adaptation techniques. Furthermore, the downstream toll that preference adaptation extracts on linguistic, semantic, and logical output diversity has remained unexamined.

To bridge this divide, this paper decomposes preference optimization under distribution shift along two orthogonal axes: the alignment objective and the domain adaptation strategy. The authors evaluate this framework across three complementary testbeds spanning distant summarization (Reddit TL;DR → CNN/DailyMail), near-domain QA helpfulness (AskEngineers → AskCulinary), and cross-category safety transfer (Cybercrime → Violence and Physical Harm). Core idea: by systematically evaluating five alignment objectives against four adaptation paradigms including target SFT and high-capacity teacher pseudo-labeling, the study establishes that adaptation strategy matters far more than the alignment loss, while demonstrating that synthetic pseudo-labeling resolves the generalization gap only by forcing severe structural mode collapse.

Method

Overall Architecture

The framework investigates domain adaptation for preference tuning when annotations in the target domain are unavailable. The input consists of a labeled source dataset \(\mathcal{D}_S\) (structured as pairwise comparisons \((x, y_w, y_l)\) or binary labels \((x, y, l)\)) and an unlabeled target corpus \(\mathcal{D}_T\) containing domain-specific prompts and responses without human preference feedback. The pipeline decomposes the transfer process into two sequential phases: the domain adaptation stage (spanning source SFT, mixed-domain SFT, target-domain SFT, and synthetic target pseudo-labeling) followed by the preference alignment stage (evaluating DPO, KTO, ORPO, PPO, and GRPO). The final policy is evaluated on held-out target splits for task performance, epistemic style appropriateness, and linguistic generation diversity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Preferences & Unlabeled Target Data"] --> B["Two-Axis Adaptation Decoupling<br/>Partition SFT initialization and alignment objectives"]
    B --> C["Target Pseudo-Labeling Formulation<br/>Pair 70B teacher candidates with raw references as hard negatives"]
    C --> D["Online Exploration & KL Anchoring<br/>GRPO group-relative advantages bound distribution drift"]
    D --> E["Domain-Aware & Diversity Auditing<br/>Syntactic/semantic/logical metrics and epistemic evaluation"]
    E --> F["Robust Generalization Policy"]

Key Designs

1. Two-Axis Adaptation Decoupling: isolated alignment objectives fail to resolve representation-level domain shift Directly deploying a policy optimized on source preferences to an unseen target domain conflates domain representation mismatch with preference learning failure. The framework explicitly decouples the transfer process into two distinct axes: the adaptation data strategy and the preference optimization objective. In the adaptation axis, policies are initialized across four data configurations: source-only baseline (\(\mathcal{D}_S \to \mathcal{D}_S\)), direct alignment omitting SFT, mixed SFT (\(\mathcal{D}_{S+T} \to \mathcal{D}_S\)), target-domain SFT (\(\mathcal{D}_T \to \mathcal{D}_S\)), and synthetic pseudo-labeling (\(\mathcal{D}_{synth}^T \to \mathcal{D}_{synth}^T\)). This separation isolates how much domain-specific language modeling priors contribute to cross-domain transfer independently of the preference loss.

2. Target Pseudo-Labeling Formulation: the scarcity of human target annotations cripples offline contrastive losses To supply explicit preference signals in target domains without expensive human labeling, the system distills target priors from a superior teacher model (Llama-3.3-70B-Instruct). For each target prompt \(x \in \mathcal{D}_T\), the teacher generates candidate completions \(y \sim \pi_{\text{teacher}}(\cdot|x)\). Crucially, the preference pairs are constructed asymmetrically: the teacher-generated response is designated as the winning completion \(y_w\), while the raw, noisy reference text from the original dataset is assigned as the losing completion \(y_l\). Empirical verification confirms that automated judges prefer teacher candidates over raw references in over 96.5% of cases, creating clean, high-margin preference pairs that effectively denoise the target preference space and format seamlessly across SFT, pairwise DPO/ORPO, and unpaired KTO.

3. Online Exploration & KL Anchoring: offline static objectives overfit fixed source contrastive pairs Offline objectives such as DPO optimize a static Bradley-Terry likelihood over pre-collected pairs, repeatedly reinforcing features that separate \(y_w\) from \(y_l\) in the source domain and leading to rigid memorization of source stylistic idiosyncrasies. Online reinforcement learning methods (PPO and GRPO) dynamically mitigate this by sampling rollouts directly from the current policy \(\pi_\theta\) and scoring them with a learned reward model. In particular, GRPO computes group-relative advantages across \(G\) sampled outputs without requiring a separate critic network: $\(A_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\}) + \eta}\)$ Because the rollout distribution shifts dynamically with policy exploration and updates are regularized by a per-token KL divergence penalty against the reference model \(\pi_{ref}\), GRPO prevents policy collapse into static source-domain feature modes while sustaining out-of-domain transfer.

4. Domain-Aware & Diversity Auditing: standard scalar benchmarks conceal epistemic drift and linguistic collapse Standard LLM-as-a-judge win rates fail to detect two major failure modes: epistemic persona mismatch (e.g., answering culinary questions with engineering thermodynamics jargon that judges score as helpful due to mere logical coherence) and structural mode collapse from synthetic distillation. To expose these vulnerabilities, the auditing suite incorporates a domain-aware judge that explicitly penalizes epistemic misalignment, alongside a three-tier diversity measurement protocol: - Syntactic diversity: Expectation-Adjusted Distinct (EAD) \(n\)-grams (\(n=1..5\)) with length-bias correction; - Semantic diversity: pairwise cosine distance over Sentence-BERT embeddings (\(1 - \text{avg\_cos\_sim}\)); - Logical diversity: Natural Language Inference (NLI) non-contradiction space \(1 - \mathbb{E}_{(a,b)}[p_{ent}(a,b) - p_{con}(a,b)]\).

Loss & Training

The framework benchmarks five distinct alignment objectives: - DPO: $\(\mathcal{L}_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)\right]\)$ - KTO: optimizes binary rewards based on Kahneman-Tversky prospect theory with asymmetric loss scaling \(\lambda_D, \lambda_U\); - ORPO: couples negative log-likelihood on chosen outputs with an odds-ratio contrastive term; - PPO & GRPO: trained with LoRA (\(r=16, \alpha=256\)) across all projection modules using AdamW, learning rates between \(8\times 10^{-7}\) and \(1\times 10^{-5}\), and KL regularization weight \(\beta=0.10\).

Key Experimental Results

Main Results

The primary evaluation across Llama-3.1-8B and OLMo-3-7B on the three benchmarks is summarized below (all figures represent win rates or safety percentages; \(\text{Gap} = \text{Src} - \text{Tgt}\)):

Method SFT Data Pref. Data Summ. Src (TL;DR) Summ. Tgt (CNN) Summ. Gap QA Src (Eng) QA Tgt (Cul) QA Gap Safety Src (Cyber) Safety Tgt (Viol) Safety Gap
Base – – 44.97 15.96 29.01 54.59 61.37 -6.78 38.40 32.90 5.50
SFT \(\mathcal{D}_S\) – 59.57 36.07 23.50 60.74 60.08 0.66 43.40 39.20 4.20
SFT \(\mathcal{D}_{S+T}\) – 61.56 57.31 4.25 59.14 60.68 -1.54 42.80 40.20 2.60
SFT \(\mathcal{D}_{synth}^T\) – 95.70 83.37 12.33 72.79 76.04 -3.25 99.60 95.90 3.70
DPO \(\mathcal{D}_S\) \(\mathcal{D}_S\) 89.87 58.09 31.78 60.73 61.45 -0.72 44.00 42.00 2.00
DPO – \(\mathcal{D}_S\) 85.17 38.29 46.88 64.01 61.96 2.05 41.90 41.90 0.00
DPO \(\mathcal{D}_{S+T}\) \(\mathcal{D}_S\) 87.72 68.50 19.22 59.40 58.80 0.60 42.20 43.10 -0.90
DPO \(\mathcal{D}_T\) \(\mathcal{D}_S\) 67.83 56.82 11.01 59.29 64.69 -5.40 44.40 43.50 0.90
DPO \(\mathcal{D}_{synth}^T\) \(\mathcal{D}_{synth}^T\) 95.79 78.50 17.29 72.76 75.52 -2.76 99.80 96.40 3.40
KTO \(\mathcal{D}_S\) \(\mathcal{D}_S\) 81.06 51.10 29.96 61.03 58.95 2.08 33.30 35.50 -2.20
KTO \(\mathcal{D}_{synth}^T\) \(\mathcal{D}_{synth}^T\) 95.37 83.01 12.36 72.39 75.38 -2.99 99.80 95.50 4.30
ORPO \(\mathcal{D}_S\) \(\mathcal{D}_S\) 60.96 35.30 25.66 54.14 58.34 -4.20 54.80 47.20 7.60
ORPO \(\mathcal{D}_{synth}^T\) \(\mathcal{D}_{synth}^T\) 96.80 82.38 14.42 72.75 76.15 -3.40 99.60 95.50 4.10
PPO \(\mathcal{D}_S\) \(\mathcal{D}_S\) 44.30 59.69 -15.39 55.10 58.05 -2.95 40.19 46.24 -6.05
GRPO \(\mathcal{D}_S\) \(\mathcal{D}_S\) 62.57 58.78 3.79 54.89 62.00 -7.11 43.90 39.50 4.40
GRPO \(\mathcal{D}_{synth}^T\) \(\mathcal{D}_{synth}^T\) 87.16 80.19 6.97 64.63 62.39 2.24 99.80 95.60 4.20

Ablation Study & Detailed Analyses

The following table provides the quantitative diversity evaluation on Llama-3.1-8B under the summarization transfer benchmark (TL;DR → CNN/DM), evidencing the mode collapse triggered by synthetic pseudo-labeling:

Adaptation Setting Metric SFT PPO GRPO DPO KTO ORPO Analysis & Implication
Source-Only Syntactic (EAD) 0.86 0.63 0.68 0.75 0.76 0.76 Preserves broad vocabulary and phrase variation
Source-Only Semantic (SBERT) 0.46 0.21 0.35 0.24 0.41 0.32 SFT retains the broadest semantic space
Source-Only Logical (NLI) 1.08 1.07 1.08 0.95 1.07 1.01 Healthy inferential reasoning variation
SFT-Mix Syntactic (EAD) 0.87 0.76 0.76 0.72 0.77 0.77 Balances robust generalization with high lexical variety
SFT-Mix Semantic (SBERT) 0.31 0.25 0.30 0.21 0.31 0.31 Moderate semantic clustering
SFT-Mix Logical (NLI) 1.10 1.07 1.09 1.05 1.10 1.10 Sound inferential flexibility
Pseudo-label Syntactic (EAD) 0.51 0.62 0.62 0.52 0.51 0.53 Heavy repetition of teacher-distilled framing markers
Pseudo-label Semantic (SBERT) 0.07 0.22 0.25 0.07 0.07 0.08 Near-zero semantic variance across offline methods
Pseudo-label Logical (NLI) 0.88 0.93 0.95 0.89 0.89 0.89 Extreme mode narrowing; answers collapse to uniform reasoning

To isolate teacher quality from domain relevance and assess sample efficiency, the authors conducted controlled ablations on Llama-3.1-8B:

Investigation Data Scale Configuration Source Win Rate (%) Target Win Rate (%) Generalization Gap Takeaway
Teacher Isolation Full (100%) SFT \(\mathcal{D}_S\) 59.57 36.07 23.50 Human ground truth baseline
Teacher Isolation Full (100%) SFT \(\mathcal{D}_{synth}^S\) 72.41 44.15 28.26 Pseudo-labeling source domain fails to close the gap
Teacher Isolation Full (100%) SFT \(\mathcal{D}_{synth}^T\) 95.70 83.37 12.33 Target relevance drives +39.22 target win rate boost
Teacher Isolation Full (100%) DPO \(\mathcal{D}_{synth}^S\) 90.52 62.88 27.64 High teacher capacity cannot substitute for domain relevance
Teacher Isolation Full (100%) DPO \(\mathcal{D}_{synth}^T\) 95.79 78.50 17.29 Target synthetic pairs outperform source synthetic by +15.62
Data Efficiency Small (10%) SFT \(\mathcal{D}_{synth}^T\) 92.75 83.68 9.07 10% target pseudo-data matches full-data target win rate
Data Efficiency Small (10%) DPO \(\mathcal{D}_{synth}^T\) 96.30 77.08 19.22 Minor 1.42 drop, demonstrating high information density
Data Efficiency Small (10%) KTO \(\mathcal{D}_{synth}^T\) 95.30 84.38 10.92 10% subset slightly outperforms 100% full dataset (83.01)

Key Findings

  • Adaptation strategy dominates the alignment objective: Across all testbeds, switching between adaptation paradigms (e.g., SFT-Mix or pseudo-labeling) produces far greater performance shifts than changing the alignment loss between DPO, KTO, ORPO, PPO, or GRPO. On summarization, target pseudo-labeling elevates target win rate from 15.96% to 83.37%.
  • Offline alignment exhibits severe in-domain over-specialization: Without target adaptation, DPO achieves stellar in-domain scores (89.87% on Llama, 91.00% on OLMo) but collapses out-of-domain, producing massive generalization gaps exceeding 30% to 46%. Static contrastive losses heavily overfit surface distribution cues.
  • Online RL (GRPO) maintains balanced cross-domain transfer: GRPO limits the generalization gap to 3.79% on summarization while retaining an in-domain score 18.27 points higher than PPO. Online rollout exploration anchored by KL regularizers effectively checks mode collapse.
  • Pseudo-labeling induces severe structural mode collapse: While pseudo-labeled models achieve top win rates, their semantic diversity crashes from 0.46 to 0.07. Qualitative auditing reveals that students rigidly adopt the 70B teacher's template: regardless of topic (food, tourism, or sports), ~10% of summaries mechanically reproduce "The article discusses ... The author argues/advises ... The article also recommends ...", compared to only 0.2% in reference or source-trained summaries.
  • Near-domain QA conceals epistemic mismatch: The near-zero QA generalization gaps (-6.78% to 3.33%) under standard judges are deceptive. Domain-aware evaluation shows that source models answer cooking queries using rigorous engineering heat/fat thermodynamics, which standard judges reward for structural coherence despite the severe pragmatic persona failure.

Highlights & Insights

  • Debunking the synthetic feedback panacea: The study empirically demonstrates that synthetic pseudo-labeling is a double-edged sword; it maximizes scalar win rates by compressing the policy's predictive distribution into the teacher's narrow single mode, exposing a major flaw in win-rate-only benchmarks.
  • Remarkable data efficiency of synthetic preferences: Utilizing just 10% of the target pseudo-labeled dataset is sufficient to achieve or exceed full-dataset target performance, providing an actionable, compute-efficient recipe for industrial domain adaptation.
  • Asymmetric temporal dynamics of SFT sequencing: Staging target SFT before source SFT (\(\mathcal{D}_T \to \mathcal{D}_S\)) yields a 56.40% target win rate, whereas reversing the order (\(\mathcal{D}_S \to \mathcal{D}_T\)) causes target performance to collapse to 35.22%, proving that grounding target linguistic priors early is vital for stable downstream alignment.

Limitations & Future Work

  • Model scale constraints: Experiments were conducted on 7B–8B parameter open-weight models; whether larger frontier models exhibit stronger intrinsic resistance to preference over-specialization remains an open question.
  • Monolingual scope: All evaluations are restricted to English textual benchmarks, leaving multimodal, cross-lingual, and complex multi-step reasoning (code/math) domains for future exploration.
  • Amplification of teacher biases: Synthetic pseudo-labeling inherently inherits the blind spots, latent biases, and hallucinations of the teacher model, underscoring the need for multi-teacher ensembling and self-play regularization.
  • vs DPO (Rafailov et al., 2023) & Kirk et al. (2024): While prior work evaluated DPO in isolation or analyzed in-domain RLHF properties, this work proves that offline contrastive alignment is exceptionally brittle under domain shift unless paired with explicit target adaptation.
  • vs RLAIF (Bai et al., 2022) & Model Collapse Theory (Shumailov et al., 2024): Expanding beyond recursive pretraining collapse, this paper provides the first systematic evidence of linguistic and structural mode collapse triggered by AI feedback distillation during preference post-training.

Rating

  • Novelty: ⭐⭐⭐⭐☆ Systematic two-axis decomposition offering the first rigorous characterization of preference tuning under domain shift
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage across 2 base models, 5 alignment objectives, 4 adaptation strategies, and 3 testbeds with multi-dimensional diversity metrics
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear conceptual structure, meticulous experimental reporting, and seamless integration of quantitative data with qualitative case studies
  • Value: ⭐⭐⭐⭐⭐ Provides indispensable practical guidance and vital caveats for domain adaptation and safety alignment in LLM post-training