When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/TokisakiKurumi2001/w2s_ood
Area: LLM Reasoning
Keywords: Weak-to-Strong Generalization, Reward Modeling, Preference Shift, Scalable Oversight, Representation Anchoring
TL;DR¶
This paper exposes that weak-to-strong reward models exhibit fragile out-of-distribution preference transfer despite high in-distribution gains, establishes a zero-shot domain-shift evaluation protocol with transfer-aware metrics, and proposes Representation Anchoring (ANCHOR) to regularize hidden representations toward a frozen pretrained reference.
Background & Motivation¶
As frontier large language models grow increasingly capable, providing dependable supervision becomes a fundamental roadblock when human annotators cannot reliably assess complex model outputs due to bounded expertise, excessive cost, or prohibitive scale. This predicament has catalyzed extensive investigation into scalable oversight, which studies how weaker supervisors can guide and align stronger learning systems without direct expert annotations. Weak-to-strong (W2S) generalization provides a concrete formal abstraction of this paradigm: a stronger student model is trained on imperfect labels elicited from a weaker teacher, aiming to elicit latent capabilities that surpass the teacher's own competence ceiling. Promising results across reasoning, mathematical problem solving, and instruction tuning suggest that strong models can effectively uncover useful structural signals from noisy supervision.
However, existing evaluations of weak-to-strong alignment predominantly test students under matched or near-matched train-test distributions. In realistic deployment pipelines, scalar reward models—the critical evaluation backbone for reinforcement learning from human feedback (RLHF)—must generalize across shifting prompt formats, diverse annotator standards, and distinct preference distributions. Current W2S reward models that exhibit high in-distribution accuracy on a held-out split of the source training domain frequently overestimate alignment reliability. When a student model fits source-domain dataset artifacts and narrow labeling idiosyncrasies, fine-tuning distorts broadly useful pretrained representations rather than internalizing transferable preference criteria.
The core tension underlying this degradation is that weak-supervised fine-tuning steers the strong student's feature space excessively toward source-specific representations (representation drift), precipitating catastrophic failure on unseen preference domains. To counter this failure mode, this work formalizes a zero-shot preference-domain shift evaluation protocol and introduces a simple yet principled regularizer. Core idea: regularize response-token hidden states of the trainable student toward a frozen pretrained reference model via Representation Anchoring (ANCHOR), enforcing controlled representation drift during weak preference learning to secure both in-distribution performance and out-of-distribution transfer.
Method¶
Overall Architecture¶
The framework establishes a rigorous zero-shot preference-domain shift benchmark and implements the ANCHOR training method for weak-to-strong reward modeling. Within a broad preference category \(\mathcal{C}\) (e.g., Helpfulness or Harmlessness), multiple distinct preference domains \(\mathcal{K}_{\mathcal{C}} = \{k_1, \dots, k_M\}\) are defined, each characterized by its own prompt distribution, response generation style, and annotation protocol. Given a source domain \(S\), a weak supervisor \(w_S\) is trained on a disjoint gold subset and produces continuous soft preference targets for the student training split. A strong student reward model \(m_S\) is trained using these soft weak labels. Simultaneously, a frozen copy of the pretrained strong model \(m_{\text{ref}}\) serves as a geometric anchor. The optimization objective jointly minimizes the weak preference loss and a response-token representation anchoring penalty. At inference time, the reference branch is completely discarded, and the learned scalar reward model is evaluated directly on both held-out source data and unseen target domains.
The complete training and transfer evaluation pipeline flows as follows:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Source Preference Pair (x, y_a, y_b)"] --> B["Weak Teacher Soft-Label Supervision"]
B --> C["Weak-to-Strong Preference Loss"]
A --> D["Response Masking & Hidden-State Extraction"]
D --> E["Representation Anchoring (ANCHOR)"]
C --> F["Joint Optimization Objective"]
E --> F
F --> G["Transfer-Aware Metric Evaluation (WRG / AOG / NTS)"]
Key Designs¶
1. Weak Teacher Soft-Label Supervision: Mitigating hard label noise amplification while preserving preference uncertainty
Hard binary labels discard the confidence distribution of the weak teacher, forcing the strong student to overfit teacher classification errors. To address this, the source dataset is partitioned into disjoint supervisor and student splits. The weak supervisor \(w_S\), trained on gold labels, scores the candidate pair \((x_i, y^a_i, y^b_i)\) to output a continuous soft preference probability: $\(q_i = \sigma\left(w_S(x_i, y^a_i) - w_S(x_i, y^b_i)\right)\)$ where \(\sigma\) denotes the sigmoid function. The strong student predicts preference probability \(p_i = \sigma(m_S(x_i, y^a_i) - m_S(x_i, y^b_i))\). Soft labeling effectively transfers calibrated uncertainty, preventing the student from memorizing hard label noise and providing gradient flexibility for the strong model to correct systematic teacher weaknesses.
2. Weak-to-Strong Preference Loss: Aligning soft preference distributions within an independent scalar reward formulation
To supervise the student while maintaining standard Bradley-Terry scalar reward properties, the method utilizes the expected negative log-likelihood under the teacher's Bernoulli distribution: $\(\mathcal{L}_{\text{w2s}} = -\frac{1}{N} \sum_{i=1}^N \left[ q_i \log p_i + (1 - q_i) \log (1 - p_i) \right]\)$ This formulation preserves scalar reward scoring, wherein each prompt-response pair receives an independent scalar score \(r_\theta(x, y)\). Unlike joint rankers that require paired evaluation, this architecture remains fully compatible with arbitrary downstream Best-of-\(N\) sampling and reinforcement learning policy gradients.
3. Response Masking & Hidden-State Extraction: Filtering prompt-level artifacts to isolate output response representations
Computing representation distance over full sequences causes the objective to be dominated by dataset-specific prompt lengths and context artifacts rather than candidate response quality. The framework constructs a binary response token mask \(m(x, y) \in \{0, 1\}^T\) that zeroes out prompt and padding tokens. At a designated transformer layer \(\ell\), hidden representations \(H_{\theta}^\ell(x, y)\) from the student and \(H_{\text{ref}}^\ell(x, y)\) from the frozen pretrained reference are extracted and normalized across the valid response sequence length.
4. Representation Anchoring (ANCHOR): Constraining representation drift to retain generalizable pretrained feature manifolds
To halt the erosion of general pretrained preference priors during weak fine-tuning, ANCHOR penalizes deviation between student and reference hidden states across candidate responses: $\(\mathcal{D}_{\text{anchor}}^\ell(x, y) = \frac{\sum_{t=1}^T m_t(x, y) \| H_{\theta, t}^\ell(x, y) - H_{\text{ref}, t}^\ell(x, y) \|_2^2}{d \sum_{t=1}^T m_t(x, y)}\)$ The total anchoring penalty is the sum over both candidate responses: \(\mathcal{L}_{\text{anchor}}^\ell(i) = \mathcal{D}_{\text{anchor}}^\ell(x_i, y^a_i) + \mathcal{D}_{\text{anchor}}^\ell(x_i, y^b_i)\). The primary implementation uses Last-Layer ANCHOR (\(\ell = L\)), directly anchoring the representations feeding into the scalar scoring head with minimal memory overhead, while Middle-Layer ANCHOR provides intermediate-layer regularization. This mechanism enforces controlled representation drift, allowing the student to adapt to weak supervisory signals while anchoring its latent geometry to transferable pretrained features.
5. Transfer-Aware Metric Evaluation: Decoupling out-of-distribution gains from in-distribution regression costs
Evaluating raw accuracy alone fails to penalize methods that achieve transfer gains by collapsing source-domain performance. The evaluation protocol reports three complementary transfer-aware metrics: Weak-to-Strong Raw Gain (WRG) measuring source-domain improvement over the weak teacher; Absolute OOD Gain (AOG) quantifying accuracy advantage on unseen target domain \(T\); and Net Transfer Score (NTS): $\(\text{NTS}(S, T; m) = \text{AOG}(S, T; m) - \max\left(0, \text{Acc}(w_S, S_{\text{test}}) - \text{Acc}(m_S, S_{\text{test}})\right)\)$ When source performance drops below the weak supervisor, the regression penalty reduces NTS accordingly, ensuring that reported OOD improvements do not come at the expense of source-domain reliability.
Loss & Training¶
The combined training objective for each sample is: $\(\mathcal{L} = \mathcal{L}_{\text{w2s}} + \lambda \mathcal{L}_{\text{anchor}}\)$ where \(\lambda\) governs the anchoring regularizer strength and is fixed at \(\lambda = 10^{-4}\) across all domains and model scales. All models are trained using LoRA (rank 16, learning rate \(2 \times 10^{-4}\), 3 epochs). Because the reference model \(m_{\text{ref}}\) is completely frozen and requires no backward pass, peak GPU VRAM remains virtually identical (38.66 GB vs. 38.71 GB for Naive W2S on A6000 GPUs), adding only ~14% training time and zero inference overhead.
Key Experimental Results¶
Main Results¶
Experiments span two major categories: Helpful (HelpSteer3, Anthropic Helpful, UltraFeedback) and Harmless (Anthropic Harmless, PKU-SafeRLHF, RAIL), evaluated on Llama (1B \(\to\) 8B) and Qwen (1.7B \(\to\) 8B) model families. The table below presents in-distribution and zero-shot out-of-distribution transfer metrics:
| Model Family / Source Domain | Method | In-Dist. WRG (%) | Target 1 AOG (%) | Target 1 NTS (%) | Target 2 AOG (%) | Target 2 NTS (%) |
|---|---|---|---|---|---|---|
| Llama (Src: HelpSteer3) | Naive W2S | 1.71 | 1.46 (AH) | 1.46 | 4.80 (UF) | 4.80 |
| Llama (Src: HelpSteer3) | Conf.-based | -2.12 | 8.06 (AH) | 5.94 | 7.64 (UF) | 5.52 |
| Llama (Src: HelpSteer3) | SEAM | -15.67 | 9.95 (AH) | -5.72 | 6.25 (UF) | -9.42 |
| Llama (Src: HelpSteer3) | ANCHOR | 3.57 | 8.19 (AH) | 8.19 | 7.93 (UF) | 7.93 |
| Qwen (Src: Anthropic Helpful) | Naive W2S | 0.45 | 5.57 (H3) | 5.57 | 5.15 (UF) | 5.15 |
| Qwen (Src: Anthropic Helpful) | Conf.-based | 1.72 | 6.78 (H3) | 6.78 | 4.92 (UF) | 4.92 |
| Qwen (Src: Anthropic Helpful) | SEAM | -3.42 | 0.55 (H3) | -2.87 | 5.49 (UF) | 2.07 |
| Qwen (Src: Anthropic Helpful) | ANCHOR | 2.75 | 8.96 (H3) | 8.96 | 5.73 (UF) | 5.73 |
| Llama (Src: PKU-SafeRLHF) | Naive W2S | 4.91 | 2.65 (AHar) | 2.65 | 2.26 (RAIL) | 2.26 |
| Llama (Src: PKU-SafeRLHF) | Conf.-based | 5.08 | 2.61 (AHar) | 2.61 | 2.47 (RAIL) | 2.47 |
| Llama (Src: PKU-SafeRLHF) | SEAM | 7.94 | 1.64 (AHar) | 1.64 | 6.27 (RAIL) | 6.27 |
| Llama (Src: PKU-SafeRLHF) | ANCHOR | 5.49 | 3.43 (AHar) | 3.43 | 4.62 (RAIL) | 4.62 |
Ablation Study¶
On the Llama/HelpSteer3 benchmark, ANCHOR was ablated against anchoring coefficients \(\lambda\), layer selection (Last-Layer vs. Middle-Layer), and alternative optimization/preservation controls:
| Regularization / Control Configuration | In-Dist. WRG (%) | Anthropic Helpful AOG (%) | Anthropic Helpful NTS (%) | UltraFeedback AOG (%) | UltraFeedback NTS (%) |
|---|---|---|---|---|---|
| Naive W2S (No regularizer) | 1.71 | 1.46 | 1.46 | 4.80 | 4.80 |
| Weight Interpolation (\(\alpha = 0.75\)) | 1.38 | 4.12 | 4.12 | 5.32 | 5.32 |
| Early Stopping (Validation-based) | 2.47 | 3.80 | 3.80 | 6.26 | 6.26 |
| LoRA (\(r = 8\)) | 1.49 | 2.86 | 2.86 | 6.14 | 6.14 |
| L2-SP (\(\lambda = 10^{-4}\)) | 2.69 | 4.16 | 4.16 | 9.03 | 9.03 |
| ANCHOR (Middle-layer \(\lfloor 0.5L \rfloor, \lfloor 0.75L \rfloor\)) | 2.14 | 4.42 | 4.42 | 8.22 | 8.22 |
| ANCHOR (Last-layer, \(\lambda = 10^{-4}\)) | 3.57 | 8.19 | 8.19 | 7.93 | 7.93 |
| ANCHOR (Last-layer, \(\lambda = 10^{-3}\)) | 2.69 | 7.10 | 7.10 | 6.69 | 6.69 |
| ANCHOR (Last-layer, \(\lambda = 10^{-2}\)) | 2.58 | 5.25 | 5.25 | 6.33 | 6.33 |
Key Findings¶
- In-distribution gains mask out-of-distribution failure: Naive W2S and confidence filtering yield high source-domain accuracy but frequently produce near-zero or negative transfer gains across domains. SEAM occasionally achieves high AOG but triggers severe source-domain collapse (WRG drops to -15.67% on Llama/H3, turning NTS heavily negative).
- ANCHOR achieves superior ID-OOD balance: Across 60 evaluation comparisons in Table 1, ANCHOR achieves the best or tied-best performance in 41 cases. Strong-ceiling decomposition shows that the extra OOD gap attributable specifically to weak supervision is reduced to just 0.23 points for ANCHOR (vs. 1.05 to 2.83 for baselines).
- Parameter constraints cannot replace representation anchoring: Early stopping and low-rank LoRA provide only marginal gains. While L2-SP parameter regularization improves UltraFeedback transfer, it severely degrades Anthropic Helpful transfer (4.16 vs. 8.19 AOG) and in-distribution WRG, proving that directly anchoring the hidden geometric manifold is far more effective than restricting parameter displacements.
- Controlled drift hypothesis substantiated: CKA/CCA geometric distance and linear-probe analyses demonstrate that excessive anchoring (\(\lambda = 10^{-2}\)) suffocates task adaptation, whereas moderate anchoring (\(\lambda = 10^{-4}\)) permits task-relevant adaptation while preserving linearly accessible preference features.
Highlights & Insights¶
- Crucial paradigm shift in W2S evaluation: Exposes the critical blind spot of testing weak-to-strong alignment solely on in-distribution splits, establishing zero-shot preference-domain shift as a necessary benchmark for scalable oversight.
- Minimalist yet highly effective mechanism: ANCHOR requires only an MSE penalty on response-token representations against a frozen reference model, requiring zero inference overhead and negligible training compute (+14% time, ~0% peak VRAM).
- Geometric representation preservation vs. parameter preservation: Proves that constraining internal representation geometry directly prevents alignment overfitting far more reliably than naive optimization damping or parameter-space penalties like L2-SP.
Limitations & Future Work¶
- Evaluation restricted to offline pairwise accuracy: The current study focuses on pairwise classification accuracy and offline Best-of-4 reranking; validating the stability of anchored reward models inside closed-loop online RLHF (PPO) and open-ended policy optimization remains to be explored.
- Confounded distribution shift factors: Cross-dataset shifts simultaneously vary prompt lengths, annotator subjective values, response styles, and model sources; the isolated contribution of each factor has not been fully disentangled.
- Model scale scope: Validated primarily on 1B to 8B parameter models; investigating whether 70B+ scale models naturally resist representation drift under weak supervision remains an open question.
Related Work & Insights¶
- vs Naive W2S (Burns et al., 2023): Pioneer of the weak-to-strong framework, but restricted to in-distribution evaluation and direct soft-label fitting; this paper identifies its out-of-distribution fragility under preference shifts and introduces representation anchoring to resolve it.
- vs SEAM (Li et al., 2025): SEAM uses multi-model rationale generation and weak-model selection, which frequently suffers catastrophic in-distribution performance degradation; ANCHOR maintains stable single-student reward modeling with preserved source performance.
- vs Continual Learning Regularization (Kirkpatrick et al., 2017; Lee et al., 2020): Traditional continual learning penalizes parameter shifts to prevent forgetting past tasks; ANCHOR demonstrates that for zero-shot transfer in LLMs, preserving the geometry of hidden activation manifolds is substantially more effective than rigid parameter constraints.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Exposes a fundamental vulnerability in W2S reward modeling and proposes a clean representation anchoring solution]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-model, cross-dataset evaluations with CKA/CCA geometry diagnostics, ceiling gap analysis, and optimization controls]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity, rigorous metric definitions, cohesive prose narrative, and transparent reporting]
- Value: ⭐⭐⭐⭐⭐ [Provides crucial guidance and a practical recipe for building robust reward models under scalable oversight]