Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding¶

Conference: ACL 2025
arXiv: 2506.07434
Code: Yes (open-sourced code, datasets, and Pilot-3B model)
Area: Others
Keywords: preference alignment, low-resource, weak-to-strong decoding, speculative decoding, alignment tax

TL;DR¶

This work proposes the Weak-to-Strong Decoding (WSD) framework, which leverages a small aligned model to draft aligned response prefixes for a large base model to continue writing. This achieves low-resource preference alignment without introducing alignment tax.

Background & Motivation¶

Large language models (LLMs) need to be aligned with human preferences, but existing methods face two core challenges: first, the alignment tax induced by fine-tuning (i.e., performance degradation on downstream tasks like math and code), and second, the immense computational overhead of fine-tuning large models. Existing low-resource alignment approaches fall into two categories: one intervenes in the decoding process via external scoring (e.g., ARGS, CARDS) but compromises text coherence; the other indirectly influences token distribution through in-context learning (e.g., URIAL) but is less direct in guiding the current query.

Through pilot experiments, the authors discover a key phenomenon: the difficulty of generating aligned responses is concentrated at the beginning of decoding. Specifically, since base models model the entire text space, aligned responses are often not the highest ranked among all candidate paths. However, once the model begins generating along an aligned path, the difficulty of continuing with aligned content decreases significantly. This perfectly instantiates the adage: "Well begun is half done."

Method¶

Overall Architecture¶

The core idea of the WSD framework is to use a small aligned model (draft model) \(m\) to generate aligned response prefixes and then switch to a large base model \(M\) to complete the rest. The entire process resembles speculative decoding, but with a completely different motivation: speculative decoding aims for acceleration, whereas WSD aims for alignment.

Given a prompt \(x\), the final response is \(y = [y^m[:k]; M(x, y^m[:k])]\), where \(y^m[:k]\) represents the first \(k\) tokens generated by the small model, and \(M\) continues generation based on it.

Key Designs¶

Pilot Empirical Validation: By sampling 700+ prompts, the authors retain the first 100 tokens of aligned responses as aligned prefixes and compare them with 9 prefixes generated by the base LLM itself. The results show that the perplexity of the aligned prefixes is ranked moderately (indicating it is difficult for base models to generate them spontaneously), but continuing along the aligned prefixes yields high-reward responses. As the number of encoded tokens increases, the base model's perplexity drops significantly, demonstrating the guiding effect of the aligned prefix.
Model Switching Mechanism (Auto-Switch): The large model \(M\) encodes the draft \(y^m\) generated by the small model and computes the confidence \(P_M(y_i^m \mid y_{<i}^m, x)\) at each position. When this confidence first exceeds a threshold \(\gamma\), it switches to the large model for continuation. To enhance robustness, a sliding window (of size \(w\)) is used to compute the geometric mean of the probabilities, preventing unstable decisions caused by fluctuations in individual token probabilities.
Draft Model Training (Pilot-3B): The authors construct the GenerAlign dataset, focusing specifically on general human value alignment (harmlessness, helpfulness, honesty). Llama-3.2-3B-Instruct is fine-tuned using DPO to obtain Pilot-3B. While the fine-tuning cost of the small model is highly controllable, validation confirms that it does suffer from the alignment tax (improvement on AlpacaEval 2 but degradation on GSM8K and HumanEval).

Loss & Training¶

The draft model is trained using DPO loss for preference alignment.
The WSD inference phase involves no extra training and serves entirely as a collaborative decoding mechanism.
Hyperparameters: window size \(w=6\), threshold \(\gamma=0.8\), maximum draft length 512.

Key Experimental Results¶

Main Results¶

Method	HH-RLHF Total↑	TruthfulQA Overall↑	AlpacaEval 2 LC-WR↑	ArenaHard↑	MT-Bench↑
Llama-3-70B Base	60.35	48.71	2.45	3.50	5.25
URIAL	87.37	76.38	7.79	6.50	6.04
WSD	96.48	87.88	20.13	15.90	7.06
Llama-3.1-70B Base	58.08	45.90	2.48	4.70	6.14
WSD	97.06	85.43	23.65	16.20	7.57
Gemma-2-27B Base	47.06	33.41	3.33	5.40	6.34
WSD	96.77	85.68	23.32	18.40	7.31

Downstream Tasks (Alignment Tax Verification)¶

Model	Method	GSM8K 4-shot Acc↑	HumanEval Pass@1↑
Llama-3-70B	Base	82.18	54.27
Llama-3-70B	WSD	82.18	56.10
Gemma-2-27B	Base	82.56	62.80
Gemma-2-27B	WSD	85.52	65.85

Time Efficiency¶

Method	Llama-3-70B	Llama-3.1-70B	Gemma-2-27B
ARGS	\(2.25\times\)	\(2.11\times\)	\(2.82\times\)
CARDS	\(3.23\times\)	\(3.67\times\)	\(2.01\times\)
WSD	\(0.84\times\)	\(1.03\times\)	\(0.99\times\)

Key Findings¶

WSD outperforms all baselines by a large margin across almost all preference alignment benchmarks, including strong baselines like URIAL and Aligner.
WSD not only avoids generating alignment tax but also slightly improves performance on GSM8K and HumanEval (as it does not modify the base model parameters).
The decoding time of WSD is even lower than that of direct decoding (\(0.84\times\)) because the small model generates the prefix faster.
Ablation studies show that GSM8K is insensitive to hyperparameters, because model switching for downstream tasks is completed in the early stages of decoding.

Highlights & Insights¶

The empirical validation of "Well begun is half done" is highly intuitive and convincing: the guiding effect of the aligned prefix is clearly demonstrated through perplexity rankings and decay curves.
Zero alignment tax is an extremely appealing feature, as WSD does not modify the base model parameters but strictly collaborates during the decoding phase.
90% of model switching occurs during the structured response or question analysis stage, indicating that once the base model recognizes a "helpful style," it can confidently take over.
The small model only serves a guiding rather than a replacing role on downstream tasks, which shares a similar spirit with speculative decoding but with a different objective.

Limitations & Future Work¶

The draft model is only trained using DPO, leaving more data preparation and training strategies unexplored.
There remains substantial room for customizing the model switching criteria.
End-to-end deployment utilizing highly efficient inference frameworks like vLLM has not yet been implemented.
Speculative-decoding-style integration schemes can be explored, though they entail higher implementation complexity.

It shares the "small model drafts, large model verifies" structure with speculative decoding, but the motivations (acceleration vs. alignment) are completely different.
It belongs to a different paradigm compared to URIAL (in-context learning): WSD directly drafts aligned prefixes for the query, whereas URIAL indirectly influences distribution through in-context examples.
Insight: The key to preference alignment lies in "path selection" rather than "full control"; future work can explore more intelligent path-guiding mechanisms.

Rating¶

Novelty: ⭐⭐⭐⭐ The motivation is clear and novel; the insight of "well begun" is supported by solid experimental evidence.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive coverage of multiple models, multiple benchmarks, ablations, scalability, time efficiency, and case studies.
Writing Quality: ⭐⭐⭐⭐ The arguments flow smoothly, and the motivation section is exceptionally compelling.
Value: ⭐⭐⭐⭐ High practical value; there is a widespread demand for preference alignment in low-resource scenarios.