โ๏ธ Alignment & RLHF¶
๐ฌ EMNLP2026 ยท 16 paper notes
๐ Same area in other venues: ๐ง NeurIPS2026 (43) ยท ๐๏ธ ECCV2026 (6) ยท ๐ท CVPR2026 (12) ยท ๐ฌ ICLR2026 (102) ยท ๐ฌ ACL2026 (38) ยท ๐งช ICML2026 (37)
๐ฅ Top topics: Alignment/RLHF ร6 ยท Personalized Generation ร2 ยท LLM ร2
- Aligning Cultural Behaviors in Arabic Language Models via Preferences
-
To address the lack of country-specific cultural behavioral alignment in Arabic and multilingual LLMs, this paper introduces ARABPREFโthe first large-scale pan-Arab preference dataset and native-validated evaluation benchmark covering all 22 Arab League nations across 16 categories and 200 aspectsโdemonstrating significant gains in cultural appropriateness and robustness via preference optimization.
- An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift
-
This paper conducts the first systematic investigation of five preference alignment objectives and various domain adaptation strategies under domain shift across summarization, QA helpfulness, and safety, proving that adaptation strategy dominates alignment objective and revealing that pseudo-labeling achieves high target win rates at the expense of severe linguistic mode collapse.
- ARCA: Attribution-Regularized Causal Alignment for Task-Aware Bias Mitigation
-
ARCA addresses the issue of social bias re-emergence during downstream fine-tuning by automatically extracting task-specific bias-propagating words via differential gradient attribution and jointly optimizing task classification with representation-level causal alignment on dynamically generated counterfactual pairs.
- Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMs
-
To tackle "personalization collapse"โwhere explicit style instructions override and suppress implicit user habitsโPsPLUG models personalization as a distributional residual via a lightweight plug-in prefix and style-conditioned preference optimization, achieving a controllable trade-off between style adherence and personalization strength under a frozen backbone.
- DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment
-
DSPA introduces a training-free inference-time preference alignment method that featurizes prompts via early-layer SAEs and routes interventions to late-layer SAE latents through a sparse conditional-difference map, editing only token-active features during decoding to achieve competitive alignment under extreme data scarcity without degrading general capability.
- Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM
-
This paper systematically demonstrates that expert persona prompts improve instruction-alignment tasks (safety and preference) while consistently degrading pretraining-dependent knowledge retrieval accuracy, and proposes PRISMโa fully bootstrapped, gated LoRA self-distillation framework that captures alignment benefits while preserving knowledge accuracy without external supervision.
- From Rollouts to Recipes: Self-Contained Post-Training for LLMs
-
Addressing the limitation that standard verifier-based post-training applies a single uniform optimization recipe to all samples regardless of learning state, this paper proposes Self-Routingโa self-contained framework that leverages rollout correctness and predictive entropy to dynamically assign samples to GRPO, on-policy self-distillation, regularization, or skipping, achieving consistent gains across mathematical reasoning benchmarks without external teachers or extra sampling overhead.
- GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training
-
Addressing the vulnerability of fixed mixing schedules in hybrid SFT-RL post-training to shifting gradient noise and objective conflict, GAC derives a closed-form mixing weight by minimizing the mean squared error (MSE) of the mixed stochastic gradient and pairs it with online lightweight proxies and a three-tier stabilization guardrail stack, achieving notable accuracy gains on math and reasoning benchmarks with less than 1% overhead.
- LQ-Rewriter: Generative Reward Model-Guided Lazy Query Rewriting for Role-Playing Dialogue
-
To tackle conversational stagnation caused by ambiguous and low-effort user inputs in role-playing interactions, this paper introduces LQ-Rewriter, a lightweight query rewriter guided by an online regeneration-derived Generative Reward Model (GRM), which leverages narrative-style rewriting strategies and two-stage alignment (RFT + CPO) to substantially enrich downstream role-play responses and boost online conversation turns by +11.1%.
- Mechanism Alignment for Socially Embedded Decision Making in LLMs
-
Addressing the failure of large language models to implement true peer-weighted utility aggregation in multi-agent social simulations, this paper introduces a network-utility benchmark and a theory-supervised mechanism alignment framework (SEDM-alignment) that significantly improves micro-level decision rule recovery and restores faithful macro emergent dynamics in financial and labor markets.
- P-Search: A Parallel PlanโActโVerify Framework with Decoupled GRPO for Deep Research
-
P-Search addresses confirmation bias and credit assignment dilution in open-web deep research by organizing evidence across a five-stage pipelineโgrounded background modeling, independent branch planning, parallel subagent execution, iterative verification, and report synthesisโtrained via star-shaped synthetic subgraphs and behavior-decoupled GRPO to achieve 59.2% on GAIA and 70.0% on Xbench-DeepSearch.
- PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization
-
To tackle signal exhaustion and lack of reasoning differentiation in outcome-reward (ORM) GRPO as well as catastrophic verbosity hacking in rubric-based process rewards (PRM), PAPO decouples advantage normalization by computing a global outcome advantage alongside a process advantage normalized strictly within the correct subset, sustaining non-zero gradients across all-correct groups while completely closing the incorrect-response hacking pathway.
- SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
-
SafeSteer addresses the alignment tax by constructing an activation-steered safety teacher to identify a sparse subset of safety-critical tokens, restricting on-policy reverse KL distillation solely to these tokens without re-normalization, thereby achieving state-of-the-art safety defense using merely 100 harmful prompts without any general-purpose data.
- Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
-
This paper introduces knowledge-aligned supervised fine-tuning (SFT) to constrain training targets strictly to the base model's parametric knowledge, proposing Recall Rewrite to probe consistent claim recall without external retrieval, which substantially suppresses factual hallucinations and enhances refusal calibration while maintaining general capabilities.
- The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment
-
Addressing the failure of existing outcome-supervised PRMs to anchor credit in intermediate step correctness, this paper casts the problem as Multiple Instance Learning under the Weakest Link principle, introducing Softmax-Weighted-Sum (SWS) pooling and stop-gradient optimization to jointly learn credit assignment and process rewards without step-level annotations or Monte Carlo rollouts.
- VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment
-
Addressing the unintended value drift caused by downstream knowledge adaptation, VISA proposes a modular shielded adaptation framework that decouples frozen base content generation from a lightweight value rewriter, leveraging GRPO to jointly optimize value steering precision and bidirectional semantic consistency without sacrificing factual integrity.