Skip to content

โš–๏ธ Alignment & RLHF

๐Ÿ’ฌ EMNLP2026 ยท 16 paper notes

๐Ÿ“Œ Same area in other venues: ๐Ÿง  NeurIPS2026 (43) ยท ๐ŸŽž๏ธ ECCV2026 (6) ยท ๐Ÿ“ท CVPR2026 (12) ยท ๐Ÿ”ฌ ICLR2026 (102) ยท ๐Ÿ’ฌ ACL2026 (38) ยท ๐Ÿงช ICML2026 (37)

๐Ÿ”ฅ Top topics: Alignment/RLHF ร—6 ยท Personalized Generation ร—2 ยท LLM ร—2

Aligning Cultural Behaviors in Arabic Language Models via Preferences

To address the lack of country-specific cultural behavioral alignment in Arabic and multilingual LLMs, this paper introduces ARABPREFโ€”the first large-scale pan-Arab preference dataset and native-validated evaluation benchmark covering all 22 Arab League nations across 16 categories and 200 aspectsโ€”demonstrating significant gains in cultural appropriateness and robustness via preference optimization.

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

This paper conducts the first systematic investigation of five preference alignment objectives and various domain adaptation strategies under domain shift across summarization, QA helpfulness, and safety, proving that adaptation strategy dominates alignment objective and revealing that pseudo-labeling achieves high target win rates at the expense of severe linguistic mode collapse.

ARCA: Attribution-Regularized Causal Alignment for Task-Aware Bias Mitigation

ARCA addresses the issue of social bias re-emergence during downstream fine-tuning by automatically extracting task-specific bias-propagating words via differential gradient attribution and jointly optimizing task classification with representation-level causal alignment on dynamically generated counterfactual pairs.

Do Implicit Personalization and Explicit Styles Conflict? PsPLUG: A Lightweight Plug-in for Balancing Personalization and Style in Customized LLMs

To tackle "personalization collapse"โ€”where explicit style instructions override and suppress implicit user habitsโ€”PsPLUG models personalization as a distributional residual via a lightweight plug-in prefix and style-conditioned preference optimization, achieving a controllable trade-off between style adherence and personalization strength under a frozen backbone.

DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment

DSPA introduces a training-free inference-time preference alignment method that featurizes prompts via early-layer SAEs and routes interventions to late-layer SAE latents through a sparse conditional-difference map, editing only token-active features during decoding to achieve competitive alignment under extreme data scarcity without degrading general capability.

Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM

This paper systematically demonstrates that expert persona prompts improve instruction-alignment tasks (safety and preference) while consistently degrading pretraining-dependent knowledge retrieval accuracy, and proposes PRISMโ€”a fully bootstrapped, gated LoRA self-distillation framework that captures alignment benefits while preserving knowledge accuracy without external supervision.

From Rollouts to Recipes: Self-Contained Post-Training for LLMs

Addressing the limitation that standard verifier-based post-training applies a single uniform optimization recipe to all samples regardless of learning state, this paper proposes Self-Routingโ€”a self-contained framework that leverages rollout correctness and predictive entropy to dynamically assign samples to GRPO, on-policy self-distillation, regularization, or skipping, achieving consistent gains across mathematical reasoning benchmarks without external teachers or extra sampling overhead.

GAC: Noise-Aware Adaptive Mixing for Hybrid SFT-RL Post-Training

Addressing the vulnerability of fixed mixing schedules in hybrid SFT-RL post-training to shifting gradient noise and objective conflict, GAC derives a closed-form mixing weight by minimizing the mean squared error (MSE) of the mixed stochastic gradient and pairs it with online lightweight proxies and a three-tier stabilization guardrail stack, achieving notable accuracy gains on math and reasoning benchmarks with less than 1% overhead.

LQ-Rewriter: Generative Reward Model-Guided Lazy Query Rewriting for Role-Playing Dialogue

To tackle conversational stagnation caused by ambiguous and low-effort user inputs in role-playing interactions, this paper introduces LQ-Rewriter, a lightweight query rewriter guided by an online regeneration-derived Generative Reward Model (GRM), which leverages narrative-style rewriting strategies and two-stage alignment (RFT + CPO) to substantially enrich downstream role-play responses and boost online conversation turns by +11.1%.

Mechanism Alignment for Socially Embedded Decision Making in LLMs

Addressing the failure of large language models to implement true peer-weighted utility aggregation in multi-agent social simulations, this paper introduces a network-utility benchmark and a theory-supervised mechanism alignment framework (SEDM-alignment) that significantly improves micro-level decision rule recovery and restores faithful macro emergent dynamics in financial and labor markets.

P-Search: A Parallel Planโ€“Actโ€“Verify Framework with Decoupled GRPO for Deep Research

P-Search addresses confirmation bias and credit assignment dilution in open-web deep research by organizing evidence across a five-stage pipelineโ€”grounded background modeling, independent branch planning, parallel subagent execution, iterative verification, and report synthesisโ€”trained via star-shaped synthetic subgraphs and behavior-decoupled GRPO to achieve 59.2% on GAIA and 70.0% on Xbench-DeepSearch.

PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalization

To tackle signal exhaustion and lack of reasoning differentiation in outcome-reward (ORM) GRPO as well as catastrophic verbosity hacking in rubric-based process rewards (PRM), PAPO decouples advantage normalization by computing a global outcome advantage alongside a process advantage normalized strictly within the correct subset, sustaining non-zero gradients across all-correct groups while completely closing the incorrect-response hacking pathway.

SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

SafeSteer addresses the alignment tax by constructing an activation-steered safety teacher to identify a sparse subset of safety-critical tokens, restricting on-policy reverse KL distillation solely to these tokens without re-normalization, thereby achieving state-of-the-art safety defense using merely 100 harmful prompts without any general-purpose data.

Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning

This paper introduces knowledge-aligned supervised fine-tuning (SFT) to constrain training targets strictly to the base model's parametric knowledge, proposing Recall Rewrite to probe consistent claim recall without external retrieval, which substantially suppresses factual hallucinations and enhances refusal calibration while maintaining general capabilities.

The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

Addressing the failure of existing outcome-supervised PRMs to anchor credit in intermediate step correctness, this paper casts the problem as Multiple Instance Learning under the Weakest Link principle, introducing Softmax-Weighted-Sum (SWS) pooling and stop-gradient optimization to jointly learn credit assignment and process rewards without step-level annotations or Monte Carlo rollouts.

VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment

Addressing the unintended value drift caused by downstream knowledge adaptation, VISA proposes a modular shielded adaptation framework that decouples frozen base content generation from a lightweight value rewriter, leveraging GRPO to jointly optimize value steering precision and bidirectional semantic consistency without sacrificing factual integrity.