Skip to content

โš–๏ธ Alignment & RLHF

๐Ÿง  NeurIPS2026 ยท 44 paper notes

๐Ÿ“Œ Same area in other venues: ๐ŸŽž๏ธ ECCV2026 (6) ยท ๐Ÿ“ท CVPR2026 (12) ยท ๐Ÿ”ฌ ICLR2026 (102) ยท ๐Ÿ’ฌ ACL2026 (38) ยท ๐Ÿงช ICML2026 (37) ยท ๐Ÿค– AAAI2026 (17)

๐Ÿ”ฅ Top topics: Alignment/RLHF ร—20 ยท LLM ร—13 ยท Adversarial Robustness ร—4 ยท Reinforcement Learning ร—3

Alignment Dynamics in LLM Fine-Tuning

By deriving closed-form updates of an alignment score via empirical Neural Tangent Kernels, this paper decomposes alignment evolution into a Driving Force and a Rebound Force, explaining why distribution narrowness amplifies alignment reversal and identifying a Rehearsal Priming Effect where prior alignment accelerates subsequent re-alignment.

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Holding the wrong answer fixed and changing its endorser, the paper uses direction removal and an independently fitted attribution patch to show selectively intervenable differences between wrong-source deference and user agreement in three open-weight families, so one sycophancy evaluation cannot substitute for the other.

Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning

To address the pervasive issue of cyclic, intransitive preferences in LLM fine-tuning where optimal policies cease to exist under scalarized multi-objective feedback, this paper introduces the Maximum Entropy Blackwell Winner (MaxEntBW) solution concept and derives PROSPER, a provably efficient single-player regression algorithm that bypasses adversarial training while establishing new state-of-the-art results on instruction-following benchmarks.

Bayesian Preference Learning for Test-Time Steerable Reward Models

This paper introduces Variational In-Context Reward Modeling (ICRM), framing test-time preference adaptation as amortized variational inference with a conjugate Beta prior to dynamically steer classifier reward models via few-shot demonstrations across single- and multi-objective alignments while theoretically mitigating reward over-optimization.

Bayesian Test-Time Inference of Task-Aligned Similarity from Weak Interactive Feedback

Introduces BASIS, a training-free Bayesian framework that leverages weak interactive feedback (scalar scores, pairwise preferences, or verifier outputs) to jointly infer candidate targets and task-aligned similarity criteria, selecting informative probes via expected mutual information gain.

Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph

To overcome the limitations of standard DPO that collapses multi-rollout responses into independent pairs and discards transitivity, this paper proposes GraphDPO, which structures rollouts into a directed acyclic preference graph and optimizes a local Plackettโ€“Luce objective over dominated neighborhoods in linear time using equivalence-class layering, achieving substantial gains across reasoning and program synthesis.

BSO: Safety Alignment Is Density Ratio Matching

This paper proves that the likelihood ratio of the optimal safe policy decomposes in closed form into data and reference ratios, formulating safety alignment as Bregman divergence-based density ratio matching (BSO) without auxiliary reward/cost models while attaining a superior helpfulness-safety Pareto frontier.

Consolidating Rewarded Perturbations for LLM Post-Training

Addressing the prohibitive test-time compute of prediction-level ensembling in weight-space post-training (e.g., RandOpt), CoRP introduces a gradient-free consolidation operator that uncovers the reproducible low-rank geometry of rewarded Gaussian perturbations, combines compatibility-aware reweighting with held-out validation gating, and deploys a single consolidated model that achieves over half of the 50-pass ensemble gain at one forward pass per test example using 1/10th the perturbation budget.

Diffusion LLMs are Natural Adversaries for any LLM

Reformulates adversarial prompt optimization as an amortized inference problem using pretrained Diffusion LLMs (DLLMs), sampling low-perplexity, highly transferable jailbreaks via conditional inpainting on target harmful responses without accessing target model parameters.

Distributionally Robust Token Optimization in RLHF

To tackle the vulnerability of large language models to minor prompt perturbations in multi-step reasoning, this paper incorporates distributionally robust optimization (DRO) into token-level RLHF by constructing KL and \(\chi^2\) divergence ambiguity sets over response spans, substantially improving out-of-distribution reasoning without additional runtime overhead.

Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

This paper introduces a pointwise-constrained supervised fine-tuning framework for language models that combines augmented Lagrangian dual ascent with adaptive resilient relaxation, effectively mitigating tail constraint violations while preserving average downstream utility.

Explaining and Preventing Alignment Collapse in Iterative RLHF

Formalizes iterative RLHF as a Stackelberg game where standard myopic policy updates drop an analytical parameter-steering gradient term, inducing catastrophic alignment collapse, and proposes Foresighted Policy Optimization (FPO) via a scalable first-order TracIn penalty to restore game-theoretic self-correction.

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

This paper characterizes catastrophic forgetting in LLM continual post-training as a state-relative update-integration failure, formulates geometry conflict via normalized Bures-Wasserstein distance, and develops Geometry-Conflict Wasserstein Merging (GCWM) to achieve data-free, compatibility-controlled parameter integration.

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

This paper decomposes agentic reinforcement learning data into task, environment, and trajectory cardinality axes, reveals that scaling exponents follow \(\alpha_{\mathrm{E}} \gg \alpha_{\mathrm{T}} \gg \alpha_{\tau}\), prices heterogeneous synthesis pipelines through empirical discount factors and collapse thresholds, and derives a Chinchilla-style compute-optimal allocation recipe.

How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis

This paper theoretically analyzes how neural reward models learn features and optimize policies within a Gaussian single-index framework, proving that exponential reward weighting reduces the first-layer feature learning complexity from the information exponent to the generative exponent at a dimension-free \(O(1)\) temperature, and establishing tight tilted-policy value-gap bounds that decouple feature learning from downstream deployment temperatures.

InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization

This paper introduces Intrinsic Self-reflective Preference Optimization (InSPO), which leverages the alternative response in preference pairs as privileged information during training via symmetric cross-conditioning, fundamentally overcoming the lack of invariance and suboptimal contrast exploitation in DPO-family algorithms while maintaining zero extra inference overhead.

IRIS: Interpolative Rรฉnyi Iterative Self-play for Large Language Model Fine-Tuning

IRIS unifies self-play fine-tuning under the Rรฉnyi divergence family by decoupling the objective into two independent tilted risk terms over annotated and synthetic data, utilizing a distributional gap feedback schedule to dynamically calibrate optimization geometry across iterations.

Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality

This paper introduces the Knowledge-Level Consistency Reinforcement Learning Framework (KLCF), which reframes long-form factual alignment as bidirectional distribution matching between the policy's expressed knowledge and the base model's parametric knowledge. Using a retrieval-free Dual-Fact Alignment mechanism consisting of a factual checklist and a lightweight truthfulness reward model, KLCF jointly optimizes factual recall and precision during online RL without requiring external retrieval.

Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment

GAPO constructs an ephemeral geometric anchor near the current policy along the direction that decreases the batch-average preference margin, then reduces brittle pairs' update weights according to their margin degradation, improving multiple alignment evaluations and controlled noise experiments without winning every modelโ€“metric combination.

LLM Active Alignment: A Nash Equilibrium Perspective

This paper develops a game-theoretic framework that models LLM actions as mixture policies over human subpopulations, deriving an analytical closed-form Nash equilibrium to diagnose, predict, and eliminate the phenomenon of "political exclusion" arising from multi-agent strategic competition.

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

MaPPO extends Maximum Likelihood Estimation in Direct Preference Optimization to a principled Maximum a Posteriori (MAP) framework by incorporating prior reward differentials, curbing the probability squeezing effect and negative over-penalization with zero extra hyperparameters.

Measuring Safety Alignment Effects in Autonomous Security Agents

This paper introduces a system-level benchmark of 30 local sandboxed security tasks with 1,500 multi-turn traces to compare stock safety-aligned models against uncensored or abliterated derivatives, revealing that visible refusal rates fail to explain agent effectiveness and that differences are governed by evidence grounding, derivative provenance, and tool-interface reliability.

Online Data Selection for Instruction Tuning via Gaussian Processes

Addressing the batch-constrained bottleneck where online data selection is restricted to local sub-selection within randomly drawn batches, GAIA models the continuous utility manifold over the entire training corpus via Gaussian Processes and employs fixed-share Hedge updates to achieve no-regret adaptive strategy selection under non-stationary quality scores, significantly accelerating convergence and boosting downstream generalization on MMLU and SamSum.

Oracle-Robust Online Alignment for Large Language Models

Addressing preference oracle misspecification in online LLM alignment, this paper formulates a worst-case objective over a pointwise Wasserstein uncertainty set, proves an exact closed-form decomposition into nominal SAIL loss plus an explicit score sensitivity penalty under log-linear policies, and establishes an \(\widetilde{\mathcal{O}}(\varepsilon^{-2})\) oracle complexity bound via projected stochastic composite gradient descent on the Moreau envelope.

PAIR: Prefix-Aware Internal Reward Model for Multi-turn Agent Optimization

Proposes PAIR, a prefix-aware internal reward model that decouples belief-consistency via a frozen hidden-state probe and corrects it toward grounded correctness using multi-layer attention features, delivering dense step-level rewards for multi-turn agent GRPO at negligible marginal compute cost.

Prompts to Proxies: Emulating Human Preferences via a Compact LLM Ensemble

Introduces preference reconstruction theory and Prompts to Proxies (P2P), a two-stage modular framework that reconstructs population-level human preference distributions via entropy-guided active persona generation and constrained L1-regularized regression, eliminating fine-tuning and demographic privacy requirements at less than $1 per survey.

Response Time Enhances Alignment with Heterogeneous Preferences

Addressing the fundamental non-identifiability of population-average preferences in anonymous, heterogeneous annotator pools under standard binary feedback, this paper incorporates zero-cost response times within a drift-diffusion model and introduces a two-stage Richardson-extrapolated consistent estimator that provably bypasses the choice-only distortion barrier.

Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks

Introduces Recursive Rubric Decomposition (RRD), a principled framework that decomposes coarse rubrics, filters directionally misaligned and redundant criteria, and applies whitened uniform weighting to significantly improve LLM judge accuracy and stabilize reinforcement fine-tuning rewards for open-ended tasks.

Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure

Reveals that single-axis bias mitigations (e.g., penalizing response length) redirect optimization pressure onto correlated proxy features (such as overconfidence) via a measurement-versus-optimization gap, proves that audit-distribution scoring cannot distinguish genuine mitigation from harmful substitution, and establishes provably sufficient multi-axis policy-distribution evaluation criteria.

Robust Nash Alignment under Preference Uncertainty

This paper proposes Robust Nash Alignment, a distributionally robust game-theoretic alignment framework that models uncertain pairwise preferences within a Bernoulli-KL divergence ball, reformulates the non-convex hard-constrained optimization into a four-player primal-dual proxy game, and achieves \(\mathcal{O}(1/\sqrt{T})\) convergence with certified worst-case win-rate guarantees.

S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

Addressing the optimization instability caused by ambiguous credit assignment from sequence-level scalar rewards in standard RLHF, this paper proposes S2T-RLHF, a two-stage hierarchical credit assignment framework that first allocates preference rewards across sentences via cooperative Nash bargaining and then applies bounded token-level refinement via a lightweight Dirichlet network, substantially improving training stability and robustness without retraining the reward model.

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

Addressing policy entropy collapse in reinforcement learning post-training (RLVR/GRPO), this paper reveals a fundamental temperature-sign asymmetry where positive and negative samples have opposite effects under different temperatures, and proposes SCOPE-RLโ€”a lightweight framework that utilizes less than 2% temperature-adaptive positive samples to achieve stable, quantitative entropy control and consistent gains across reasoning benchmarks.

SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training

SCOPE-RL tackles the pervasive entropy collapse in GRPO-based post-training of reasoning LLMs by uncovering a temperature-sign asymmetry in entropy dynamics and introducing a lightweight (<2% compute overhead) temperature-adaptive positive-sample regularizer that reliably stabilizes exploration and boosts both Pass@1 and Pass@k.

SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization

To tackle under-thinking and severe token redundancy caused by large reasoning models abandoning viable reasoning paths despite discovering them early, SteadyThought identifies critical thought-switching junctions via dual linguistic-entropy signals, synthesizes single-path commit trajectories under logit control, and conducts thought-level preference optimization (STPO) directly at divergence points, reducing reasoning token consumption by up to 63.8% while boosting task accuracy.

Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing

This paper introduces the Consensus Clustering LinUCB Bandit (CCLUB) framework, which steers frozen large language models via online system-prompt routing using dual-feature context representations and conservative consensus graph intersections, achieving adaptive social alignment with sublinear regret guarantees.

Theoretical Limits of Language Model Alignment

This paper derives the exact information-theoretic limits of KL-regularized language model alignment, proving that optimal expected reward gain is characterized by Jeffreys divergence and can be estimated training-free via base-model sample covariance, while formally explaining exponential reward hacking under proxy rewards and proving that Best-of-N closely tracks the Pareto frontier whereas PPO and GRPO remain substantially suboptimal.

Think-with-Rubrics: From External Evaluator to Internal Reasoning Guidance

Internalizes rubric generation from an external post-hoc reward evaluator into an explicit thinking step within the model's reasoning trace, and jointly optimizes the policy using consistency rewards from both golden and self-generated rubrics, substantially boosting instruction-following precision and demonstrating self-evolution capability.

TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback

Addressing the failure of clipping on negative advantages and horizon variance explosion in Group Relative Policy Optimization (GRPO), TIC-GRPO introduces an up-only cap and full-trajectory importance correction to reduce response-level variance from \(\mathcal{O}(1)\) to \(\mathcal{O}(1/T)\), significantly improving optimization stability and performance across reasoning benchmarks.

Training-Free Cultural Alignment of Large Language Models via Persona Disagreement

This paper proposes DISCA, a training-free cultural alignment method that treats within-country demographic disagreement across World Values Survey (WVS) persona agents as a calibration reliability signal, steering decisions via loss-averse Prospect Theory importance sampling and dual-pass shrinkage to reduce MultiTP cultural misalignment by 10โ€“24% across open-weight models without any weight updates.

Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation

This paper reveals an implicit advantage symmetry inherent in Group Relative Advantage Estimation (GRAE) in GRPO at both group and sample levels, explains why it hinders unseen solution exploration and freezes difficulty allocation, and proposes Asymmetric GRAE (A-GRAE) to dynamically break this symmetry, consistently improving accuracy and Pass@k across LLMs and VLMs.

Visual-ERM: Reward Modeling for Visual Equivalence

Proposes Visual-ERM, a multimodal generative reward model that evaluates vision-to-code quality directly in the rendered image space with fine-grained and task-agnostic feedback, improving Qwen3-VL-8B by +8.4 on chart-to-code and substantially boosting test-time reflection.

When Helpfulness Becomes Sycophancy: Sycophancy is a Boundary Failure Between Social Alignment and Epistemic Integrity in Large Language Models

This position paper reframes sycophancy in LLMs as a boundary failure between social alignment and epistemic integrity, proposing a three-condition decision rule and a three-dimensional taxonomy encompassing alignment targets, interaction mechanisms, and impact severity.

When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy

Unveils the paradox between constant policy entropy and perceptual diversity collapse in flow-based RLHF, introducing perceptual entropy and regularized reward shaping that dramatically enhances diversity without compromising visual fidelity.

Avoiding Obfuscation with Prover-Estimator Debate

To overcome the foundational vulnerability where dishonest debaters construct intractable subclaims to trigger the "obfuscated arguments problem" in recursive debate, this paper proposes an asymmetric Prover-Estimator Debate protocol equipped with a \(\rho\)-stability condition, theoretically proving that an honest debater can win in approximate Stackelberg equilibria without requiring computational superiority over dishonest opponents.