Skip to content

Personalized Alignment of Language Models with Human Value Profiles

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/UMichHCI/SchwartzValueAlignment
Area: LLM Reasoning
Keywords: Value Alignment, Personalized Alignment, Schwartz Human Values, Activation Steering, LoRA-DPO

TL;DR

This paper introduces an end-to-end framework for personalized value alignment grounded in Schwartz's Refined Theory of Basic Human Values (19 discrete values), combining scenario-aware relevance gating to isolate top target and deprioritized values with targeted prompting, residual stream activation steering, and LoRA-DPO adapters, achieving state-of-the-art alignment with individual value priorities while preserving rigorous argumentation quality.

Background & Motivation

Dominant post-training alignment paradigms for modern large language models, including reinforcement learning from human or AI feedback (RLHF and RLAIF), optimize a centrally determined normative policy that applies a single uniform standard across all users. While highly effective in objective tasks with broad societal consensus such as factual question answering and code generation, this one-size-fits-all approach inevitably falters in inherently subjective domains like career counseling, lifestyle dilemmas, and interpersonal conflict navigation. In these settings, there is no universally correct recommendation; enforcing a monolithic normative stance frequently leads models to generate bland, conventional platitudes or impose unexamined cultural biases that actively conflict with the user's worldview.

To advance beyond monolithic defaults, existing pluralistic alignment methods calibrate models to demographic groups or assigned personas, but this group-level conditioning risks algorithmic profiling and exacerbates stereotyping by presuming intra-group homogeneity. Conversely, existing individual-level alignment techniques rely heavily on task-specific preference elicitations, immediate interaction demonstrations, or superficial stylistic mirroring. Such signals are fragmentary and situationally bound: they reflect how an individual responds within a specific, narrow interaction rather than capturing the deeper, enduring principles that govern how they balance competing trade-offs across novel contexts.

Psychological and cultural research demonstrates that human values are trans-situational, enduring motivational principles that systematically guide how individuals process evidence, weigh trade-offs, and resolve conflicting goals. Grounded in Schwartz's Refined Theory of Basic Human Values, which organizes 19 validated values into a circumplex capturing motivational compatibility and conflict, this work rethinks model personalization as value alignment. Core idea: ground personalized alignment in Schwartz's stable 19-value circumplex space, using a context-aware relevance gate to dynamically isolate query-implicated target priorities and deprioritized values, and orthogonally combining targeted value prompting with internal representation steering (activation steering or modular LoRA-DPO) to achieve faithful value personalization without degrading logical argumentation quality.

Method

Overall Architecture

The framework encompasses four core stages: first, Individual Value Profiling and Sampling, which derives mean-centered, normalized 19-dimensional value priority vectors from empirical 57-item Portrait Values Questionnaire (PVQ) survey responses; second, Context-Aware Target Value Selection, where a structured LLM classifier scores the relevance of each Schwartz value to a given scenario and gates it with the user's personal profile to extract positive steering targets and negative deprioritization targets; third, Multi-Modal Value Intervention, which operationalizes these targets via Top-k targeted prompting, residual stream activation steering, modular LoRA-DPO adapters, or their hybrid combinations; and fourth, Multi-Metric Evaluation, assessing both profile alignment (Spearman rank correlation, Jensen-Shannon divergence, top-k overlap) and argument logical quality grounded in the Toulmin model.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["User Scenario / Query x<br/>(Value-Tension Dilemma)"] --> B["Individual Value Profiling & Sampling<br/>PVQ 57-item survey·19D mean-centered vector u"]
    A --> C["Context-Aware Target Value Selection<br/>Classifier rv gated with uv·Top-K positive & negative"]
    B --> C
    C --> D["Targeted Value Prompting with Deprioritization<br/>Top-k prompt: explicit target priorities & contrastive exclusion"]
    C --> E["Internal Representation Steering<br/>Residual stream PCA steering vectors / Modular LoRA-DPO adapters"]
    D --> F["Frozen Base LLM Inference<br/>Llama-3.1-8B-Instruct / Qwen3-8B"]
    E --> F
    F --> G["Multi-Metric Alignment & Logic Evaluation<br/>Spearman rank ρ·JS divergence·Top-k overlap·Toulmin LQ score"]

Key Designs

1. Individual Value Profiling and Sampling: Resolving Demographic Stereotyping and Profile Space Under-Coverage Rather than relying on crude demographic categories or synthetic personas, the framework derives continuous individual value vectors from a nationally representative U.S. sample of 959 individuals completing the 57-item Portrait Values Questionnaire. Each respondent's raw Likert scores across items mapped to the 19 Schwartz values are averaged and mean-centered within-person to produce a profile \(u \in \mathbb{R}^{19}\), where positive coordinates denote priorities above the user's personal baseline and negative values denote lower priorities. To ensure rigorous evaluation across the entire circumplex without over-representing dense normative clusters, the authors perform Farthest Point Sampling (FPS) over a 20-dimensional standardized space (19 value weights plus 3x-upweighted variance to capture both peaked and flat profiles) grouped by dominant value, selecting a diverse test bank of 152 maximally divergent profiles.

2. Context-Aware Target Value Selection: Overcoming Full-Profile Information Dilution and In-Context Noise Real-world advice scenarios engage only a small subset of the 19 human values; supplying an unselected full profile to an LLM dilutes the steering signal with irrelevant preferences. The framework implements a context-aware gate using a ternary LLM classifier that assigns query relevance scores \(r_v \in \{0, 0.5, 1\}\) across all 19 values. A profile-conditioned relevance score is then computed: $\(s_v = u_v \cdot r_v\)$ Because \(u_v\) is mean-centered, \(s_v\) naturally down-weights query-relevant values that are personally unimportant to the user while amplifying those they strongly endorse. The system selects the top \(K=2\) values under \(s_v\) as positive steering targets \(\mathcal{T}(x, u) = \text{TopK}_v(s_v)\), alongside up to \(K\) scenario-relevant values with negative personal scores as explicit deprioritization targets, providing focused and context-specific steering coordinates.

3. Targeted Value Prompting with Deprioritization: Guiding Decision Boundaries Without Terminology Leakage At the prompt layer, rather than injecting an exhaustive catalog of 19 values, Top-k targeted prompting inserts concise definitions of only the 2 selected target values into the system prompt with strict instructions to embody these principles in the recommendation while never mentioning the value names or Schwartz terminology explicitly. Critically, the prompt incorporates a deprioritization clause enumerating the scenario-relevant low-priority values to steer away from. Ablation demonstrates that this contrastive exclusion prevents the model from settling into non-committal, compromise-driven advice, sharpening the decision boundary between competing options.

4. Internal Representation Steering: Eliminating Surface Prompt Brittleness via Latent Space Intervention To ensure deep, structural alignment that moves beyond prompt compliance, the framework introduces two complementary model-level interventions. The first is Activation Steering, where contrastive pairs from 1,501 curated value-tension scenarios are used to extract layer-wise activation difference vectors, followed by PCA to extract a primary steering vector \(v_\ell^{(k)}\) per value. During generation, these vectors are injected into the residual stream at mid-to-late transformer layers (layers 15–21) scaled by strength \(\alpha=1\): $\(h_{\ell,t} \leftarrow h_{\ell,t} + \alpha \sum_{k \in \mathcal{K}} v_\ell^{(k)}\)$ The second is Modular LoRA-DPO, where 19 value-specific LoRA adapters (rank 16, alpha 32) are trained offline using Direct Preference Optimization on advice-style contrastive pairs. At inference time, the adapters corresponding to the target values are composed. Combining targeted prompting with internal steering pushes model outputs significantly closer to the user's specific value priorities than either mechanism can achieve alone.

Loss & Training

For the LoRA-DPO condition, the training dataset is derived from 1,501 validated value-tension scenarios generated across 132 unique opposing Schwartz value pairs. Each scenario provides contrastive advice-style candidate responses \((x, \tilde{s}_v^+, \tilde{s}_v^-)\) validated by a multi-judge panel to ensure authentic dilemma structure and rhetorical balance. The DPO loss optimizes the LoRA parameters \(\theta_v\) against the frozen reference model \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{DPO}}(\theta_v; \theta_{\text{ref}}) = -\mathbb{E}_{(x, \tilde{s}_v^+, \tilde{s}_v^-)} \left[ \log \sigma \left( \beta \log \frac{\pi_{\theta_v}(\tilde{s}_v^+ \mid x)}{\pi_{\text{ref}}(\tilde{s}_v^+ \mid x)} - \beta \log \frac{\pi_{\theta_v}(\tilde{s}_v^- \mid x)}{\pi_{\text{ref}}(\tilde{s}_v^- \mid x)} \right) \right]\)$ Adapters are inserted into the attention projection matrices (\(W_q, W_k, W_v, W_o\)) with rank \(r=16\), \(\alpha_{\text{lora}}=32\), and dropout 0.05. For activation steering, hyperparameter optimization balances normalized logic quality against relative value-alignment gain across layers and steering strengths, identifying layers 15–20 and 16–21 with steering strength \(\alpha=1.0\) as the Pareto-optimal configuration.

Key Experimental Results

Main Results

The primary evaluation assesses 20,064 generated responses per condition using Llama-3.1-8B-Instruct across 152 diverse held-out human profiles and 1,501 value-tension scenarios. Metrics include Spearman profile rank correlation \(\rho\), Jensen-Shannon (JS) divergence over scenario-relevant value distributions, Top-k value set overlap (Top-1 exact, Top-2, and Top-3 Jaccard overlap), and average logical quality score (LS, on a 1–5 scale based on Toulmin argument components evaluated by a GPT-4o judge).

Condition Spearman \(\rho\) \(\uparrow\) JS Divergence \(\downarrow\) Top-1 Overlap \(\uparrow\) Top-2 Overlap \(\uparrow\) Top-3 Overlap \(\uparrow\) Logic Score (LS) \(\uparrow\)
Base Model (No Personalization) 0.131 0.176 0.224 0.367 0.530 3.995
Base + Demographics 0.133 0.172 0.224 0.368 0.530 3.953
Base + Full Value Profile 0.292 0.152 0.337 0.445 0.573 3.637
Top-k Value Prompt 0.372 0.141 0.348 0.500 0.594 3.811
Activation Steering Alone 0.336 0.151 0.322 0.481 0.587 3.460
Activation + Top-k Prompt 0.478 0.133 0.402 0.563 0.625 3.163
LoRA-DPO Alone 0.377 0.139 0.335 0.500 0.602 3.699
LoRA-DPO + Top-k Prompt (Ours Best) 0.522 0.126 0.412 0.608 0.637 3.410

Supplementary evaluations confirm that the performance advantages of targeted selection and representation augmentation generalize robustly to Qwen3-8B (LoRA + Top-k reaches \(\rho=0.593\), JS=0.124) and Gemini 2.5 Flash-Lite (Top-k achieves \(\rho=0.354\) vs. Full Profile \(\rho=0.214\) and Base \(\rho=0.064\)).

Ablation Study

The ablation study validates the specific impact of the contrastive deprioritization clause within Top-k prompting, followed by linear mixed-effects model (LMM) planned contrasts across condition pairs (evaluated with multivariate-t adjusted p-values):

Configuration / Contrast Spearman \(\rho\) JS Divergence Logic Score (LS) Significance (\(p_{mvt}\))
Top-k Prompt (Full) 0.363 0.146 3.807 -
w/o Lower-Priority Deprioritization Clause 0.295 0.155 3.875 \(p < .001\)
Top-K vs. Full Profile +0.080 -0.011 - \(p < .001\)
LoRA + Top-K vs. Top-K +0.150 -0.015 - \(p < .001\)
LoRA + Top-K vs. LoRA +0.145 -0.014 - \(p < .001\)
Activation + Top-K vs. Top-K +0.106 -0.008 - \(p < .001\)
LoRA + Top-K vs. Activation + Top-K +0.044 -0.007 - \(p < .001\)

In human validation studies, 52 participants rated logical quality: human ratings averaged 4.01/5 versus 3.75 from the LLM judge, exhibiting identical condition ordering while penalizing LoRA + Top-k significantly less severely (+0.47 gap), with judge-human quadratic-weighted agreement (\(\kappa=0.212\)) doubling human-human agreement (\(\kappa=0.101\)). In value-expression validation across 51 participants and 3,060 ratings, human ratings increased for steered target values by +0.19 points (\(p=0.031\)) while other values remained unchanged, demonstrating high human-LLM agreement (\(\rho=0.48\)).

Key Findings

  • Focused Selection Outperforms Mere Exposure: Targeted Top-k prompting outperforms full-profile prompting by 0.08 in Spearman rank correlation (\(p < .001\)), demonstrating that selective conditioning on query-relevant priorities and explicit negative exclusion is vastly superior to exposing the model to the user's uncurated full profile.
  • Orthogonal Synergy Between Prompting and Representation Interventions: Combining targeted prompting with either activation steering (\(\rho=0.478\)) or LoRA-DPO (\(\rho=0.522\)) produces statistically significant gains over any single method alone, showing that latent representation steering and in-context conditioning act as mutually reinforcing channels that push the model past its default behavioral basins.
  • Asymmetric Alignment Across the Value Circumplex: Value steering efficacy is strongly non-uniform across Schwartz quadrants. Socially focused values (Self-Transcendence such as benevolence and universal concern, and Conservation such as tradition) are aligned easily with high baseline affinity, whereas personally focused values (Self-Enhancement including dominance, power resources, and achievement) encounter severe steering resistance due to the base model's ingrained prosocial, cautious alignment priors.

Highlights & Insights

  • Grounded Psychological Principles Over Fragmentary Signals: Anchoring personalization in Schwartz's 19-value circumplex replaces brittle persona prompts and demographic proxies with a mathematically continuous, culturally validated, trans-situational foundation that generalizes across unseen dilemmas.
  • Contrastive Negative Shaping in Value Steering: Incorporating explicit deprioritization of context-relevant low-priority values in both prompts and DPO pairs forces the model to take decisive, trade-off-aware stances, directly overcoming the default tendency of LLMs to offer equivocal non-answers.
  • Modular Zero-Overhead Scalability: The decoupling of independent value-specific LoRA adapters and cached activation vectors enables dynamic, inference-time linear composition for arbitrary multi-value profiles without requiring expensive per-user model fine-tuning.

Limitations & Future Work

  • Shared vs. Subjective Relevance Estimation: The framework uses a shared, objective value relevance classifier, whereas real-world human perception of which values are at stake in a given dilemma is itself subjective and varies across individuals; future work should explore personalizing the relevance estimation step.
  • Linear Composition of Conflicting Vectors: Both activation steering and LoRA composition currently rely on linear summation, which may cause unintended geometric cancellation or distortion when target values occupy diametrically opposed quadrants of the circumplex; non-linear, context-adaptive routing is an open challenge.
  • Ethical Dual-Use and Persuasion Risks: Aligning language model advice with an individual's core values inherently enhances rhetorical resonance and persuasiveness; in adversarial hands, this mechanism could be weaponized for targeted micro-targeting, manipulative propaganda, or deceptive marketing, requiring mandatory deployment disclosures.
  • vs. Demographic and Persona Conditioning (Argyle et al., 2023; Sorensen et al., 2024): Group-based pluralistic alignment assumes within-group homogeneity and risks stereotyping. This work achieves true fine-grained individual alignment by conditioning on psychometrically validated continuous value profiles.
  • vs. In-Context Preference and Interaction Steering (Han, 2023; Bo et al., 2025): Prior individual personalization relies on superficial stylistic imitation or narrow prompt demonstrations. Grounding in the Schwartz circumplex provides a generalizable cognitive scaffold that successfully navigates completely novel value-tension trade-offs.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering framework operationalizing Schwartz's 19-value circumplex for individual LLM alignment across prompts, activations, and adapters]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [1,501 curated scenarios, 152 empirical human profiles, three foundation model architectures, and dual IRB human validation studies]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear argumentation, robust mathematical formalization, and rigorous statistical hypothesis testing]
  • Value: ⭐⭐⭐⭐⭐ [Establishes a scalable, principled paradigm for personalized AI advisors facing subjective, value-laden dilemmas]