VISA: Value Injection via Shielded Adaptation for Personalized LLM Alignment¶
Conference: EMNLP 2026
OpenReview: https://openreview.net/forum?id=2026.emnlp-main.303
Paper: ACL Anthology
Code: https://github.com/opqrst-chen/VISA
Area: Alignment & RLHF
Keywords: Personalized Alignment, Value Drift, Shielded Adaptation, GRPO, Schwartz Basic Values
TL;DR¶
Addressing the unintended value drift caused by downstream knowledge adaptation, VISA proposes a modular shielded adaptation framework that decouples frozen base content generation from a lightweight value rewriter, leveraging GRPO to jointly optimize value steering precision and bidirectional semantic consistency without sacrificing factual integrity.
Background & Motivation¶
Post-deployment large language models are routinely adapted to new domains, tasks, or diverse user and cultural communities. However, conventional adaptation paradigms based on parameter updates—such as supervised fine-tuning (SFT) or preference optimization—fail to cleanly disentangle task-specific knowledge, stylistic nuances, and latent normative signals in the training corpus. Even when fine-tuning corpora are purely knowledge-intensive (e.g., mathematics, medicine, or contract law), models unavoidably absorb incidental normative patterns. This gives rise to "value drift," an insidious manifestation of the alignment tax where acquiring new capabilities destabilizes pre-existing alignment.
A uniform, monolithic global alignment policy is inherently too coarse to accommodate the heterogeneous expectations of varied cultures and enterprises regarding authority, politeness, and risk. Conversely, maintaining separate specialized fine-tuned checkpoints incurs prohibitive training and storage overheads. Crucially, attempting to alter a model's normative stance via naive prompting or direct parameter updates frequently compromises factual reasoning, induces severe hallucinations, or causes catastrophic degradation in domain competence, because factual generation and normative framing are deeply entangled within the autoregressive decoding process.
To resolve this conflict between reliable content preservation and controlled value steering, the fundamental angle of attack is to decouple stable content generation from explicit value modification, transforming value steering from an uncontrollable side effect of weight updates into a measurable, bounded operation. Core idea: Decouple a frozen base LLM from a lightweight Value Rewriter, using a vector-space Value Detector and Instruction Translator to establish explicit targets, and optimize the rewriter via GRPO with dual rewards for directional value alignment and bidirectional entailment consistency.
Method¶
Overall Architecture¶
The VISA pipeline comprises three learnable components: an Instruction Translator (\(T_\phi\)), a Value Detector (\(D_\psi\)), and the core Value Rewriter (\(\pi_\theta\)). Given a user query \(x\), an initial response \(y_{\text{orig}}\) generated by the frozen base model, and a natural-language steering instruction \(I\), \(T_\phi\) translates the instruction into a relative value shift vector \(\Delta v\). Concurrently, \(D_\psi\) evaluates the intrinsic Schwartz value vector \(v_{\text{orig}}\) of the source response. The target profile is established via \(v_{\text{target}} = \text{clip}(v_{\text{orig}} + \Delta v, -1, 1)\). Finally, \(\pi_\theta\) generates the rewritten output \(y_{\text{rewr}}\) conditioned on \(x\), \(y_{\text{orig}}\), and \(v_{\text{target}}\). In implicit adaptation regimes where the ideal value target is unknown, VISA functions as a reusable value-conditioned operator embedded within an Adaptive Value Search procedure.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
In["Input: User Query x and Instruction I"] --> Base["Base Generation<br/>Frozen base model produces y_orig"]
Base --> D1["1. Auxiliary Space & Target Construction<br/>T_phi predicts offset + D_psi estimates baseline"]
D1 --> D2["2. Dual-Objective Policy Optimization via GRPO<br/>Group rollout + Group-relative advantage estimation"]
D2 --> D3["3. Joint Reinforcement Learning Rewards<br/>Cosine value alignment R_val + Bidirectional entailment R_cons"]
D3 --> D4["4. Adaptive Value Search under Drift Constraints<br/>Bi-level candidate sampling + Advantage filtering update"]
D4 --> Out["Output: Factually faithful rewritten response y_rewr"]
Key Designs¶
1. Auxiliary Space & Target Construction: Continuous Metric Grounding via Schwartz Dimensions To eliminate the ambiguity and lack of standardized metrics in textual value specifications, this design adopts the cross-cultural Schwartz Theory of Basic Values (spanning 10 dimensions: Self-Direction, Stimulation, Hedonism, Achievement, Power, Security, Conformity, Tradition, Benevolence, Universalism) to parameterize value tendencies into a bounded continuous space \([-1.0, 1.0]^{10}\). The Value Detector \(D_\psi\), initialized from DeBERTa-v3-large, is trained with GPT-4o-assisted comparative annotations using Mean Squared Error (MSE): $\(\mathcal{L}_{\text{det}}(\psi) = \mathbb{E}_{(x, y, v_{\text{gt}}) \sim \mathcal{D}_{\text{det}}} \|D_\psi(x, y) - v_{\text{gt}}\|_2^2\)$ Similarly, the Instruction Translator \(T_\phi\) is trained via MSE to map natural-language steering requests into ground-truth delta vectors \(\Delta v_{\text{gt}}\). Together, they establish a deterministic coordinate system enabling multi-attribute algebraic operations, while remaining frozen during downstream rewriter policy optimization to serve as an objective evaluation benchmark.
2. Dual-Objective Policy Optimization via GRPO: Critic-Free Stable Generative Alignment Standard actor-critic reinforcement learning algorithms such as PPO suffer from significant training instability and value estimation errors introduced by training an auxiliary Critic network, while standard SFT or pairwise preference methods (DPO/SimPO) tend to overfit to stylistic markers and severely erode original factual details. VISA leverages Group Relative Policy Optimization (GRPO). At each training step, for an input tuple \((x, y_{\text{orig}}, v_{\text{target}})\), the rewriter samples a group of \(G\) candidate completions \(\{y_1, \dots, y_G\}\). By computing the group-level mean reward \(\bar{R} = \frac{1}{G}\sum_{j=1}^G R(y_j)\) and standard deviation \(\sigma_R\), the advantage is estimated directly as: $\(\hat{A}_j = \frac{R_{\text{total}}(y_j) - \bar{R}}{\sigma_R + \epsilon}\)$ This mechanism bypasses the overhead and variance of a Critic network, ensuring monotonic surrogate lower-bound improvements within a local trust region and anchoring the rewriter policy tightly within the base model's high-consistency generation manifold.
3. Joint Reinforcement Learning Rewards: Balancing Directional Steering and Bidirectional Semantic Integrity To overcome the severe trade-off where steering value framing induces content distortion and hallucinations, VISA formulates a composite reward \(R_{\text{total}} = R_{\text{val}} + R_{\text{cons}}\) with complementary non-conflicting objectives. The value alignment reward \(R_{\text{val}}\) evaluates the cosine similarity between the predicted vector and the target vector, scaled to \([0, 2]\): $\(R_{\text{val}}(y_j) = \frac{v_{\text{pred}} \cdot v_{\text{target}}}{\|v_{\text{pred}}\|_2 \|v_{\text{target}}\|_2} + 1\)$ By prioritizing directional alignment over Euclidean magnitude, it encourages capturing relative value profiles robust to embedding norm scale variations. The semantic consistency reward \(R_{\text{cons}}\), computed via an independent Fact Analyzer \(M_{\text{fact}}\), evaluates bidirectional NLI entailment. Forward entailment \(P(y_{\text{orig}} \to y_j)\) penalizes critical information omissions, while backward entailment \(P(y_j \to y_{\text{orig}})\) penalizes unsupported additions and hallucinations. This bidirectional constraint restricts modifications strictly to adjectives, adverbs, emphasis hierarchy, and rationales without mutating core facts.
4. Adaptive Value Search under Drift Constraints: Bi-Level Exploration for Unsupervised Adaptation In domain adaptation scenarios, developers target downstream capability improvements (e.g., mathematical reasoning) without knowing which value direction will prevent value drift. This design utilizes VISA as a synthetic adaptation generator within a bi-level optimization scheme. The inner loop samples candidate value vectors \(v_i\) from a distribution \(\mathcal{P}_t = \mathcal{N}(\mu_t, \Sigma_t)\), synthesizes value-conditioned datasets via the rewriter, applies rapid mock updates to the base model, and scores downstream composite rewards combining accuracy and inverse drift. The outer loop selects candidates with positive advantages \(\hat{A}_i > 0\) into a set \(S_{\text{pos}}\) and iteratively refines the search distribution parameters \(\mu_{t+1}\) and \(\Sigma_{t+1}\), discovering the optimal value direction that minimizes drift while enhancing task accuracy.
Loss & Training¶
The Rewriter is optimized through a two-stage regime: 1. Phase 1: SFT Cold Start: Full-parameter fine-tuning on VCR-45K using DeepSpeed ZeRO-3 (FP8) with a learning rate of \(1.0 \times 10^{-5}\), Cosine scheduler, warmup ratio of 0.1, per-device batch size 16, for 2 epochs, establishing baseline instruction adherence. 2. Phase 2: GRPO Training: Learning rate of \(2.0 \times 10^{-6}\) for 2 epochs, gradient accumulation steps 8, group size \(G=8\), max prompt length 512, max completion length 1536, with vLLM engine support. \(D_\psi\) and \(T_\phi\) remain entirely frozen.
Key Experimental Results¶
Main Results¶
On the held-out value-conditioned rewriting test set, VISA (initialized from Qwen3-4B and trained with GRPO) is compared against the vanilla Qwen3-4B base model and leading closed-source models across three prompting strategies (Simple, Rule-Based-Instruct, and CoT-Instruct). Evaluated metrics include NLI-based consistency (mean, forward, and backward entailment) and value alignment (L2 distance and cosine similarity).
| Method | Model | Consistency (Mean) | Consistency (Fwd) | Consistency (Bwd) | Value L2 Distance (↓) | Value Cosine Sim (↑) |
|---|---|---|---|---|---|---|
| Simple-Prompt | GPT-4o-Mini | 0.8406 ± 0.1496 | 0.8523 ± 0.1599 | 0.8172 ± 0.1979 | 0.7986 ± 0.3011 | 0.69 |
| Simple-Prompt | GPT-4o | 0.7831 ± 0.1777 | 0.8171 ± 0.1883 | 0.7151 ± 0.2666 | 0.7717 ± 0.2877 | 0.71 |
| Simple-Prompt | Gemini-3-Pro | 0.6128 ± 0.2174 | 0.6230 ± 0.2385 | 0.5924 ± 0.3494 | 0.7095 ± 0.2551 | 0.75 |
| Rule-Based-Instruct | GPT-4o-mini | 0.7652 ± 0.1721 | 0.7564 ± 0.1994 | 0.7829 ± 0.2042 | 0.8033 ± 0.3040 | 0.70 |
| Rule-Based-Instruct | GPT-4o | 0.6792 ± 0.2349 | 0.6971 ± 0.2563 | 0.6434 ± 0.3078 | 0.7719 ± 0.2893 | 0.72 |
| Rule-Based-Instruct | Gemini-3-Pro | 0.4931 ± 0.2360 | 0.4925 ± 0.2681 | 0.4942 ± 0.2998 | 0.7251 ± 0.2661 | 0.76 |
| CoT-Instruct | GPT-4o-mini | 0.6812 ± 0.1947 | 0.6386 ± 0.2404 | 0.7665 ± 0.2275 | 0.8079 ± 0.3044 | 0.70 |
| CoT-Instruct | GPT-4o | 0.6514 ± 0.1821 | 0.5908 ± 0.2439 | 0.7725 ± 0.2170 | 0.7570 ± 0.2728 | 0.75 |
| CoT-Instruct | Gemini-3-Pro | 0.5828 ± 0.2230 | 0.5520 ± 0.2559 | 0.6444 ± 0.2959 | 0.7228 ± 0.2507 | 0.75 |
| Vanilla | Qwen3-4B | 0.2340 ± 0.1694 | 0.1740 ± 0.1615 | 0.3538 ± 0.2821 | 0.9081 ± 0.2832 | 0.67 |
| Ours (VISA) | Qwen3-4B | 0.8732 ± 0.1070 | 0.8733 ± 0.1179 | 0.8729 ± 0.1354 | 0.7756 ± 0.3049 | 0.71 |
Ablation Study¶
Ablations isolate training objectives across architectures, as well as input control formats and two-stage architectures on 869 held-out evaluation samples.
| Config / Model | Objective / Setting | Consistency (↑) | Consistency Fwd (↑) | Consistency Bwd (↑) | Value L2 Distance (↓) | Joint Success Rate (↑) |
|---|---|---|---|---|---|---|
| Llama-3.1-8B | Vanilla | 0.4194 ± 0.2013 | 0.3340 ± 0.2341 | 0.5901 ± 0.3221 | 0.8156 ± 0.2391 | 0.3671 |
| Llama-3.1-8B | SFT | 0.1757 ± 0.1824 | 0.1799 ± 0.2164 | 0.1675 ± 0.2081 | 0.8761 ± 0.2871 | 0.0817 |
| Llama-3.1-8B | DPO | 0.4019 ± 0.2015 | 0.3312 ± 0.2360 | 0.5433 ± 0.3171 | 0.8138 ± 0.2389 | 0.3395 |
| Llama-3.1-8B | SimPO | 0.4098 ± 0.2012 | 0.3359 ± 0.2309 | 0.5575 ± 0.3173 | 0.8127 ± 0.2399 | 0.3510 |
| Llama-3.1-8B | GRPO (Ours) | 0.8195 ± 0.1067 | 0.7676 ± 0.1347 | 0.9233 ± 0.1323 | 0.7421 ± 0.2225 | 0.6502 |
| Qwen3-4B | GRPO (Ours) | 0.7604 ± 0.1398 | 0.7621 ± 0.1412 | 0.7570 ± 0.2850 | 0.8207 ± 0.2240 | 0.5029 |
| Control Format (869 eval) | Vector Input | 0.9000 (weighted) | 0.9414 | 0.8174 | 0.9921 | Failure rate: 1.3% |
| Control Format (869 eval) | Text Input | 0.6383 (weighted) | 0.6129 | 0.6890 | 0.9091 | Failure rate: 31.2% |
| Control Format (869 eval) | Direct-Gen (No Source) | 0.5902 (weighted) | 0.8770 | 0.0167 | 0.7795 | Backward entailment collapses |
In downstream drift-constrained domain adaptation on MATH-500 (Table 2): - Qwen3-0.6B: Base model scores 0.3645 MMLU-Math accuracy (0.0000 drift); standard SFT achieves 0.3807 accuracy with 0.1191 value drift; VISA increases accuracy to 0.3855 (+1.3%) while curbing value drift to 0.0775 (-34.9% relative to SFT). - Qwen3-8B: Base model scores 0.5224; standard SFT reaches 0.5769 with 0.0769 drift; VISA achieves 0.5760 accuracy (-0.2%) while reducing drift to 0.0437 (-43.2% relative to SFT).
Key Findings¶
- SFT suffers from catastrophic semantic forgetting: On Llama-3.1-8B, standard SFT overfits to target value markers at the expense of content, causing semantic consistency to drop to 0.1757 (forward entailment 0.1799) and joint success rate (JSR) to collapse to 0.0817; GRPO elevates JSR to 0.6502.
- Prompt engineering exhibits adverse trade-offs: More verbose, constraint-heavy prompts in closed-source models actually decrease factual fidelity (e.g., Gemini-3-Pro consistency drops from 0.6128 to 0.4931 under Rule-Based prompting).
- Two-stage rewriting is essential for joint alignment and factual fidelity: Eliminating the original response in the Direct-Gen baseline causes backward entailment to collapse to 0.0167, demonstrating that value-conditioned generation from scratch causes unacceptable factual drift.
Highlights & Insights¶
- Continuous geometric parameterization of human values: Compressing 10 Schwartz basic values into a continuous vector space provides orthogonal, disentangled steering signals, slashing consistency failure rates from 31.2% (textual) down to 1.3%.
- Critic-free stabilization with GRPO: Using group-relative advantage estimation eliminates Critic estimation drift and allows a compact 4B parameter model to outperform massive proprietary models like GPT-4o and Gemini-3-Pro on semantic preservation.
- Dual-use utility across explicit rewriting and implicit search: In addition to explicit steering, VISA acts as a plug-and-play operator in bi-level search, reducing domain adaptation value drift by up to 43.2% without degrading downstream mathematical reasoning capability.
Limitations & Future Work¶
- Absence of factual error correction: VISA is designed to faithfully preserve source content; if the frozen base model emits factual errors or logical flaws, the rewriter preserves them under the target framing.
- Bound to the Schwartz value taxonomy: The 10-dimensional coordinate system does not encompass exhaustive ethical, cultural, or legal taxonomies; incorporating alternative moral frameworks requires retraining auxiliary components.
- Multi-turn dynamics and adversarial robustness: Current validations focus on single-turn interactions; dynamics over extended dialogues and risks of malicious framing manipulation warrant further investigation.
Related Work & Insights¶
- vs Concept Activation Vectors (CAVs) / Representation Engineering: Internal activation intervention methods require per-model internal calibration and cannot be applied to black-box APIs; VISA operates externally on text, simultaneously steering ten dimensions with continuous precision.
- vs Direct Preference Optimization (DPO / SimPO): Pairwise preference optimization frequently overfits to stylistic markers and degrades semantic structure; VISA's GRPO objective with bidirectional NLI constraints ensures superior content-style disentanglement.
- vs Prompt-based Steering (In-Context Alignment): Complex prompting dilutes model attention and sacrifices semantic fidelity; VISA establishes that a lightweight specialized rewriter achieves far superior factual preservation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates the problem of unintended value drift in knowledge adaptation and introduces a clean decoupled rewriting and search paradigm.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 45K training samples, open and proprietary baselines, independent NLI/BERTScore judges, and downstream domain fine-tuning.
- Writing Quality: ⭐⭐⭐⭐⭐ Methodical formulation, mathematically grounded objectives, and clear narrative progression.
- Value: ⭐⭐⭐⭐⭐ Provides an effective, practical blueprint for mitigating the alignment tax in real-world personalized LLM deployments.