Modular Representation Steering for Multi-Attribute LLM Alignment¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: LLM Alignment, Multi-Attribute Alignment, Representation Steering, LoReFT, Compatibility Filtering
TL;DR¶
To overcome severe optimization interference in shared parameter tuning and directional cancellation when composing activation steering vectors, POLAR trains attribute-specific low-rank representation intervention modules on a frozen LLM and composes them at inference time via log-z demand calibration and greedy cosine compatibility filtering.
Background & Motivation¶
Aligning large language models (LLMs) to serve as reliable, general-purpose assistants requires satisfying multiple heterogeneous behavioral objectives simultaneously. In real-world deployment, models must uphold truthfulness, mitigate social stereotypes and demographic bias, adhere to common-sense morality and ethical standards, and suppress offensive or toxic generation. Traditional alignment paradigms such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and parameter-efficient LoRA adapt models by tuning shared parameters on combined preference datasets. However, entangling diverse objectives within a single parameter space inevitably triggers competing gradient dynamics and severe trade-offs, where boosting one alignment dimension often degrades others and precipitates catastrophic over-refusal.
To circumvent the rigidity and interference of updating shared model weights, representation steering techniques have emerged as an agile alternative. By injecting learned activation shifts directly into intermediate hidden representations at inference time, methods like RepE, CAA, and ITI achieve effective control over isolated behaviors without re-tuning the backbone. Nevertheless, existing steering paradigms remain largely confined to single-attribute interventions and lack principled mechanisms for multi-attribute composition. When multiple separately trained steering modules are naively mixed in activation space, empirical diagnostics reveal widespread geometric friction: in over 80% of token-layer records, demand-weighted combinations trigger severe scale mismatches and directional oppositions where vectors cancel each other out or drive generation off-target.
These observations reveal that multi-attribute alignment should neither force competing objectives into shared weights nor blindly sum uncalibrated activation edits. Instead, alignment requires a modular architecture capable of learning isolated behavioral directions while dynamically reconciling demand and directional compatibility in hidden-state space. By maintaining the base language model frozen, each attribute specialist can be trained independently, while inference-time routing evaluates calibrated urgency and filters out mutually conflicting edits. Core idea: train independent low-rank representation intervention modules (LoReFT) for distinct alignment objectives, normalize their cross-layer demand via log-z calibration, and dynamically compose compatible interventions via greedy cosine similarity filtering within the frozen LLM hidden-state space.
Method¶
Overall Architecture¶
The POLAR framework comprises three modular, strictly decoupled phases: attribute-specific intervention training, cross-layer demand calibration, and compatibility-aware inference composition. During training, individual LoReFT modules are learned independently on attribute-specific corpora while keeping the base language model frozen, capturing dedicated behavioral subspaces without cross-task gradient entanglement. During offline calibration, activation norms across all layers are tracked on unlabelled prompt sets to establish log-normal baselines, rectifying static norm disparities across attributes and depths. During inference, for every token's hidden state, POLAR selects top-k candidate modules based on calibrated demand scores, greedily eliminates directionally conflicting interventions whose cosine similarities with accepted candidates fall below a threshold, and performs set-restricted softmax weighting to inject the composed edit back into the hidden state.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Token Sequence and Intermediate Hidden States"] --> B["Stage 1: Attribute-Wise Low-Rank Intervention Training<br/>Independent LoReFT specialists with frozen LLM backbone"]
B --> C["Stage 2: Cross-Layer Demand Log-z Score Calibration<br/>Estimating log-norm mean and std across layers and modules"]
C --> D["Stage 3: Cosine Directional Compatibility Filtering<br/>Greedy selection of non-conflicting top-k candidates"]
D --> E["Stage 4: Set-Restricted Softmax Composition and State Injection<br/>Dynamic token-level multi-attribute activation editing"]
E --> F["Aligned Generation Across Truthful/Unbiased/Ethical/Safe Axes"]
Key Designs¶
1. Attribute-Wise Low-Rank Intervention Training: Eliminating Parameter-Space Interference Across Objectives
Joint parameter tuning entangles multi-attribute gradients, precipitating Pareto frontier collapse. POLAR circumvents this bottleneck by decoupling behavioral acquisition into modular subspace interventions via Low-rank Representation Fine-tuning (LoReFT). Given a frozen language model with parameters \(\theta\), let \(h_{\ell,p} \in \mathbb{R}^d\) represent the hidden state at layer \(\ell\) and token position \(p\). For each attribute \(a_t\), POLAR trains an independent intervention module \(\Phi_t\) parameterized by \(\phi_t = \{R_{t,\ell}, W_{t,\ell}, b_{t,\ell}\}\). The module projects \(h_{\ell,p}\) into a low-rank subspace of rank \(r \ll d\), performs a learned non-linear rewrite, and lifts the delta back to the base representation space: $\(\Delta_{t,\ell}(h_{\ell,p}) = R_{t,\ell}^\top \left( f(W_{t,\ell} h_{\ell,p} + b_{t,\ell}) - R_{t,\ell} h_{\ell,p} \right)\)$ where \(R_{t,\ell} \in \mathbb{R}^{r \times d}\) is the projection matrix, \(W_{t,\ell} \in \mathbb{R}^{r \times d}\) and \(b_{t,\ell} \in \mathbb{R}^r\) parameterize the affine subspace transform, and \(f(\cdot)\) is an activation function. Training optimizes the causal language modeling loss over response tokens on attribute dataset \(\mathcal{D}_t\) while masking prompt tokens. Because parameters are never shared across attributes, each specialist independently acquires sharp behavioral steering vectors without cross-task degradation.
2. Cross-Layer Demand Log-z Score Calibration: Harmonizing Intrinsic Module Scale Mismatches
Using the raw Euclidean norm of the lifted intervention \(q_t(h) = \| \Delta_t(h) \|_2\) as the measure of module demand introduces severe bias: certain attributes (e.g., morality or toxicity) naturally produce larger edit magnitudes across all layers due to weight scale differences, unfairly dominating composition regardless of input relevance. To establish parity, POLAR estimates module- and layer-specific statistics over an offline calibration corpus \(\mathcal{C}\) of up to 5,000 prompts: $\(\mu_{t,\ell} = \mathbb{E}_{h \sim \mathcal{C}}[\log(q_t(h) + \epsilon)], \quad \sigma_{t,\ell} = \mathrm{Std}_{h \sim \mathcal{C}}[\log(q_t(h) + \epsilon)]\)$ During inference, the raw intervention norm is converted into a standardized log-z score: $\(\tilde{q}_t(h) = \frac{\log(q_t(h) + \epsilon) - \mu_{t,\ell}}{\sigma_{t,\ell} + \epsilon}\)$ This dimensionless score measures the relative urgency of module \(t\) relative to its expected activation level at layer \(\ell\), ensuring that candidate selection reflects contextual responsiveness rather than arbitrary parameter scaling.
3. Cosine Directional Compatibility Filtering: Suppressing Opposing Vectors and Steering Cancellation
Diagnostic probes reveal that high intervention demand does not guarantee collaborative steering: the top-scoring module pair points in opposing directions in 10.4% of token-layer activations, and naive addition causes destructive interference. POLAR introduces a greedy compatibility filter. Given calibrated scores \(\{\tilde{q}_t(h)\}_{t=1}^T\), it first extracts the top-k candidate set \(K(h) = \mathrm{TopK}(\{\tilde{q}_t(h)\}, k)\) (default \(k=2\)). Initializing \(S(h) = \emptyset\), candidates are evaluated in descending order of \(\tilde{q}_t(h)\). A candidate \(t\) is accepted into \(S(h)\) if and only if its cosine similarity against every already-accepted module \(j \in S(h)\) satisfies: $\(c_{tj}(h) = \frac{\Delta_t(h)^\top \Delta_j(h)}{\|\Delta_t(h)\|_2 \|\Delta_j(h)\|_2} \ge \gamma\)$ with default threshold \(\gamma = 0\). If no secondary candidate passes the geometric check, the algorithm gracefully falls back to the single highest-demand specialist \(S(h) = \{\arg\max \tilde{q}_t(h)\}\). This design cleanly purges negative vector cancellations while preserving mutually reinforcing complementary directions.
4. Set-Restricted Softmax Composition and State Injection: Enabling Token-Level Dynamic Representation Editing
For all modules admitted into the compatible set \(S(h)\), POLAR applies a normalized set-restricted softmax to compute dynamic mixing weights: $\(\alpha_t(h) = \frac{\exp(\tilde{q}_t(h)/\tau)}{\sum_{j \in S(h)} \exp(\tilde{q}_j(h)/\tau)} \quad (t \in S(h)), \quad \alpha_t(h) = 0 \quad (t \notin S(h))\)$ where \(\tau\) is the softmax temperature hyperparameter (default \(\tau = 1.0\)). The composed edit is then added to the original hidden state at each intervention site \((\ell, p)\): $\(h'_{\ell,p} = h_{\ell,p} + \sum_{t \in S(h_{\ell,p})} \alpha_t(h_{\ell,p}) \Delta_{t,\ell}(h_{\ell,p})\)$ Because routing and composition operate per token position and layer, the model naturally delegates factual grounding to early/middle layers and toxicity suppression to deeper layers, producing flexible, multi-objective aligned responses without retraining.
Loss & Training¶
During the training of individual LoReFT modules, the base model \(\theta\) remains strictly frozen. The optimization minimizes the response-only negative log-likelihood: $\(\min_{\phi_t} - \sum_{(x,y) \in \mathcal{D}_t} \log p_{\theta,\Phi_t}(y \mid x)\)$ The LoReFT subspace rank is configured to \(r=8\), spanning all Transformer layers across the first 7 and last 7 prompt tokens (f7+l7 positioning). The maximum sequence length is 512, with batch size 16, gradient accumulation 2, linear learning rate warmup (ratio 0.1), and AdamW optimizer. Learning rates are set to \(9 \times 10^{-4}\) on Llama3-8B, and \(1.2 \times 10^{-3}\) on Qwen2.5-3B (except \(9 \times 10^{-4}\) for bias). For inference, the total additional parameters across all four modules on Llama3-8B amount to only \(8.39\text{M}\) (\(TL(2rd+r)\), representing \(0.1045\%\) of the backbone), requiring less than half the parameters of LoRA (\(r=32\), \(18.87\text{M}\)).
Key Experimental Results¶
Main Results¶
Evaluation spans two distinct model families (Llama3-8B and Qwen2.5-3B) across four heterogeneous benchmarks: TruthfulQA (multiple-choice accuracy), BBQ (bias mitigation accuracy), ETHICS (commonsense morality accuracy), and RealToxicityPrompts (mean toxicity probability, over-refusal rate, Distinct-2 lexical diversity, and pairwise LLM-judge win rate).
| Model Backbone | Alignment Method | Extra Data | TruthfulQA (Acc ↑) | BBQ (Acc ↑) | ETHICS (Acc ↑) | Mean Tox. (↓) | Over-ref. (↓) | Dist-2 (↑) | Avg. Rank (↓) |
|---|---|---|---|---|---|---|---|---|---|
| Llama3-8B | Base | ✗ | 0.4720 | 0.3352 | 0.3650 | 0.3660 | 0.2066 | 0.3391 | 5.38 |
| RepE (Single-Attr) | ✓ | 0.2546 | 0.3009 | 0.4546 | 0.2618 | 0.0705 | 0.5171 | 6.75 | |
| CAA (Single-Attr) | ✓ | 0.2803 | 0.3016 | 0.4561 | 0.0548 | 0.0924 | 0.3019 | 5.94 | |
| ITI (Single-Attr) | ✓ | 0.2558 | 0.3016 | 0.4602 | 0.2701 | 0.0808 | 0.5109 | 5.69 | |
| LoReFT (Single-Attr) | ✗ | 0.4945 | 0.5551 | 0.9134 | 0.0027 | 0.3658 | 0.1318 | 2.66 | |
| SFT (Multi-Attr) | ✗ | 0.4517 | 0.5543 | 0.8836 | 0.0078 | 0.6000 | 0.1514 | 4.12 | |
| LoRA (Multi-Attr) | ✗ | 0.4418 | 0.5501 | 0.8845 | 0.0027 | 0.5989 | 0.0539 | 4.47 | |
| MAT-Steer (Multi-Attr) | ✓ | 0.2375 | 0.3307 | 0.1891 | 0.1963 | 0.1133 | 0.1750 | 7.50 | |
| POLAR (Ours) | ✗ | 0.5667 | 0.5799 | 0.8831 | 0.0183 | 0.1350 | 0.2509 | 2.50 | |
| Qwen2.5-3B | Base | ✗ | 0.2240 | 0.4901 | 0.5964 | 0.1373 | 0.4036 | 0.3863 | 6.50 |
| RepE (Single-Attr) | ✓ | 0.3917 | 0.5513 | 0.5069 | 0.1483 | 0.1404 | 0.4993 | 5.56 | |
| CAA (Single-Attr) | ✓ | 0.4027 | 0.5568 | 0.5039 | 0.0229 | 0.0401 | 0.3055 | 5.56 | |
| ITI (Single-Attr) | ✓ | 0.3892 | 0.5501 | 0.5064 | 0.1507 | 0.1379 | 0.5056 | 6.31 | |
| LoReFT (Single-Attr) | ✗ | 0.5427 | 0.6458 | 0.7977 | 0.0021 | 0.6598 | 0.1262 | 3.12 | |
| SFT (Multi-Attr) | ✗ | 0.4761 | 0.6495 | 0.7134 | 0.0210 | 0.5607 | 0.1298 | 3.56 | |
| LoRA (Multi-Attr) | ✗ | 0.5080 | 0.6184 | 0.5903 | 0.0058 | 0.1498 | 0.1359 | 3.88 | |
| MAT-Steer (Multi-Attr) | ✓ | 0.3880 | 0.4892 | 0.1967 | 0.1032 | 0.2421 | 0.1938 | 8.19 | |
| POLAR (Ours) | ✗ | 0.5532 | 0.6469 | 0.7954 | 0.0456 | 0.2053 | 0.2310 | 2.31 |
Ablation Study¶
Ablation experiments conducted on Llama3-8B dissect the relative contributions of demand calibration and cosine compatibility filtering.
| Variant | TruthfulQA (↑) | BBQ (↑) | ETHICS (↑) | Mean Tox. (↓) | Avg. Rank (↓) | Mechanism & Diagnostic Effect |
|---|---|---|---|---|---|---|
| compat.+log-z (POLAR Full) | 0.5667 | 0.5799 | 0.8831 | 0.0183 | 2.50 | Optimal trade-off via log-z scaling and cosine filtering |
| compat.+mean (Mean-ratio calibration) | 0.5545 | 0.5789 | 0.8804 | 0.0145 | 3.00 | Scaling by mean alone ignores high inter-layer activation variance |
| w/ equal (Uniform uncalibrated weights) | 0.4957 | 0.5706 | 0.8848 | 0.0749 | 4.50 | Naive unweighted sum leads to severe factual and safety degradation |
| w/ mean only (Mean scaling, no filter) | 0.6071 | 0.5775 | 0.8566 | 0.0384 | 3.75 | Lack of geometric filtering allows conflicting vectors to double toxicity |
| w/ log-z only (Log-z scaling, no filter) | 0.5435 | 0.6159 | 0.8653 | 0.0418 | 4.00 | Opposing vectors are not removed; toxicity jumps from 0.0183 to 0.0418 |
| w/o calib. (No calibration, no filter) | 0.5716 | 0.5497 | 0.8755 | 0.0124 | 3.25 | Modules with naturally high norm dominate unconditionally; BBQ drops |
Key Findings¶
- Superior Multi-Objective Trade-Off: Compared to the strongest multi-attribute baseline (SFT/LoRA), POLAR achieves a 37.4% improvement in average rank, improving TruthfulQA by 8.0 points on average while maintaining a 71.7% non-tie win rate against multi-attribute baselines in toxicity audits.
- Sparse, Surgical Correction by Compatibility Filtering: Under \(k=2\), compatibility filtering modifies only 10.39% of token-level decisions, decreasing the average compatible set size modestly from 2.0 to 1.896. Rejected candidate pairs consistently exhibit negative cosine similarities (e.g., \(-0.0668\) for truthfulness–ethics), proving that filtering surgically eliminates destructive interference without suppressing legitimate activations.
- Minimal Inference Latency and Parameter Footprint: On Llama3-8B, POLAR introduces only an 8.39M parameter footprint (less than half of LoRA's 18.87M). Benchmarked over 500 prompts, POLAR increases median latency (P50) by only 4.5% over the base unaligned model (144.89ms \(\to\) 151.36ms) while achieving 4.3% higher throughput than LoRA.
Highlights & Insights¶
- Disentangled Representation-Space Alignment: By decoupling attribute training into separate low-rank subspaces and combining them at inference time, POLAR sidesteps the gradient conflicts and trade-offs inherent in multi-task parameter fine-tuning.
- Geometric Conflict Management: Demonstrating that high intervention demand does not imply directional alignment, POLAR couples offline log-z normalization with online cosine filtering, establishing an effective blueprint for multi-adapter activation merging.
- Balanced Safety Without Over-Refusal: POLAR reduces over-refusal on toxicity benchmarks from ~60% (in SFT/LoRA) to 13.5% while preserving lexical diversity (Dist-2 of 0.2509 vs. 0.0539 in LoRA), effectively steering away from harmful completions without succumbing to robotic refusals.
Limitations & Future Work¶
- Attribute Library Scalability: The empirical study evaluates compositions of up to five attributes (truthfulness, bias, ethics, toxicity, helpfulness); scaling to dozens of fine-grained behavioral attributes may require hierarchical routing or clustered subspace indexing.
- Static Intervention Sites: POLAR applies interventions across all layers and fixed prompt token positions (f7+l7); developing dynamic layer-selection or token-adaptive sparsity could further boost efficiency.
- Unidirectional Feedback: Intervention modules are trained independently without cross-module feedback during training; exploring cooperative subspace regularizers remains an open research avenue.
Related Work & Insights¶
- vs SFT & LoRA (Ouyang et al., 2022; Hu et al., 2022): Parameter fine-tuning entangles heterogeneous objectives in shared weights, causing optimization trade-offs and excessive refusals; POLAR freezes the base weights and intervenes dynamically in activation space, outperforming joint tuning across all dimensions.
- vs Static Representation Steering (RepE / CAA / ITI): Contrastive vector injection methods extract static global directions that lack input-conditioned adaptability and multi-attribute composability; POLAR utilizes LoReFT subspace rewrites with token-level dynamic routing.
- vs MAT-Steer (Nguyen et al., 2025): MAT-Steer relies on contrastive datasets and confines gated steering to a single layer; POLAR eliminates the need for negative contrastive data, operates across all transformer layers with cross-layer calibration, and provides rigorous geometric conflict filtering.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering analysis of geometric conflicts in multi-attribute activation steering, accompanied by a principled calibration and compatibility filtering architecture.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-model evaluations (Llama3 and Qwen2.5), four heterogeneous benchmarks, five-attribute scaling sweeps, detailed layer-wise traces, and rigorous LLM-judge blind audits.
- Writing Quality: ⭐⭐⭐⭐⭐ Methodical formulation, mathematically grounded derivations, and lucid diagnostic visualizations.
- Value: ⭐⭐⭐⭐⭐ Offers a practical, plug-and-play framework for composing modular alignment capabilities without retraining or modifying foundational LLM parameters.