Skip to content

Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Safety
Keywords: LLM Safety, Alignment Amplification, Rank-One Weight Editing, Refusal Direction, Jailbreak Defense

TL;DR

This paper introduces RANK-ONE SAFETY INJECTION (ROSI), a fine-tuning-free method that extracts a one-dimensional safety refusal vector via difference-in-means on contrasting instructions and permanently injects it as a rank-one perturbation into residual stream write matrices, significantly cutting jailbreak vulnerability and re-aligning uncensored models while preserving general utility.

Background & Motivation

Large language models possess impressive generality across open-domain tasks, but pre-training on vast uncurated web corpora inevitably endows them with the latent capability to generate harmful and dangerous content. Although post-training alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) have become the de facto standard for behavioral steering, aligned models remain fragile against an expanding arsenal of jailbreak vectors (e.g., DAN, adversarial prompt suffixes, multi-turn role-play) and downstream fine-tuning on even completely benign data.

Recent advances in mechanistic interpretability reveal that refusal behavior in decoder-only transformers is surprisingly low-dimensional, mediated predominantly by a single linear direction in the activation space. Adversarial work like representation abliteration demonstrated that projecting out this one-dimensional refusal direction from model weights trivially neutralizes safety guardrails, producing uncensored models. Conversely, defensive attempts to reinforce safety typically lean on compute-intensive adversarial fine-tuning or inference-time activation steering, which respectively suffer from catastrophic utility degradation and runtime inference overhead.

Inspired by these findings, this work inverts the abliteration vulnerability: rather than ablating the refusal subspace to strip safety, can we systematically and permanently inject this direction into model weights? Core idea: extract a one-dimensional safety direction from a small set of contrasting harmful/harmless instructions and apply a lightweight, fine-tuning-free rank-one weight injection directly to all residual stream write projection matrices (\(W_{\text{out}}\)), permanently amplifying refusal robustness and re-aligning uncensored models with negligible side effects.

Method

Overall Architecture

ROSI operates in two main phases: safety direction extraction in activation space, followed by rank-one weight editing in parameter space. First, using a small set of harmful and harmless instruction pairs, the model computes residual activations across candidate layers to isolate the mean difference vector at the prompt-final token position; for uncensored models that lack native refusal tendencies, a temporary safety system prompt is prepended during the forward pass to elicit distinct geometric separation. Second, the extracted normalized safety direction is combined with the row-mean vector of each residual write projection matrix to form a rank-one update matrix, which is permanently added to the original weights without gradient-based training.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Contrasting Instruction Pairs<br/>(Harmful & Harmless Sets)"] --> B["Safety Direction Extraction<br/>Activation difference-in-means & norm"]
    B --> C["Weight Mean Projection<br/>Outer product with row-wise weight mean"]
    C --> D["Rank-One Weight Injection<br/>Additive update to Attn and MLP write matrices"]
    D --> E["ROSI Model<br/>Permanently amplified refusal & preserved utility"]

Key Designs

1. Safety Direction Extraction: Isolating Causal Refusal Subspace via Difference-in-Means

To identify the specific linear direction mediating refusal, ROSI leverages the difference-in-means formulation over two small, contrasting datasets: harmful instructions \(\mathcal{D}_{\text{harmful}}\) and harmless instructions \(\mathcal{D}_{\text{harmless}}\). Forward passes are executed to record the residual stream activations \(x_i^{(l)}\) at the final prompt token position across candidate layers \(l\). Computing the empirical means \(\mu^{(l)}\) and \(\nu^{(l)}\) yields the raw direction vector:

\[s^{(l)} = \mu^{(l)} - \nu^{(l)} = \frac{1}{|\mathcal{D}_{\text{harmful}}|} \sum_{t \in \mathcal{D}_{\text{harmful}}} x_i^{(l)}(t) - \frac{1}{|\mathcal{D}_{\text{harmless}}|} \sum_{t \in \mathcal{D}_{\text{harmless}}} x_i^{(l)}(t)\]

The optimal extraction layer \(l^*\) is identified on a small validation set by selecting the layer that maximizes the harm refusal rate while maintaining a KL-divergence of \(\le 0.1\) on benign inputs to prevent over-refusal. The chosen vector is then L2-normalized into unit direction \(\hat{s} \in \mathbb{R}^{d_{\text{model}}}\). Empirically, as few as 50 instruction pairsโ€”and even a single contrasting pairโ€”suffice to isolate an effective direction.

2. Rank-One Weight Injection: Parameter-Space Hardening Without Fine-Tuning

Rather than ablating \(\hat{s}\) via projection \(W' \leftarrow (I - \hat{s}\hat{s}^\top)W\), ROSI permanently strengthens it by editing every projection matrix that writes into the residual streamโ€”specifically, the attention output projection \(W_O\) and MLP output projection \(W_{\text{out}} \in \mathbb{R}^{d_{\text{model}} \times d_{\text{input}}}\). The update rule is formulated as a clean rank-one outer product:

\[W_{\text{out}}' = W_{\text{out}} + \alpha \cdot \hat{s} \cdot \bar{w}^\top\]

Here, \(\alpha\) is a scalar hyperparameter governing injection magnitude, and \(\bar{w} \in \mathbb{R}^{d_{\text{input}}}\) denotes the mean of the row vectors of the original weight matrix \(W_{\text{out}}\). Because \(\bar{w}^\top\) approximates the expected input projection baseline, the update term \(\alpha(\hat{s}\bar{w}^\top)\) imparts a persistent bias along \(\hat{s}\) upon every forward token write, permanently steering intermediate representations toward the refusal boundary without any runtime steering overhead or modified inference kernels.

3. Prompt-Conditioned Re-Alignment: Resurrecting Latent Safety in Uncensored Models

When applied to uncensored models (such as the Dolphin series) whose safety alignment was intentionally ablated during fine-tuning, standard activation subtraction fails because the baseline model yields almost no refusal signals. ROSI resolves this by prepending a explicit safety system prompt during the forward extraction pass, commanding the model to reject harmful tasks. This system prompt cleanly disentangles harmful and harmless activation clusters (demonstrated via increased cluster centroid distance in PCA projections). Once the latent safety vector is captured and injected into the weights, the system prompt is discarded, and the modified model retains strong refusal behavior autonomously.

Loss & Training

ROSI is entirely training-free and requires no gradient descent or loss optimization. Calibration is limited to tuning the injection scalar \(\alpha\) and selecting the target layer set. Layer ablation experiments show that injecting into a contiguous window comprising 30%โ€“40% of layers centered around the extraction layer achieves near-optimal refusal gains, while applying the update across all residual stream write projections delivers consistent cross-architecture robustness.

Key Experimental Results

Main Results

ROSI was evaluated across eight open-weight aligned models on CATQA (550 harmful prompts across 11 categories judged by Llama Guard 3) for Harm Refusal (HR %), Alpaca (512 benign prompts) for Benign Compliance (BC %), and four major jailbreak attack suites (DAN, HarmBench, WildGuardTest, and WildJailbreak).

Model Method Harm Refusal (HR โ†‘) Benign Compliance (BC โ†‘) DAN ASR (โ†“) HarmBench ASR (โ†“)
Gemma-2B-Instruct Baseline
+ ROSI
98.4%
99.8% (+1.5)
99.4%
99.0% (-0.4)
5.3%
1.0% (-4.3)
42.3%
8.2% (-34.1)
Llama-2-7B-Chat Baseline
+ ROSI
99.8%
100.0% (+0.2)
98.8%
99.8% (+1.0)
0.0%
0.0% (0.0)
3.5%
0.1% (-3.4)
Llama-3.1-8B-Instruct Baseline
+ ROSI
98.2%
99.1% (+0.9)
99.6%
99.6% (0.0)
0.3%
0.0% (-0.3)
14.8%
1.8% (-13.0)
Llama-3.2-1B-Instruct Baseline
+ ROSI
79.5%
92.7% (+13.2)
99.2%
95.9% (-3.3)
1.3%
0.0% (-1.3)
18.7%
7.5% (-11.2)
Qwen2.5-3B-Instruct Baseline
+ ROSI
89.8%
99.6% (+9.8)
99.6%
98.6% (-1.0)
52.7%
6.7% (-46.0)
93.7%
61.5% (-32.2)
Qwen2.5-7B-Instruct Baseline
+ ROSI
95.8%
100.0% (+4.2)
100.0%
99.0% (-1.0)
40.3%
11.7% (-28.6)
90.7%
36.7% (-54.0)
Qwen2.5-14B-Instruct Baseline
+ ROSI
98.9%
100.0% (+1.1)
100.0%
99.4% (-0.6)
32.3%
5.0% (-27.3)
81.2%
43.9% (-37.3)
Yi-6B-Chat Baseline
+ ROSI
81.3%
99.5% (+18.2)
99.6%
97.7% (-1.9)
52.0%
15.3% (-36.7)
89.4%
44.6% (-44.8)

General capabilities were tested on six benchmarks (MMLU, HellaSwag, ARC-Easy, ARC-Challenge, BoolQ, TruthfulQA). Across all models, the average benchmark score difference remained within \(\pm 0.5\%\), confirming that the rank-one safety direction is largely orthogonal to factual knowledge and reasoning capabilities.

Ablation Study

Ablations investigated uncensored model re-alignment, vaccination against downstream fine-tuning degradation, cross-model transfer, and data sample efficiency.

Model / Setting Configuration Variant Harm Refusal (HR %) Key Robustness / Utility Notes
Dolphin3.0-Qwen2.5-3B Baseline (Uncensored)
ROSI (with safety prompt)
ROSI (โข without safety prompt)
50.0%
86.0% (+36.0)
33.6% (-16.4)
BC 99.6%; DAN ASR drops to 44.0%
Without prompt, overlapping activations yield negative transfer
Dolphin3.0-Llama3.1-8B Baseline (Uncensored)
ROSI (with safety prompt)
ROSI (โข without safety prompt)
65.8%
100.0% (+34.2)
88.9% (+23.1)
Full re-alignment to 100.0% safety with 100.0% BC maintained
Llama-2-7B-Chat (Dolly 15K FT) Base (No FT)
LoRA FT
SafeLoRA
LoRA \(\to\) ROSI (post-hoc)
ROSI \(\to\) LoRA (vaccination)
ROSI \(\to\) SafeLoRA (joint)
99.8%
82.7%
95.5%
85.5%
98.6%
100.0%
Standard FT causes catastrophic safety drop (-17.1%)
Post-hoc injection cannot fully recover broken manifolds
Pre-injection immunizes model against fine-tuning erosion
Joint defense achieves complete 100.0% safety preservation
Qwen2.5-3B-Instruct (Sample Efficiency) Base
ROSI (1 pair)
ROSI (10 pairs)
ROSI (50 pairs)
ROSI (100 pairs)
89.8%
99.1%
96.5%
99.5%
98.6%
A single instruction pair elevates refusal to 99.1%
Validates extreme dominance of refusal direction in representation space

Key Findings

  • Amplifying Vulnerable Models: ROSI provides dramatic improvements for models with weaker baseline alignment, raising HR by +18.2% on Yi-6B-Chat and +13.2% on Llama-3.2-1B-Instruct, while slashing HarmBench jailbreak success rates by 30 to 50 percentage points.
  • Uncensored Model Recovery: Uncensored models retain latent safety circuits; prompt-conditioned extraction allows ROSI to reinstate refusal guardrails (achieving 100.0% HR on Dolphin3.0-Llama3.1-8B) without expensive re-training.
  • Safety Vaccination Mechanism: Injecting the ROSI vector prior to downstream parameter-efficient fine-tuning (ROSI \(\to\) LoRA) shields against safety erosion (98.6% vs. 82.7%), and combining it with SafeLoRA attains 100.0% refusal retention.
  • Cross-Lingual & Cross-Model Transfer: Safety directions extracted from English prompts generalize effectively to Arabic tests. Furthermore, between models sharing identical hidden dimensions (\(d_{\text{model}}\)), transferring the safety vector from Qwen2.5-32B into Qwen2.5-14B yields even stronger defense than the 14B model's own vector.

Highlights & Insights

  • Inverting Abliteration into Construction: Translates mechanistic insights from adversarial attacks (direction erasure) into an elegant, low-cost defensive intervention that directly hardens model weights.
  • Ultra-Lightweight Rank-One Parameter Update: Avoids matrix inversions, optimization loops, and gradient propagation; injects a simple outer product perturbation in seconds.
  • Zero Runtime Overhead: Unlike test-time activation steering or external guardrail models, ROSI requires no customized inference runtime, extra latency, or memory overhead.

Limitations & Future Work

  • Strictly Rank-One Assumption: The study restricts updates to rank-one perturbations; higher-rank updates or subspace modeling may be required to counter complex multi-turn or compositional attack vectors.
  • Complementary Rather than Standalone: While ROSI substantially curtails automated jailbreaks, it does not guarantee absolute invulnerability and should be combined with input filtering and external guardrails in deployment.
  • Slight Benign Degradation on Small Models: Models with \(\le 1\text{B}\) parameters exhibit minor benign compliance drops (e.g., -3.3% on Llama-3.2-1B), highlighting the difficulty of cleanly separating safety and utility in constrained capacity regimes.
  • vs. Arditi et al. (2024) (Abliteration): While Arditi et al. use orthogonal projection \((I - \hat{s}\hat{s}^\top)W\) to remove refusal, ROSI inverts this paradigm by adding \(\alpha \hat{s}\bar{w}^\top\) to reinforce the safety direction.
  • vs. SafeLoRA (Hsu et al., 2025): SafeLoRA projects adapter weights during adaptation; ROSI acts as an upstream initialization vaccine, with ROSI \(\to\) SafeLoRA achieving flawless 100.0% retention against fine-tuning degradation.
  • vs. Test-Time Activation Steering (AdaSteer / SafeSteer): Activation steering incurs runtime computational overhead and requires customized serving infrastructure; ROSI modifies static weights once with zero inference overhead.

Rating

  • Novelty: โญโญโญโญโญ Inverts mechanistic representation ablation into a simple, elegant rank-one static weight injection defense.
  • Experimental Thoroughness: โญโญโญโญโญ Evaluated across 12 diverse models (8 aligned, 4 uncensored), multiple jailbreak benchmarks, 6 utility tasks, and fine-tuning vaccination scenarios.
  • Writing Quality: โญโญโญโญโญ Clean narrative structure, crisp mathematical formulation, and strong conceptual cohesion with mechanistic interpretability.
  • Value: โญโญโญโญโญ Offers a highly practical, compute-free tool for open-source model safety hardening and post-hoc un-censoring defense.