Does Fine-Tuning Undo Activation Steering? Behavioural Recovery Without Weight-Edit Reversal¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/p-e-w/heretic
Area: Interpretability
Keywords: activation steering, weight embedding, fine-tuning, behavioural recovery, mechanistic stability
TL;DR¶
This paper investigates whether embedding activation steering directly into model weights survives downstream non-adversarial fine-tuning (SFT and RLHF), demonstrating that while fine-tuning can significantly restore steered behaviors (e.g., recovering 69% of ablated refusal under SFT), it reverses less than 0.8% of the weight edit, as the model routes around the intact edit via near-orthogonal weight updates.
Background & Motivation¶
Activation steering has emerged as a lightweight, modular alternative to traditional model retraining, enabling targeted behavioral control by identifying one-dimensional feature directions in activation space and perturbing internal representations. To bypass the runtime overhead and engineering complexity of registering forward hooks during inference, recent advances fold linear projection operators directly into model weights writing to the residual stream (such as attention output and MLP down-projection matrices). This creates pre-aligned models with baked-in behavioral modifications at zero added inference latency. However, open-weight models are rarely deployed as released; they are routinely subjected to downstream post-training and domain-specific fine-tuning by downstream practitioners, who inevitably introduce minor conflicting signals.
This standard deployment workflow poses a fundamental mechanistic and safety question: do weight-embedded linear interventions endure routine downstream adaptation, or are they fragile artifacts that gradient descent immediately washes away? Intuitively, if fine-tuning simply pulls model parameters back along the editing direction toward the pre-intervention manifold, embedded steering would offer little lasting value for pre-deployment safety filtering. Conversely, if the baked-in weight modification remains untouched while the original behavior still resurfaces, the underlying mechanism of post-training adaptation must involve complex computational routing that circumvents parameter-space modifications. While prior security literature has focused heavily on deliberate, adversarial jailbreak fine-tuning, the stability of embedded steering under ordinary, benign adaptation containing incidental opposing optimization pressure remains uncharacterized.
To resolve this question, this paper systematically explores five instruction-tuned models spanning 3B to 14B parameters under both supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), evaluating both negative steering (refusal ablation) and positive steering (brevity amplification). Core idea: by decoupling behavioral preservation from mechanistic weight-space recovery, the study demonstrates that downstream fine-tuning does not erase the embedded weight edit, but instead navigates around it via parameter updates nearly orthogonal to the steering axis, with behavioral degradation governed by the intensity of incidental optimization pressure rather than intrinsic edit fragility.
Method¶
Overall Architecture¶
The experimental pipeline embeds activation steering interventions directly into model parameter tensors, subjects the modified checkpoints to full-parameter downstream fine-tuning, and performs a dual-perspective evaluation across behavioral and weight-space metrics. For each evaluated base architecture, candidate concept directions are isolated in activation space using contrastive prompt distributions. The corresponding linear projection operator is absorbed into the static weights of the residual-writing matrices. The steered models then undergo full-parameter adaptation under SFT (with incidental explicit refusal completions) or RLHF (with sparse reward penalties on compliance). Finally, the checkpoints are analyzed to decouple macro-level behavioral shifts from micro-level parameter perturbations.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Data<br/>Contrastive Prompt Distributions"] --> B["Feature Direction Extraction<br/>Mean Difference in Residual Stream"]
B --> C["Weight Embedding Projection<br/>Absorbing Projection Operator into Weights"]
C --> D["Optimization Pressure Tuning<br/>Full-Parameter SFT and RLHF Adaptation"]
D --> E["Decoupled Dual Evaluation<br/>Behavioral Preservation and Vector Recovery"]
E --> F["Mechanistic Conclusion<br/>Routing Around Without Reversing Edits"]
Key Designs¶
1. Feature Direction Extraction: Contrastive representation probing across layers
Isolating a precise semantic direction in activation space requires neutralizing syntactic and lexical confounds across prompt distributions. For refusal ablation, contrastive datasets \(D^+\) (eliciting harmful requests) and \(D^-\) (benign prompts) are paired; for brevity amplification, prompts with explicit brevity instructions are contrasted with open-ended long-answer requests. The model records residual-stream activations \(h^{(l)}(x)\) at the first generated token across layers \(l\), computing the normalized difference of means vector \(d^{(l)}\):
In full orthogonalization, candidate vectors are chosen by maximizing a logit-based refusal score on held-out prompts while constraining KL divergence on harmless queries. In variable orthogonalization and amplification, the layer index is relaxed to a continuous interpolation parameter optimized alongside trapezoidal per-layer weight schedules via Tree-structured Parzen Estimators (TPE), achieving surgical behavioral control with minimal distortion to general model capabilities.
2. Weight Embedding Projection: Absorbing runtime activation operators into weight matrices
Standard inference-time projection steering projects out direction \(d\) from the activation stream via \(h' = h - d(d^\top h)\), which requires intercepting activations across every layer. Because the residual stream updates from attention output projections (\(W_{out}\)) and MLP down-projections (\(W_{down}\)) are linear mappings (\(h = W_{out}z\)), and the outer product projection operator is associative, the projection transformation can be folded directly into the static weight matrices prior to deployment:
To accommodate partial modulation or positive amplification, the transformation is generalized using an intensity scaling factor \(a \in (0, 1]\) such that \(W'_{out} = W_{out} - a(d d^\top W_{out})\) for ablation and \(W'_{out} = W_{out} + a(d d^\top W_{out})\) for amplification. This zero-cost parameter baking establishes an exact baseline for inspecting whether subsequent gradient updates dismantle the embedded intervention.
3. Optimization Pressure Tuning: Downstream training with incidental opposing signals
Testing intervention durability requires realistic downstream datasets that exert incidental, non-adversarial optimization pressure against the steered behavior. In the SFT setting, models are fine-tuned for 1,024 optimizer steps on a 262k subset of OpenOrca, where roughly 0.1% of training targets contain explicit refusal statements (e.g., "I'm sorry, but I cannot..."), exerting strong localized pressure to restore refusal. In the RLHF setting, models are tuned for ~420 PPO steps on Anthropic hh-rlhf using the Skywork-Reward-V2 reward model, which assigns low scalar rewards to harmful completions, imposing a sparser negative pressure against refusal ablation. All runs employ full-parameter updates across 4 \(\times\) H100 GPUs, creating a challenging setting for weight-space preservation.
4. Decoupled Dual Evaluation: Separating behavioral recovery from weight-space reversal
To formally distinguish whether behavioral reversion stems from parameter erasure or alternative routing pathways, two sets of decoupled metrics are formulated. At the behavioral level, preservation ratio \(P\) quantifies the retained steering effect relative to the base model:
where \(R\) denotes the refusal rate on harmful prompts evaluated by a dedicated LLM judge, and \(L\) denotes average token length on long-answer prompts. At the parameter level, the vector recovery ratio \(\rho\) and full-matrix reversal fraction \(rev\) track whether gradient updates undo the edit:
where \(E = W_{steer} - W_{orig}\) represents the initial weight intervention and \(U = W_{ft} - W_{steer}\) represents the accumulated fine-tuning weight update. A value of \(\rho = 0\) indicates perfect preservation of the projection suppression along \(d\), while \(rev\) measures the direct cancellation of \(E\) by \(U\).
Key Experimental Results¶
Main Results¶
Evaluated across five models (Llama-3.2-3B, Llama-3.1-8B, Llama-3-8B, Qwen3-14B, and SOLAR-10.7B), Table 1 presents refusal rates (\(R\)) and response lengths (\(L\)) alongside post-training preservation scores (\(P\)) under SFT and RLHF.
| Model | Refusal \(R_{base}\) | Refusal \(R_{steer}\) | Refusal \(R_{sft}\) | Refusal \(P_{sft}\) | Length \(L_{base}\) | Length \(L_{steer}\) | Length \(L_{sft}\) | Length \(P_{sft}\) |
|---|---|---|---|---|---|---|---|---|
| Llama-3.2-3B | 0.97 | 0.08 | 0.63 | 0.38 | 82.2 | 53.2 | 53.2 | 1.00 |
| Llama-3.1-8B | 0.96 | 0.01 | 0.51 | 0.47 | 169.8 | 85.1 | 70.2 | 1.18 |
| Llama-3-8B | 0.98 | 0.04 | 0.45 | 0.56 | 129.6 | 76.0 | 64.3 | 1.22 |
| Qwen3-14B | 0.99 | 0.12 | 0.86 | 0.15 | 30.8 | 17.1 | 26.8 | 0.29 |
| SOLAR-10.7B | 0.60 | 0.04 | 0.62 | -0.04 | 126.5 | 72.1 | 128.2 | -0.03 |
| Mean | 0.90 | 0.06 | 0.61 | 0.31 | 107.8 | 60.7 | 68.6 | 0.73 |
Note: Under RLHF, both steering interventions are preserved: refusal ablation yields a mean refusal rate of 0.07 (\(P_{rlhf}=0.98\)), while brevity amplification yields a mean length of 59.4 tokens (\(P_{rlhf}=1.03\)).
Ablation Study¶
Table 2 presents mechanistic measurements on weight updates \(U = W_{ft} - W_{steer}\), including relative projection along direction \(d\) (\(u(d)\)), ratio over 1,000 random unit directions (\(u(d)/u(r)\)), matrix reversal fraction (\(rev\)), and vector recovery ratio (\(\rho\)).
| Experimental Condition | Relative Projection \(u(d)\) | Ratio to Null \(u(d)/u(r)\) | Reversal Fraction \(rev\) (Mean) | Vector Recovery \(\rho\) (Mean) | Max Vector Recovery \(\rho\) |
|---|---|---|---|---|---|
| Refusal Ablation (SFT) | 0.0213 | 2.52ร | 0.00268 (0.27%) | 0.0102 | 0.0186 |
| Refusal Ablation (RLHF) | 0.0071 | 4.06ร | 0.00042 (0.04%) | 0.0053 | 0.0090 |
| Brevity Amplification (SFT) | 0.0135 | 1.65ร | 0.00043 (0.04%) | 0.0004 | 0.0010 |
| Brevity Amplification (RLHF) | 0.0065 | 3.45ร | 0.00009 (0.01%) | 0.0001 | 0.0003 |
Key Findings¶
- Substantial behavioral reversion without weight reversal: Under SFT, refusal ablation loses 69% of its behavioural efficacy (refusal rate rises from 0.06 to 0.61). However, the vector recovery ratio \(\rho\) averages only 0.0102 (maximum 0.0186 across all runs), and the matrix reversal fraction \(rev\) averages only 0.09% (maximum 0.79%).
- Orthogonality of weight updates along the editing direction: While fine-tuning preferentially engages direction \(d\) (2.92\(\times\) higher update magnitude than random null vectors), the component \(d^\top U\) is nearly orthogonal to the pre-edit pattern \(d^\top W_{orig}\) (mean cosine similarity 0.071). Approximately 93% of the update magnitude along \(d\) points away from the vector that would undo the edit.
- Degradation correlates with optimization pressure: SFT contains explicit refusal completions that directly reward declining requests, causing substantial degradation in refusal ablation (\(P_{sft}=0.31\)). In contrast, brevity amplification encounters minimal contradictory data and is well preserved (\(P_{sft}=0.73\)). Under RLHF with sparser opposing signals, both interventions are almost fully preserved (\(P_{rlhf} \ge 0.98\)).
- Lack of correlation between weights and behavior: Across all 20 experimental conditions, behavioral recovery (\(1-P\)) and vector recovery (\(\rho\)) show no statistically significant correlation (\(r = +0.23, p = 0.32\)), confirming that behavioral recovery occurs through alternative computational circuits rather than linear edit reversal.
Highlights & Insights¶
- Debunks the prevailing assumption that behavioral recovery after fine-tuning is driven by gradient descent erasing or reversing parameter-level interventions.
- Introduces an orthogonal geometric perspective on neural network adaptation: overparameterized models preferentially construct alternative routing pathways on orthogonal manifolds rather than undoing localized rank-one weight edits.
- Provides crucial practical guidance for model developers: embedded safety guards cannot be assumed to remain durable after downstream fine-tuning, necessitating behavioral verification even when weight-space signatures remain intact.
Limitations & Future Work¶
- Experiments are restricted to open-weight models up to 14B parameters under full fine-tuning; behavior under parameter-efficient fine-tuning (LoRA), quantization, and frontier-scale LLMs remains to be verified.
- While the work establishes that models route around intact edits, it does not isolate the exact circuit-level pathways or identify whether the recovered behavior relies on newly synthesized or pre-existing subnetworks.
- The concentration of contradictory signals in the training distribution was not systematically titrated to establish precise mathematical decay curves between opposing data fractions and degradation rates.
Related Work & Insights¶
- vs Arditi et al. (2024): Showed that refusal is mediated by a single direction and proposed weight-embedded ablation; this paper investigates the downstream post-training lifecycle of those embedded weights and discovers the routing-around phenomenon.
- vs Yu et al. (2025) & Shairah et al. (2025a): Explored adversarial training that builds distributed pathways to resist steering-based jailbreaks; this work shows that routine, non-adversarial fine-tuning unintentionally achieves a similar bypass effect.
Rating¶
- Novelty: โญโญโญโญโญ Decouples behavioral recovery from weight-edit reversal, revealing an unexpected geometric mechanism behind model post-training adaptation.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous coverage across five model families, two steering targets, two training algorithms, and extensive geometric null models with bootstrap confidence intervals.
- Writing Quality: โญโญโญโญโญ Clear exposition, elegant mathematical formulation, and transparent separation of claims and empirical bounds.
- Value: โญโญโญโญโญ Delivers high-impact insights for AI safety, open-weight deployment governance, and mechanistic interpretability.