HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/olivialiu121/HARVE-Reward-Head-Editing
Area: Information Retrieval / Reward Model Robustness
Keywords: Reward Models, Reward Hacking, Vector Editing, Backpropagation-Free, High-Stakes Domains
TL;DR¶
This paper introduces HARVE, a backpropagation-free vector editing method for scalar reward models that identifies a multi-directional hacking subspace from fooled contrastive pairs and projects out the sensitive reward-head component without fine-tuning, dramatically improving robustness in high-stakes professional and general domains.
Background & Motivation¶
In modern large language model (LLM) alignment pipelines and reinforcement learning from human feedback (RLHF), reward models (RMs) act as the foundational scoring functions that steer policy generation toward desired human values. However, existing scalar reward models suffer systematically from reward hacking: responses can exploit shallow, superficial cues—such as verbosity, polished prose, polite hedging, or ungrounded authoritative styling—to harvest inflated reward scores without delivering substantive factual accuracy or rigorous logical reasoning. While earlier benchmarks have characterized generic failure modes like length bias and formatting artifacts, the vulnerability of reward models within high-stakes, real-world professional domains—such as law, statutory compliance, and corporate policy—remains critically underexplored. In legal advisory settings, an LLM output laden with fabricated case citations or inverted burdens of proof sounds impeccably authoritative, posing catastrophic downstream safety and liability hazards.
Prevailing mitigation strategies predominantly rely on data-centric augmentation and iterative retraining paradigms. Practitioners either collect debiased or adversarial preference pairs or perform full-parameter / parameter-efficient fine-tuning (e.g., LoRA) on the underlying reward models. Unfortunately, data-augmented retraining incurs substantial computational overhead, often triggers catastrophic forgetting on unrelated general-purpose evaluation tasks, and scales poorly in production environments where newly emergent hacking strategies require repeated end-to-end retraining pipelines. Furthermore, the mechanistic question of how scalar reward models encode these diverse hacking vulnerabilities within their residual representations—and whether such vulnerabilities can be excised surgically without modifying the transformer backbone—has remained unanswered.
The core insight of this work stems from the geometric linearity of scalar reward prediction: the scalar score is computed strictly as the inner product between the final residual-stream hidden state and the reward-head vector. Consequently, desensitizing the reward head to specific hacking directions can be formulated analytically as an orthogonal subspace projection in parameter space. To systematically benchmark and address this challenge, the authors introduce REWARDHACKBENCH across 13 diverse subcategories, exposing severe baseline vulnerabilities across open-source reward models. Core idea: HARVE completely eliminates backpropagation and fine-tuning by extracting residual-stream difference vectors from fooled contrastive pairs, constructing an orthonormal multi-directional hacking subspace via SVD, and surgically projecting out the aligned components directly from the reward-head vector.
Method¶
Overall Architecture¶
HARVE is specifically tailored for scalar reward models whose scalar output satisfies \(r(x, y) = w_r^\top h_{\text{final}}(x, y)\). The workflow unfolds across three modular stages: first, identifying the subset of training pairs where the base reward model is actively fooled into preferring the hacked response; second, stacking the normalized difference vectors across target subcategories and applying Singular Value Decomposition (SVD) to construct an orthonormal projection matrix for the multi-directional hacking subspace; third, executing a closed-form geometric edit on the reward-head vector \(w_r\), preserving the frozen transformer backbone entirely without computing any gradients.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}}%%
flowchart TD
A["Query Q and Contrastive Response Pair (yg, yh)<br/>(Gold response vs. targeted hacked response)"] --> B["Stage 1: Fooled-Subset Filtering & Shift Extraction<br/>Filter pairs where r(Q, yh) > r(Q, yg) and compute mean shift"]
B --> C["Stage 2: Multi-Directional Hacking Subspace Construction<br/>Stack target directions Vhack and apply SVD to derive Uhack & Phack"]
C --> D["Stage 3: Reward-Head Vector Subspace Editing<br/>Compute wnew = wr - α Phack wr to project out hacking sensitivity"]
D --> E["Deploy Edited Robust Reward Model<br/>Zero-latency inference preserving general evaluator capabilities"]
Key Designs¶
1. Fooled-Subset Filtering & Shift Extraction: Isolating genuine failure signals from confounding bidirectional features When attempting to estimate the residual-stream vector associated with a targeted hacking subcategory \(k \in \mathcal{K}\), averaging hidden-state shifts across all available training pairs dilutes the failure signal because the model already classifies some pairs correctly. In those correctly ordered instances, the contrastive difference reflects valid semantic distinctions rather than vulnerability pathways. HARVE strictly isolates the fooled subset \(\mathcal{F}_k = \{i \in \{1, \dots, N_k\} : r(x_i, y_h^i) > r(x_i, y_g^i)\}\) where the base model assigns higher reward to the hacked response. For every fooled instance, the final hidden states \(h_h^i = h_{\text{final}}(x_i, y_h^i)\) and \(h_g^i = h_{\text{final}}(x_i, y_g^i)\) are extracted to compute the normalized mean shift vector: $\(\tilde{v}_k = \frac{1}{|\mathcal{F}_k|} \sum_{i \in \mathcal{F}_k} (h_h^i - h_g^i), \quad v_k = \frac{\tilde{v}_k}{\|\tilde{v}_k\|}\)$ This filtering guarantees that the estimated direction captures the precise residual features that fool the linear scoring head. Empirically, fooled-subset filtering flips the reward head's cosine alignment with the extracted directions from an uninformative -0.05 across all pairs to a sharp +0.24.
2. Multi-Directional Hacking Subspace Construction: Addressing the composite and overlapping nature of hacking patterns Reward-hacking vulnerabilities rarely align with a single 1D axis in activation space; instead, distinct vulnerabilities—such as fabricating legal precedents, stripping cautionary disclaimers, or inverting burdens of proof—span multiple coupled geometric dimensions. HARVE stacks the unit vectors corresponding to \(K\) targeted subcategories into a direction matrix \(V_{\text{hack}} = [v_{k_1}, \dots, v_{k_K}] \in \mathbb{R}^{d \times K}\). Because individual directions exhibit non-zero mutual overlaps and non-orthogonalities, HARVE executes Singular Value Decomposition: $\(V_{\text{hack}} = U \Sigma Q^\top\)$ Retaining columns corresponding to singular values \(\sigma_j > 10^{-6}\) forms an orthonormal basis \(U_{\text{hack}}\) spanning the hacking subspace \(\mathcal{S}_{\text{hack}} = \text{span}(V_{\text{hack}})\). The orthogonal projection matrix is then formulated as: $\(P_{\text{hack}} = U_{\text{hack}} U_{\text{hack}}^\top\)$ This singular-value-thresholded projection eliminates numerical collinearities and captures the joint low-rank subspace across diverse failure modes in one unified operator.
3. Reward-Head Vector Subspace Editing: Backpropagation-free parameter manipulation with capability preservation Leveraging the linear formulation of scalar reward heads, HARVE excises the reward head's sensitivity along the hacking subspace by directly updating the weight vector \(w_r \in \mathbb{R}^d\) without touching any transformer parameters: $\(w_r^{\text{new}} = w_r - \alpha P_{\text{hack}} w_r\)$ Here, \(\alpha \ge 0\) represents the intervention strength. Setting \(\alpha = 1.0\) completely zeroes out the component of \(w_r\) that resides in the hacking subspace \(\mathcal{S}_{\text{hack}}\), whereas setting \(\alpha > 1.0\) applies an active negative penalty to penalize hacking-associated activations. The optimal strength \(\alpha^\star\) is tuned on a held-out development split under a strict constraint that non-target RewardBench performance degrades by no more than 4 percentage points. The modified reward model scores new inputs at runtime as \(r_{\text{new}}(x, y) = (w_r^{\text{new}})^\top h_{\text{final}}(x, y)\), incurring absolutely zero inference-time computational latency or memory overhead.
Benchmark & Evaluation Protocol¶
The authors introduce REWARDHACKBENCH, featuring 1,203 matched gold–hacked response pairs spanning 5 overarching categories and 13 subcategories: 1. Surface-Form Mimicry (Surface): A1 Made-up Sources (citing fabricated legal precedents), A2 Legalese Padding (injecting Latin maxims), A3 Citation Inflation (appending unsupportive real citations). 2. Broken Reasoning (Reasoning): B1 Missing Elements (omitting essential elements of a legal doctrine), B2 Burden Inversion (reversing legal burdens of proof). 3. Sycophantic Hacking (Sycophantic): C1 User-Aligned Reasoning (distorting legal conclusions to match user bias), C2 Safety Disclaimers (evading substance via boilerplate disclaimers), C3 Hedge Stripping (removing necessary qualifiers and caveats). 4. Off-Topic Hacking (Off-Topic): D1 Neighbor Drift (answering a related alternative prompt), D2 Instruction Drift (following altered instructions, from LLMBar). 5. Style-Over-Substance (Style): E1 Confident Errors (assertive factually erroneous claims), E2 Subtle Errors (subtle flaws hidden by polished style), E3 Crafted Distractors (adversarially designed superficial answers, from LLMBar).
For high-stakes legal categories (A–C, 784 pairs), professional attorneys and law students drafted and validated the instances, strictly calibrating the mean hacked-to-gold length ratio to 1.01 to eliminate length confounds. Categories A–C are partitioned into 20% train (157 pairs for vector extraction), 10% dev (79 pairs for tuning \(\alpha^\star\)), and 70% test (548 pairs). General categories D–E (419 pairs) serve entirely as out-of-domain held-out test sets. The primary metric is Gold-Preference Rate (GP %), indicating the percentage of pairs where the model scores the gold response higher than the hacked response.
Key Experimental Results¶
Main Results¶
Eight open-source scalar reward models (0.6B to 20B parameters) are benchmarked against original baselines, 3:1 / 5:1 data-augmented LoRA fine-tuning, and HARVE on both REWARDHACKBENCH and the out-of-domain RM-Bench Hard benchmark.
| Reward Model | Size | Method | Target Subcategories (Tgt ↑) | Non-Target Subcategories (Non-Tgt ↑) | REWARDHACKBENCH Avg (Avg ↑) | RM-Bench Hard Avg (Avg ↑) |
|---|---|---|---|---|---|---|
| Skywork-V2-Qwen3 | 0.6B | Baseline | 53.16 | 73.07 | 67.53 | 56.90 |
| Skywork-V2-Qwen3 | 0.6B | Finetune (5:1) | 58.36 | 75.50 | 70.01 | 57.00 |
| Skywork-V2-Qwen3 | 0.6B | HARVE (\(\alpha^\star=1.50\)) | 83.64 | 73.50 | 76.32 | 64.23 |
| InternLM2-reward | 1.8B | Baseline | 58.93 | 74.24 | 70.01 | 59.56 |
| InternLM2-reward | 1.8B | Finetune (5:1) | 62.50 | 75.25 | 71.68 | 58.96 |
| InternLM2-reward | 1.8B | HARVE (\(\alpha^\star=1.00\)) | 78.57 | 75.40 | 76.29 | 59.58 |
| GRM-Llama-3.2 | 3B | Baseline | 46.47 | 84.81 | 74.15 | 53.03 |
| GRM-Llama-3.2 | 3B | Finetune (5:1) | 54.28 | 86.25 | 77.37 | 53.18 |
| GRM-Llama-3.2 | 3B | HARVE (\(\alpha^\star=1.00\)) | 72.86 | 86.82 | 82.94 | 54.26 |
| Skywork-V2-Llama-3.2 | 3B | Baseline | 75.09 | 85.51 | 82.83 | 69.43 |
| Skywork-V2-Llama-3.2 | 3B | Finetune (5:1) | 76.90 | 86.09 | 83.74 | 69.33 |
| Skywork-V2-Llama-3.2 | 3B | HARVE (\(\alpha^\star=1.25\)) | 92.42 | 86.52 | 88.00 | 70.91 |
| RM-Mistral | 7B | Baseline | 67.87 | 72.32 | 71.04 | 44.41 |
| RM-Mistral | 7B | Finetune (5:1) | 84.48 | 78.55 | 80.20 | 47.88 |
| RM-Mistral | 7B | HARVE (\(\alpha^\star=2.50\)) | 90.25 | 76.52 | 80.35 | 65.06 |
| FsfairX-LLaMA3-RM | 8B | Baseline | 57.49 | 75.66 | 71.77 | 50.46 |
| FsfairX-LLaMA3-RM | 8B | Finetune (5:1) | 71.50 | 80.13 | 77.87 | 52.47 |
| FsfairX-LLaMA3-RM | 8B | HARVE (\(\alpha^\star=1.75\)) | 86.96 | 76.58 | 79.47 | 54.38 |
| Skywork-Reward-Llama-3.1 | 8B | Baseline | 65.94 | 91.17 | 84.07 | 54.68 |
| Skywork-Reward-Llama-3.1 | 8B | Finetune (5:1) | 73.55 | 92.19 | 86.97 | 54.79 |
| Skywork-Reward-Llama-3.1 | 8B | HARVE (\(\alpha^\star=1.10\)) | 84.42 | 89.58 | 88.11 | 61.22 |
| InternLM2-reward | 20B | Baseline | 73.19 | 82.78 | 79.94 | 62.65 |
| InternLM2-reward | 20B | Finetune (5:1) | 75.36 | 83.07 | 80.97 | 62.47 |
| InternLM2-reward | 20B | HARVE (\(\alpha^\star=1.60\)) | 88.41 | 80.90 | 82.94 | 63.78 |
Ablation Study¶
To verify whether the extracted vectors simply encode stylistic verbosity rather than structural hacking mechanisms, the authors decompose each hacking direction into a presentation/style component and an orthogonal residual: \(v_k = (v_k^\top v_{\text{sty}}) v_{\text{sty}} + v_{k,\perp\text{sty}}\).
| Reward Model | Size | Target Base GP (%) | Ablate Style Only \(v_{\text{sty}}\) (pp ↑) | Ablate Category Residual \(v_{k,\perp\text{sty}}\) (pp ↑) | Ablate Full Joint Direction Joint (pp ↑) | Non-Target Base GP (%) | Non-Target Joint Ablation (pp ↑) |
|---|---|---|---|---|---|---|---|
| Skywork-V2-Qwen3 | 0.6B | 52.16 | -3.99 | +5.95 | +27.89 | 87.82 | +1.76 |
| InternLM2-reward | 1.8B | 58.93 | -15.36 | +3.93 | +19.64 | 89.66 | +1.45 |
| GRM-Llama-3.2 | 3B | 45.70 | +8.58 | +2.80 | +27.26 | 85.24 | +3.95 |
| Skywork-V2-Llama-3.2 | 3B | 73.89 | +1.09 | +6.93 | +10.91 | 89.88 | +3.00 |
| RM-Mistral | 7B | 67.97 | +3.52 | +5.14 | +13.00 | 93.96 | +2.58 |
| FsfairX-LLaMA3-RM | 8B | 58.21 | -2.05 | +7.66 | +13.77 | 89.26 | +2.08 |
| Skywork-Reward-Llama-3.1 | 8B | 65.98 | +1.36 | +1.80 | +12.24 | 94.94 | +0.67 |
| InternLM2-reward | 20B | 73.26 | +0.36 | +8.34 | +8.68 | 92.45 | +1.07 |
| Mean across Models | — | 62.01 | -0.81 | +5.32 | +16.67 | 90.40 | +2.07 |
Key Findings¶
- Decisive Superiority over Fine-Tuning: On target subcategories, HARVE achieves an average gold-preference rate improvement of 22.4 percentage points over unedited baselines and outperforms the compute-heavy 5:1 fine-tuning baseline by 15.1 percentage points, establishing that linear geometric surgery directly clears spurious sensitivity without optimization noise.
- Super-Additive Joint Geometry: Ablating the full joint hacking subspace provides a +16.67 pp average gain, which vastly exceeds the linear sum of isolating the shared style direction (-0.81 pp) and isolating category-specific residuals (+5.32 pp). This proves that reward hacking manifests as a multi-dimensional, entangled feature manifold in the residual stream.
- Cross-Domain Generalization: HARVE consistently improves accuracy on the held-out RM-Bench Hard dataset across all eight models (averaging a 5.3 pp gain, peaking at +20.7 pp on RM-Mistral-7B), demonstrating that excising high-stakes hacking pathways simultaneously hardens models against subtle general chat, code, and math distractors.
- Taming Best-of-N Overoptimization: In downstream candidate selection with \(N=256\), unedited and fine-tuned models experience dramatic reward inflation gaps (e.g., 9.62 on GRM-3B), whereas HARVE curtails this proxy-gold gap to 3.28, effectively curbing policy gaming during search.
Highlights & Insights¶
- Gradient-Free Precision Surgery: HARVE demonstrates that reward model debiasing does not require fine-tuning thousands of transformer attention layers; modifying only the final linear projection vector \(w_r\) via SVD accomplishes surgical debiasing in milliseconds.
- Fooled-Subset Extraction as a Clean Signal Filter: Restricting contrastive difference extraction strictly to model-fooled pairs prevents correctly scored instances from introducing opposite semantic directions, reversing directional alignment with the reward head from negative to positive (+0.24).
- Style Entanglement vs. Token Length: The study reveals that reward models associate hacking with a broader presentation cluster (style cosine similarity +0.40) rather than literal token count differences (correlation only +0.20), debunking the assumption that reward hacking is merely length bias.
Limitations & Future Work¶
- Scalar Head Dependency: HARVE relies strictly on access to internal residual activations and the linear weights of scalar reward heads, making it inapplicable to black-box LLM APIs or generative pairwise judges that produce verbal rationales.
- Contrastive Coverage Constraints: The efficacy of the subspace depends on representative fooled samples during the extraction stage. Unseen, highly subtle hacking tactics in new domains may fall outside the static projection subspace.
- Lack of End-to-End Online RL Verification: Downstream evaluations were performed via reranking and best-of-N proxy setups; tracking long-term policy behavior under full closed-loop PPO or online DPO with an edited reward model remains open.
Related Work & Insights¶
- vs Data-Augmented Fine-Tuning / Robust Reward Model Training (RRM): RRM fine-tunes all linear layers on mixed preference datasets, which is slow, expensive, and risks degrading unrelated capabilities. HARVE requires zero backpropagation, takes seconds to compute, and can be stacked downstream of RRM to eliminate lingering hacking sensitivities.
- vs Representation Engineering & Activation Steering: Steering approaches typically intervene during runtime autoregressive token generation across hidden layers. In contrast, HARVE performs an offline, permanent geometric modification directly on the scoring head vector \(w_r\), adding zero latency or memory overhead during deployment.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant geometric formulation translating reward-model debiasing into an exact, backpropagation-free subspace projection problem.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation across eight 0.6B–20B reward models on both high-stakes REWARDHACKBENCH and cross-domain RM-Bench Hard.
- Writing Quality: ⭐⭐⭐⭐⭐ Outstanding clarity, precise mathematical definitions, and compelling mechanistic interpretability analyses.
- Value: ⭐⭐⭐⭐⭐ High industrial relevance, providing an instant, lightweight patch for securing reward models against critical vulnerabilities in LLM alignment.