Find fooled pairs
For each selected attack subcategory, keep training examples on which the original reward model scores the hacked response above the gold response.
Contrastive examplesHARVEHacking-Aware Reward-Head Vector Editing
for Robust Reward Models
1 Carnegie Mellon University2 University of Virginia3 Harvard University4 Stanford University5 University of Michigan6 New Jersey Institute of Technology7 The Chinese University of Hong Kong, Shenzhen
WHEN CONVINCING ANSWERS FOOL THE REWARD MODEL
Find the hacking directions.
Edit the reward head.
HARVE uses matched gold and hacked answers to identify residual-stream directions associated with reward hacking. A geometric edit to the scalar reward head then reduces sensitivity to those directions, without gradient updates or changing the transformer backbone.
At a glance
Reward models can prefer fluent, polished answers that contain a targeted error. The paper introduces RewardHackBench to expose these failures and HARVE to address them. Across eight scalar reward models, editing a subspace of the reward head improves preference for correct answers on targeted attack categories while largely preserving performance elsewhere. The method uses contrastive examples to calculate the edit, but requires no fine-tuning or backpropagation. The full abstract and experiments are in the paper.
REWARDHACKBENCH
A benchmark built around matched attacks.
Each pair contains a gold response and a plausible hacked response to the same query. The held-out test split contains 548 professional-domain and 419 general-purpose pairs.
1,203Matched response pairs967 held out for testing
13Hacking subcategoriesGrouped into five attack families
8Reward models evaluatedModel sizes from 0.6B to 20B
Surface-form mimicryBroken reasoningSycophantic hackingOff-topic answersStyle over substance
Gold-preference rate is the share of pairs on which a reward model scores the gold response above the hacked one; higher is better. Table 1 shows that even models with strong overall scores can fail sharply on particular attack subcategories.
THE METHOD
Three steps. One small edit.
Build a hacking subspace
Average the hacked-minus-gold final hidden-state differences within each selected subcategory. Use SVD to obtain an orthonormal basis for their combined span.
Multiple directionsEdit the reward head
Subtract a scaled projection of the reward-head vector onto that subspace. Select the edit strength on development data, then score with the unchanged backbone.
No gradient updateswrnew = wr − α Phack wr
HARVE targets selected low-performing subcategories for each reward model. The strength α is tuned on a development split under a general-capability preservation constraint; the reported test split and RM-Bench are held out.
EIGHT REWARD MODELS · HELD-OUT RESULTS
Targeted robustness rises across every model.
Compared with the original reward models, HARVE raises gold preference on the target subcategories by 22.4 percentage points on average. It also exceeds the stronger fine-tuning baseline by 15.1 points on average.
+22.4ppTarget vs. original modelAverage across eight reward models
+15.1ppTarget vs. stronger fine-tuningBest of 3:1 and 5:1 data mixtures
+5.3ppRM-Bench Hard vs. originalAverage across eight reward models
Gold-preference rate on targeted subcategories
Table 2 · percent, higher is better| Reward model | Original | Best fine-tune | HARVE |
|---|---|---|---|
| Skywork-V2-Qwen3 · 0.6B | 53.16 | 58.36 | 83.64 |
| InternLM2-reward · 1.8B | 58.93 | 62.50 | 78.57 |
| GRM-Llama-3.2 · 3B | 46.47 | 54.28 | 72.86 |
| Skywork-V2-Llama-3.2 · 3B | 75.09 | 76.90 | 92.42 |
| RM-Mistral · 7B | 67.87 | 84.48 | 90.25 |
| FsfairX-LLaMA3-RM · 8B | 57.49 | 71.50 | 86.96 |
| Skywork-Reward-Llama-3.1 · 8B | 65.94 | 73.55 | 84.42 |
| InternLM2-reward · 20B | 73.19 | 75.36 | 88.41 |
Each reward model has its own selected target subcategories. “Best fine-tune” is the stronger of the two data-augmented fine-tuning baselines for that model. On non-target subcategories, HARVE improves over the original model for six of eight models; full results and trade-offs are in Table 2 of the paper.
WHY MULTIPLE DIRECTIONS MATTER
The signal is broader than surface style.
A separate component ablation tests whether the effect comes from a shared presentation/style direction, category-specific residual directions, or the full joint hacking directions.
Style direction alone
Mean change in target gold preference.
Style-orthogonal residual
Mean change in target gold preference.
Joint hacking directions
Mean change in target gold preference.
These figures come from the component ablation in Table 4 and have a different setup from the main evaluation above. They support the paper’s conclusion that a single superficial cue does not capture the full hacking signal.
INTERPRETING THE RESULT
A score improvement has limits.
In proxy best-of-N selection, HARVE slows the growth of the reward model’s own score gap under stronger search. At N = 256, the gap falls from 9.62 to 3.28 for GRM-Llama-3.2-3B and from 9.26 to 1.84 for Skywork-V2-Llama-3.2-3B.
These gaps use each reward model’s score scale and are not an independent measure of answer quality. The paper reports that judged response quality remains broadly similar across methods in this proxy evaluation.
REFERENCE
Citation
@inproceedings{liu2026harve,
title = {HARVE: Hacking-Aware Reward-Head Vector Editing
for Robust Reward Models},
author = {Liu, Shuang and Bo, Yuxuan and Zhao, Qiuyang and
Huang, Caiyue and Chen, Xiaorong and Liu, Yanguang
and Du, Mengnan},
booktitle = {Proceedings of the 2026 Conference on Empirical Methods
in Natural Language Processing},
year = {2026},
url = {https://arxiv.org/abs/2606.03131}
}