Reward Models Are Lying to Your Training Pipeline: What Engineering Leaders Need to Know Before Scaling RLHF

Most engineering teams evaluating reward models ask a single question: does it rank good outputs above bad ones? That question is necessary but not sufficient. A reward model can pass every standard benchmark you throw at it and still actively degrade the policy you are trying to train, by assigning meaningfully different scores to outputs that are, in practice, equally good. This is not a theoretical edge case. It is a structural property of how continuous reward models work, and it has direct consequences for any team running RLHF at scale.

The Problem With Continuous Scores

Reward models produce continuous scores because continuous scores seem more informative. A model that can distinguish between a response that scores 0.91 and one that scores 0.87 appears more capable than one that simply says "good" or "bad." The problem is that this apparent precision is often false precision.

When two responses are genuinely equivalent in quality, a reward model that assigns them different scores is not capturing a real signal. It is capturing noise, and your RL training loop has no way to tell the difference. The policy then optimises toward whichever noise the reward model happens to favour, which is the mechanism behind reward hacking in settings where you would not expect it to occur.

Viswanathan et al. (arXiv 2026) formalise this as oversensitivity: the tendency of reward models to assign distinct scores to equally good responses. Their key finding is that a reward model can appear accurate by conventional measures while remaining highly oversensitive. These two properties are separable, which means your current evaluation framework is probably not measuring the thing that matters most.

Discriminative Ability and Specificity: The Evaluation Frame You Are Missing

Discriminative Ability

Discriminative ability measures whether a reward model can reliably distinguish good responses from bad ones. This is what most teams are already measuring, in some form, when they evaluate reward model accuracy. It is a necessary property.

A reward model without discriminative ability is useless. It cannot tell the training signal which direction to move. Most standard benchmarks test for this, and most production reward models pass at a level that gives teams confidence to proceed.

Specificity

Specificity is the complement of oversensitivity. It measures whether a reward model assigns the same score to equally good responses. A high-specificity reward model does not introduce spurious variance between outputs that a human judge would rate identically.

This is the property that most teams do not measure, because standard benchmarks do not surface it. You can have a reward model with strong discriminative ability and poor specificity simultaneously. When that happens, your training pipeline is being shaped by the reward model's noise floor rather than by genuine quality differences (Viswanathan et al., arXiv 2026). The policy learns to satisfy the reward model's idiosyncrasies rather than the underlying objective.

The commercial implication is straightforward: if you have not evaluated your reward model for specificity, you do not know whether your fine-tuning runs are producing better models or just models that are better at exploiting reward noise.

How Oversensitivity Produces Policy Degradation

The mechanism is worth being precise about. In RLHF, the policy receives gradient updates that push it toward higher-reward outputs. If the reward model assigns different scores to equivalent outputs, the gradient signal contains directional information that is not grounded in real quality differences.

Over many training steps, these spurious gradients accumulate. The policy drifts toward whatever surface features the reward model happens to score more highly, rather than toward the underlying quality dimension you are trying to optimise. This is a form of reward hacking that does not look like reward hacking from the outside, because the reward model scores are going up.

The failure is particularly difficult to detect because it does not manifest as a sudden collapse. It manifests as a policy that scores well on your reward model but underperforms on held-out human evaluation or downstream task metrics. By the time you identify the gap, you have already spent compute on a training run that was optimising the wrong thing.

Discretization as a Practical Mitigation

Viswanathan et al. (arXiv 2026) propose a training-free approach: rather than using the continuous output of a reward model directly, apply Monte Carlo dropout to produce discrete reward clusters. The effect is to collapse scores that fall within the same uncertainty band into a single value, removing the spurious precision that drives oversensitivity.

The theoretical result they prove is that discretizations exist which reduce oversensitivity without meaningfully sacrificing discriminative ability. The empirical results across both controlled and natural RL settings show reduced reward hacking and better final policies compared to training on continuous rewards.

For engineering teams, the practical value of this approach is that it requires no retraining of the reward model. You apply it as a post-processing step on an existing model. The tradeoff is that you lose some gradient resolution at the boundary between clusters, but this is a better tradeoff than the alternative of training on noise-contaminated continuous scores.

What This Means for Teams Scaling RLHF

The immediate action is to add specificity as an explicit evaluation criterion during reward model selection. This means constructing evaluation sets that include pairs of outputs which a human judge would rate as equivalent, and measuring whether your reward model assigns them similar scores. If it does not, you have a specificity problem before you have started training.

The second consideration is infrastructure. Discretization changes the shape of the reward signal your training loop receives. Teams using off-the-shelf RLHF frameworks should verify that their pipeline can accommodate discrete or clustered rewards without introducing artefacts at the training step level.

The broader point is that reward model evaluation is a risk management question, not just a model selection question. A reward model with poor specificity does not fail visibly. It produces policies that appear to be improving while quietly drifting away from the objective you care about. Treating discriminative ability and specificity as separate, measurable properties is the first step toward catching that failure before it reaches production.