Patterning in Practice: Debiasing Reward Models with Susceptibilities

arXiv:2609.00699v1 Announce Type: new Abstract: Reward models trained on human preferences are known to suffer from length, formatting, and other stylistic biases. In this paper we use patterning, which reweights each preference pair according to its measured effect on posterior expectation values of benchmark losses (its susceptibility), to debias a Gemma 2 9B Instruct reward model trained on Skywork-Reward-Preference v0.2. We obtain +14.2 pm 1.2 pp on RM-Bench Hard, the split where style cues point against correctness (mean pm s.e. over 5 seeds), with overall RM-Bench accuracy preserved, comparable to the strongest Hard-split gain reported by the closest published comparator (SteerRM, +13.2 pp). We demonstrate in a simple case that the reweighting is interpretable by tracing a side effect of the intervention (a regression on a safety subset of RM-Bench) to a small class of training pairs, which we confirm by ablation. The weights also transfer: those computed on Gemma 2 9B debias Gemma 2 2B and 27B with no recomputation, and transfer partially to Llama 3.1 8B. This is the first application of patterning, a program grounded in singular learning theory, beyond small models and synthetic tasks.

Related

• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
• Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization
• Online Distributionally Robust LLM Alignment via Regression to Relative Reward
• Convex Optimization for Alignment and Preference Learning on a Single GPU
• AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals

Source: arXiv cs.LG | 2026-09-02