Online Distributionally Robust LLM Alignment via Regression to Relative Reward

arXiv:2509.19104v2 Announce Type: replace Abstract: Reinforcement Learning with Human Feedback (RLHF) has become crucial for aligning Large Language Models (LLMs) with human intent. However, existing offline RLHF approaches suffer from overoptimization, where language models degrade by overfitting inaccuracies and drifting from preferred behaviors observed during training. Distributionally robust optimization (DRO) is a natural solution, but existing DRO-DPO methods are sample-inefficient, ignore heterogeneous preferences, and lean on brittle heuristics. We introduce DRO-REBEL, a family of robust online REBEL updates built on type-p Wasserstein, Kullback-Leibler (KL), and hi^2 ambiguity sets. Strong duality reduces each update to a relative-reward regression, retaining REBEL's scalability without PPO-style clipping or value networks. Under linear rewards, log-linear policies, and a standard coverage condition, we prove widetilde{O}(sqrt{d/n}) bounds on squared parameter error, with sharper constants than prior DRO-DPO analyses, and give the first parametric widetilde{O}(d/n) rate for DRO-based alignment under preference shift, matching non-robust RLHF in benign regimes. Each divergence yields a tractable SGD-based algorithm: gradient regularization for Wasserstein, importance weighting for KL, and a 1-D dual solve for hi^2. On Emotion Alignment, the ArmoRM multi-objective benchmark, and HH-Alignment, DRO-REBEL outperforms prior robust and non-robust baselines across unseen preference mixtures, model sizes, and dataset scales.

Related

• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
• Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
• LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
• From log pi to pi: Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight

Source: arXiv cs.LG | 2026-04-20