Constitutional Value Potentials: reading and steering internal priority margins in language models

arXiv:2606.15420v2 Announce Type: replace-cross Abstract: Safety evaluations test a policy on prompts that omit the incentive information deployment supplies: a commission, a performance score, a dashboard naming which action pays best. We measure what that omission hides. In MoneyWorld, a synthetic workplace environment, we train five instruction-tuned models from three families with RL on non-safety tasks in which a visible payoff signal identifies a rewarded shortcut that sacrifices task quality. We then freeze each policy, present held-out safety conflicts, and change only the displayed signal. Each menu contains one compliant action and three violations. We report three findings, with rates for Qwen2.5-14B-Instruct. (i) Payoff signals control frozen safety choices: unsafe choice is 100% when the signal names an unsafe option and 0% when it is hidden or names the safe one. Hidden- and random-signal training controls stay at or below 0.3%, and the switch reproduces on all five bases. Numerical payouts reproduce it under sampled-action rewards, reaching 98.6% unsafe choice at a $1 advantage. (ii) Payoff identification and unsafe choice separate under a training-menu intervention: training on task-completing actions at the same payouts retains 99.8% identification while reducing unsafe choice to 7.9% at matched update budgets. Payoff-reading competence alone does not explain transfer. (iii) The switch does not reproduce in executed retail customer-service tasks using the same frozen adapters. In MoneyWorld, omitting incentive information conceals unsafe choices that appear when the same policy sees which action pays best.

Related

• BarrierSteer: LLM Safety via Learning Barrier Steering
• Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs
• DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
• Culturally Grounded Personas in Large Language Models: Characterization and Alignment with Socio-Psychological Value Frameworks

Source: arXiv cs.AI | 2026-10-02