Model vs. User: Whose Values Prevail? Inference-Time Value Alignment in Multi-Turn Conversations

arXiv:2610.19817v1 Announce Type: new Abstract: Human values shape how we expect language models to behave. Dominant alignment approaches treat values as global objectives, although the appropriate value orientation is personal and contextual. Training-free alignment can steer models toward target values at inference time, yet has been largely evaluated in single-turn settings and with a single steering direction (e.g., toward safety). We evaluate four prompt- and activation-based steering methods in multi-turn conversations, where user and assistant models push toward the same or opposing, equally desirable values. We track alignment with a linear probe across three models and two value trade-offs. When assistant and user pull in opposite directions, the side matching the assistant's baseline bias often prevails, and effects that look successful early often fade or reverse after a few turns. Steering should therefore be evaluated over full conversations and in both directions of a value trade-off.

Related

• MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
• Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning
• Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time
• Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignment

Source: arXiv cs.CL | 2026-10-15