Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context

arXiv:2608.17499v1 Announce Type: new Abstract: User-facing tool agents must coordinate dialogue and tool use as user goals unfold over multiple turns. Yet interactive reinforcement learning typically reduces each rollout to a terminal reward, assigning the same credit to effective elicitation, errors, and later repair. The next user turn is more than context: it also provides noisy, temporally local evidence about the preceding user-to-user segment. We introduce extbf{F}eedback-extbf{A}ware extbf{C}redit extbf{A}ssignment (extsc{FACA}), which aligns each reaction with that segment, derives a locally normalized reaction advantage, and adds it to verified terminal outcome advantage without an extra critic or rollout. Against an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization, extsc{FACA} improves the nine-domain au-family average across three independently trained runs by 5.91 and 10.22 percentage points at 8B and 14B, respectively. Gains concentrate in Telecom; at 8B, randomizing reaction polarity removes the Telecom gain. The same ordering holds zero-shot on Pare-Bench and Co-Gym. These results demonstrate that next-turn user reactions provide actionable local credit for improving multi-turn user-interacting agents.

Related

• ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
• UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
• Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning

Source: arXiv cs.AI | 2026-08-19