Safety
IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents
arXiv:2608.24588v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings
arXiv:2608.24588v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: {au^2}-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.
Related
- Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment
- HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
- STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
- DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignment
- TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents
Source: arXiv cs.LG | 2026-08-26