Safety
PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planning
arXiv:2602.13691v2 Announce Type: replace Abstract: Recent advancements in Large Language Model (LLM) agents have demonstrated strong capabilities in executing complex tasks through tool use. However,
arXiv:2602.13691v2 Announce Type: replace Abstract: Recent advancements in Large Language Model (LLM) agents have demonstrated strong capabilities in executing complex tasks through tool use. However, long-horizon multi-step tool planning is challenging, because the exploration space suffers from a combinatorial explosion. In this scenario, even when a correct tool-use path is found, it is usually considered an immediate reward for current training, which would not provide any reusable information for subsequent training. In this paper, we argue that historically successful trajectories contain reusable tool-transition patterns, which can be leveraged throughout the whole training process. Inspired by ant colony optimization where historically successful paths can be reflected by the pheromone, we propose Pheromone-Guided Policy Optimization (PhGPO), which learns a trajectory-based transition pattern (i.e., pheromone) from historical trajectories and then uses the learned pheromone to guide policy optimization. This learned pheromone provides explicit and reusable guidance that steers policy optimization toward historically successful tool transitions, thereby improving long-horizon tool planning. Comprehensive experimental results demonstrate the effectiveness of our proposed PhGPO.
Related
- STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training
- Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
- HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
- SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents
Source: arXiv cs.AI | 2026-10-02