How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

arXiv:2608.07118v1 Announce Type: new Abstract: Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.

Related

• Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
• Rethinking Token-Level Credit Assignment in RLVR: A Polarity-Entropy Analysis
• HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents
• AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment

Source: arXiv cs.AI | 2026-08-10