Drowning in Routine: Signal Dilution in Multi-Turn Agent Training

arXiv:2606.22164v1 Announce Type: new Abstract: Multi-turn agents interleave consequential decisions with routine execution: some actions change the downstream return distribution, while others are necessary but reward-equivalent. The cost of trajectory-level credit assignment, often attributed to long horizons, is in fact governed by decision density rho: the fraction of turns whose actions affect the return. When decision density is low, routine turns create signal dilution: they add gradient variance to trajectory-level estimators such as GRPO without adding expected signal. Under explicit assumptions, the resulting turn-level to trajectory-level signal-to-noise ratio scales as rho^{-1/2}, provided critic error remains controlled. The same analysis identifies the complementary regime: at high decision density, trajectory-level methods can remain competitive while avoiding the cost of a critic. In a controlled environment where rho is exactly tunable, the predicted scaling is recovered with R^2 = 0.999, and the training-step gap widens significantly as rho o 0.

Source: arXiv cs.LG | 2026-06-23