Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

arXiv:2608.24593v1 Announce Type: new Abstract: Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether this delayed transport is large enough to change prospective short-horizon decisions. On committed future-minibatch sequences, we differentiate eight-step AdamW trajectories through the complete model--optimizer state and select exposure-matched Math--Code loss schedules before independent evaluation. Across 12 unused 0.3M Transformer histories, full transport lowers token-disjoint loss relative to an optimizer-aware immediate derivative in 10/12 histories (mean benefit 4.71imes10^{-4}; exact one-sided sign test, p=0.0193). The two controllers act equally often but select different schedules in 60/96 windows. Crossed checkpoint--future-path tests attribute this reordering to the interaction between optimizer state and near-future data, while an independent Ising--CNN experiment shows that deleting moment-state transport destroys accurate response prediction. Full-transport scores also concentrate exact-rollout winners in larger candidate libraries, focusing finite-amplitude evaluation on a shortlist. On these committed short paths, optimizer memory and near-future data order are therefore actionable components of the training state, providing a mechanism-based criterion for when finite-horizon rather than one-step intervention is required.

Related

• LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication
• Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
• Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning

Source: arXiv cs.LG | 2026-08-26