Early Memory Selection for Balanced Adam

arXiv:2610.08624v1 Announce Type: cross Abstract: We propose a method for choosing the shared memory parameter eta_1=eta_2=eta in Adam from a short pilot training. The selected eta remains fixed during the subsequent full training. A local model of Adam's normalized direction balances sampling variability against the delay introduced by averaging past gradients. This balance gives a cubic memory rule, whose two coefficients are estimated from gradient probes at a few pilot checkpoints. The estimator uses the numerator and denominator jointly, preserving their covariance. With a 200-update pilot and sixteen probe gradients at each of four checkpoints, a seed-matched retrospective evaluation on eleven vision and language workloads reduces mean relative validation gap by 40.7% and worst-quarter mean gap by 44.3% against the grid representative of shared eta=0.95. The mean gap is also 32.3% lower than that of the best constant eta chosen across all eleven workloads.

Related

• Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
• FOAM: Blocked State Folding for Memory-Efficient LLM Training
• OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

Source: arXiv cs.AI | 2026-10-07