Gaussian Limits for SGD Without Stationary Moments

arXiv:2610.05599v1 Announce Type: cross Abstract: Temporal dependence can separate the Gaussian approximation of stochastic gradient descent from its stationary moments. For unmodified least-squares SGD, we construct a design with standard Gaussian marginals whose stationary error has every positive moment infinite. Independent observations with the same marginals instead give finite stationary variance. Both regimes retain a Gaussian small-step limit. Our general theory establishes pathwise contraction from a finite second design moment, then uses score cancellation and localization to obtain stationary Gaussian and Ornstein--Uhlenbeck limits. Independent Gaussian regression errors yield an exact conditional Gaussian law and total-variation convergence under the same design integrability. Stronger design conditions identify a positive first-order total-variation constant and a deterministic covariance correction with o(a) error. A scalar coverage expansion translates this correction into its inference consequence. Experiments examine distributional error, coverage, and calibration with dependent scores. Together, these results establish precise probability-law approximation beyond moment-based stationary analysis.

Related

• Gaussian Approximation and Multiplier Bootstrap for Federated Linear Stochastic Approximation
• Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation

Source: arXiv cs.LG | 2026-10-06