Safety
Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction
arXiv:2604.21203v1 Announce Type: cross Abstract: We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as p
arXiv:2604.21203v1 Announce Type: cross Abstract: We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of n^{(alpha-1)/2} sqrt{log n}, outperforming existing Hessian-free alternatives.
Related
- Truncated Kernel Stochastic Gradient Descent with General Losses and Spherical Radial Basis Functions
- Online Covariance Estimation in Averaged SGD: Improved Batch-Mean Rates and Minimax Optimality via Trajectory Regression
- Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs
- From Recency Bias to Stable Convergence Block Kaczmarz Methods for Online Preference Learning in Matchmaking Applications
Source: arXiv cs.LG | 2026-04-24