Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions

arXiv:2604.23212v1 Announce Type: cross Abstract: Existing large-dimensional theory for spectral algorithms resolves either the optimally tuned point or the interpolation limit, but leaves the under-regularized regime unexplored. We study the learning curve and benign overfitting of spectral algorithms in the large-dimensional setting where the sample size and dimension are of comparable order, i.e., n asymp d^{gamma} for some gamma>0. We first consider inner-product kernels on the sphere S^{d-1} and establish a sharp asymptotic characterization of the excess risk across the full regularization path under various source conditions s geq 0, where s measures the relative smoothness of the regression function. Our results reveal that the learning curve is not simply U-shaped but instead consists of three distinct regimes: over-regularized, under-regularized, and interpolation regimes. This characterization allows us to fully capture the benign overfitting phenomenon, demonstrating that benign overfitting arises consistently across both the under-regularized and interpolation regimes whenever s is positive but no larger than a critical threshold. We further show that, in the sufficiently regularized regime, the kernel learning curve is recovered by an associated sequence model. Finally, we extend the learning-curve analysis to large-dimensional KRR for a class of kernels on general domains in R^d whose low-degree eigenspaces satisfy spectral-scaling and hyper-contractivity conditions.

Source: arXiv cs.LG | 2026-04-28

Related