When Does ell_2-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ell_1 Implicit Bias

arXiv:2605.06314v2 Announce Type: replace Abstract: Benign overfitting is well-characterized in ell_2 geometries, but its behavior under the ell_1 implicit bias of greedy ensembles remains challenging. The analytical barrier stems from the non-linear coupling of coordinate selection thresholds, which invalidates standard spectral resolvent tools. To isolate this algorithmic bias, we characterize the high-dimensional risk of continuous-time ell_2-Boosting over p features and n samples. By coupling the Convex Gaussian Minimax Theorem with delicate asymptotic expansions of double-sided truncated Gaussian moments, we analytically resolve the non-smooth ell_1 interpolant. Under an isotropic pure-noise model, we prove that benign overfitting fails at the linear rate: greedy selection localizes noise into sparse active sets, and the excess variance decays at a logarithmic rate Theta(sigma^2/log(p/n)) for noise variance sigma^2. We remark that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem. For spiked-isotropic designs with k^* head eigenvalues and r_2 = p - k^* tail dimensions, the risk converges to zero when r_{2} gg n, but only at a logarithmic rate Theta(sigma^2/log(r_2/n)), which is slower than the linear decay observed in ell_2 geometries. To avoid this slow convergence, we analyze the non-smooth subdifferential dynamics of the boosting flow. This yields a tuning-free early stopping rule that, under a bounded ell_1-path condition, recovers the Lasso basic inequality and attains the minimax-optimal empirical prediction rate for ell_1-bounded signals.

Source: arXiv cs.LG | 2026-05-13