The Fourth Quadrant: A Stylized View of Benign Misfitting

arXiv:2608.01032v1 Announce Type: new Abstract: Training error is what we can observe on a training set; test error is the quantity we actually care about. We study linear regression with squared-error in a deterministic (d+1)-dimensional single-spike model. Each stylized training vector has the same informative spike coordinate, of amplitude sqrt{gamma} with gamma>1. The remaining directions are nuisance, and the nuisance components of distinct training vectors all have equal norm and are mutually orthogonal. The training labels are all 1. Fresh test points are drawn from ec{x}{rm test} sim N(ec{0},operatorname{diag}(gamma,1,ldots,1)), with the noise-free test labels being the normalized spike coordinate x{rm test}[1]/sqrt{gamma}. We focus on linear predictors in the span of the training vectors, the class naturally reached by zero-initialized linear gradient methods. We exhibit a range of training-set sizes n in which every span predictor that generalizes well must fit the training data worse than the zero predictor. We call this regime benign misfitting, or the fourth quadrant. The best span predictor begins to generalize when ngg d/gamma^2, while interpolation does not generalize until the later threshold ngg d/gamma. In the window d/gamma^2 ll n ll d/gamma, useful prediction within the linear span lies beyond interpolation: predictions on the training points overshoot the labels. We show that one-pass stochastic gradient descent (SGD), with a large constant learning rate, reaches small test error throughout this window---matching the best span predictor up to a logarithmic factor. We also verify directly that it indeed has large empirical training error (despite the descent premise in its name). Finally, we show that the unavoidable nuisance component responsible for the training misfit also controls the predictor's adversarial sensitivity.

Source: arXiv cs.LG | 2026-08-04