A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization

arXiv:2508.17412v4 Announce Type: replace Abstract: Conventional regularization is designed to control variance, but in small-data regression it can also aggravate underfitting when predictive signal is concentrated in weak directions of a restricted representation. We study a negative-capable ridge family that permits a feasible negative region whenever the estimator remains well posed, and show that negative regularization acts there as controlled anti-shrinkage by increasing effective complexity most strongly along weak eigendirections. Building on this mechanism, we formalize weak-spectrum underfitting, derive a sign-switch result under conservative baseline shrinkage, and study criterion-based automatic selection over the full negative-capable family. Synthetic and semi-synthetic experiments support the theory by verifying feasibility, spectral complexity increase, sign-switch behavior, and effective recovery of negative adjustments in the predicted regimes.

Related

• Optimal L2 Regularization in High-dimensional Continual Linear Regression
• Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization
• Minimax Optimality and Spectral Routing for Majority-Vote Ensembles under Markov Dependence
• PAC-Bayesian Bounds on Constrained f-Entropic Risk Measures

Source: arXiv cs.LG | 2026-04-21