Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless l^p Norm Solution for Fast Adversarial Training

arXiv:2505.02360v2 Announce Type: replace-cross Abstract: Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants. While existing solutions rely on noise injection, regularization, or gradient clipping, we propose a novel solution that purely controls the l^p training norm to mitigate CO. Our study is motivated by the empirical observation that CO is more prevalent under the l^{infty} norm than the l^2 norm. Leveraging this insight, we develop a framework for generalized l^p attack as a fixed point problem and craft l^p-FGSM attacks to understand the transition mechanics from l^2 to l^{infty}. This leads to our core insight: CO emerges when highly concentrated gradients where information localizes in few dimensions interact with aggressive norm constraints. By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive l^p-FGSM that automatically tunes the training norm based on gradient information. Extensive experiments demonstrate that this approach achieves strong robustness without requiring additional regularization or noise injection, providing a novel and theoretically-principled pathway to mitigate the CO problem.

Source: arXiv cs.AI | 2026-05-19