Near-optimal estimates for the ell^p-Lipschitz constants of deep random ReLU neural networks

arXiv:2506.19695v2 Announce Type: replace-cross Abstract: This paper studies the ell^p-Lipschitz constants of ReLU neural networks Phi: R^d o R with random parameters for p in [1,infty]. The distribution of the weights follows a variant of the He initialization. In the case of zero-bias networks, we derive high probability upper and lower bounds for wide networks that differ at most by a factor that is logarithmic in the network's depth. Remarkably, the behavior of the ell^p-Lipschitz constant varies significantly between the regimes p in [1,2) and p in [2,infty] . For p in [2,infty], the ell^p-Lipschitz constant behaves similarly to Vert gVert_{p'}, where g in R^d is a d-dimensional standard Gaussian vector and 1/p + 1/p' = 1. In contrast, for p in [1,2), the ell^p-Lipschitz constant aligns more closely to Vert g Vert_{2}. We extend our analysis to networks with possibly non-zero biases drawn from arbitrary symmetric distributions. In this case, we obtain high probability upper and lower bounds that differ at most by a factor that is logarithmic in the network's width and linear in its depth.

Related

• Path-conditioned training: a principled way to rescale ReLU neural networks
• Adversarial Robustness of NTK Neural Networks
• Worst-Case Distance-Aware Error Bounds for Neural Networks

Source: arXiv cs.LG | 2026-09-11