Critical initialization destabilizes higher input derivatives in wide scalar-input networks

arXiv:2609.09244v1 Announce Type: cross Abstract: The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.

Related

• CAWI: Copula-Aligned Weight Initialization for Randomized Neural Networks
• Taming the Loss Landscape of PINNs with Noisy Feynman-Kac Supervision: Operator Preconditioning and Non-Asymptotic Error Bounds
• Examining the robustness of Physics-Informed Neural Networks to noise for Inverse Problems
• Do physics-informed neural networks (PINNs) need to be deep? Shallow PINNs using the Levenberg-Marquardt algorithm

Source: arXiv cs.LG | 2026-09-10