Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox

arXiv:2609.32060v1 Announce Type: new Abstract: How many bits does a network need before its accuracy collapses, and how does this grow with depth? We study the precision floor, the perturbation level or bit-width at which accuracy falls halfway to chance, in MLPs, CNNs, Vision Transformers and nine pretrained language models, under post-training quantization (PTQ) and quantization- or noise-aware training (QAT). (i) A first-order theory sets the floor through one full-precision quantity, the predictive amplification G: eta_c=Lambda/G, and G^2 grows linearly in depth at a rate proportional to the squared residual branch scale. (ii) The predicted equality alpha_{PTQ}=rho of depth exponents holds within 95% intervals in twelve of thirteen trained architectures and in GPT-2 from 12 to 48 layers, with Lambda=1.45pm14% across trained architectures. (iii) Residual branches scaled by 1/sqrt{D} and pre-normalisation remove the depth penalty, and each quantizer turns noise into bits at a rate fixed by its step rule, giving b_c=(alpha/gamma)log_2 D+C. (iv) A QAT paradox: noise-aware training roughly doubles the tolerable noise of shallow networks, but the gain decays with depth, so the depth law steepens (alpha_{QAT}/alpha_{PTQ}=1.45-1.47 on two datasets, ten seeds each). Decision margins, cross-layer error cancellation and heavy tails do not set the floor.

Source: arXiv cs.LG | 2026-09-29