Dimension-Free PAC-Bayes Generalization for Deep Neural Networks via Dirichlet Processes

arXiv:2610.21340v1 Announce Type: cross Abstract: Classical PAC-Bayesian generalization theory places prior distributions over Euclidean parameter space Theta subseteq R^D, causing relative entropy to scale with network dimension D. For modern overparameterized neural networks where D gg n, this parameter-counting dependency yields vacuous generalization bounds (> 1{,}000%). In this work, we resolve this barrier by shifting prior specification from internal parameters to the non-parametric space of data distributions M_1(Z) via a extbf{Dirichlet Process (DP)}, DP(alpha, F_0), inducing parameter distributions through push-forward measures under the learning algorithm. By the Data Processing Inequality, parameter-space relative entropy is strictly bounded by the data-space DP divergence, which we evaluate in exact closed form using Digamma functions. The resulting PAC-Bayesian certificate is strictly invariant to network parameter count D, width, and depth. On intrinsic d^-dimensional submanifolds, optimal partition scaling achieves the minimax non-parametric rate O(n^{-1/(d^+2)} sqrt{ln n}). Across vision architectures scaling up to 4.2 million parameters (D = 7{,}850 to D = 4{,}203{,}522) on the full MNIST benchmark, while classical weight-space bounds scale past 940% (becoming vacuous), our data-space DP-PAC-Bayes certificate remains stable, non-vacuous, and certified at extbf{8.8%--9.4%}.

Source: arXiv cs.LG | 2026-10-16