Generalization behavior of OPTQ and the role of regularization

arXiv:2609.31560v1 Announce Type: new Abstract: Large neural networks can be compressed by rounding or "quantizing" their weights to numbers that admit representations with fewer bits. One algorithm for quantization, OPTQ, progressively quantizes the weights of a neural network so that the squared quantization error on a specified calibration dataset is as small as possible. We study the performance of OPTQ and a variant algorithm, stochastic OPTQ, in a generalization setting and derive bounds for the expected squared error accrued by the algorithm when a test point is drawn from a fixed distribution. We prove two results. One result relates the generalization error to the error on a calibration dataset comprising independent samples from the same distribution as the test distribution. The other result bounds the generalization error of stochastic OPTQ for all sufficiently nice distributions, regardless of the calibration dataset. In both of these results, the regularization term lambda plays an important role. We use insights from these results to make a new recommendation for the choice of lambda and see that this choice of lambda preforms favorably in experiments when compared to prior recommendations in the literature.

Related

• GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
• Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks

Source: arXiv cs.LG | 2026-09-28