SandwichQuant: Which Parameters Matter Before and After Quantization?

arXiv:2608.24173v1 Announce Type: new Abstract: Quantization correction methods usually optimize weights, quantization parameters, or reconstruction objectives, while the underlying parameter subspaces responsible for effective correction remain unclear. In this work, we study quantization correction from a parameter subspace perspective and reveal that correction capability is highly non-uniform across parameter groups. By decomposing trainable parameters into backbone weights, normalization-affine parameters, and quantization parameters, we show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets. Based on this finding, we propose SandwichQuant, a two-stage normalization-affine correction framework that performs adaptation before and after quantization. The pre-stage improves quantization robustness, while the post-stage compensates residual errors after the quantized graph is fixed. Extensive experiments on vision models and large language models demonstrate consistent improvements under various low-bit quantization settings, validating the effectiveness of subspace-aligned correction.

Related

• iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models
• Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization
• Decompose, Mix, Adapt: A Unified Framework for Parameter-Efficient Neural Network Recombination and Compression

Source: arXiv cs.CV | 2026-08-26