KLQ: Training-free measured rotation quantization. Beats all training-free rotation-based quantization methods on W4A4KV4-bits. Llama 3.2 1B KLQ-quantized beats SpinQuant and gets close to ReSpinQuant without GPTQ/LDLQ rounding.

First of all, I'm not a lab, this was a solo summer research project that finally culminated into the github repo and the writeup. The repo includes a much deeper dive with methods, findings about quantization and geometry, limitations, and proposed experiments. I'll also mention that this is far from production-grade, it's mostly a theoretical framework with a "fake" quantization demo as it lacks real kernels. The geometry of LLMs embedding spaces is highly uneven with a few features having the most magnitude, this has been known for years by now and it's in great part why rotation-based quantizers do so well against uniform quantization: While uniformly quantizing tries to allocate bits evenly in a naturally uneven space, rotations can forcefully make that space even again so uniformly allocating bits is the best strategy (DuQuant, 2nd half of ResQ, QuaRot...). Generic rotations (Hadamard) even the space out on average but can't match a specific model's geometry, leaving residual damage. This can be fixed by using learnable rotations (SpinQuant, ReSpinQuant) but this is computationally intensive as it requires extensive post-training gradient descent. KLQ takes a different approach to quantization, instead of trying to make the space even and then quantize uniformly. KLQ measures how uneven the space is, ranks directions of the eigenbasis from most important to least important, and with a price function treating each direction as a independent information transmission channels uses the provably optimal (under some idyllic assumptions about damage anyways) waterfilling algorithm to give the most bit-width to the most important directions and least bit-width to least important directions. Another thing that sets KLQ apart is the use of causal KL damage measurements, there are a few quantization algorithms that do try to measure the space and then quantize unevenly. CoQuant, for example, does measure the activation space, but then ranks directions by magnitude/variance and applies a simple two-ranked bit allocation that quantizes the top 12.5% to 8 bits and the bottom 87.5% to 4 bits. Unlike CoQuant, KLQ doesn't use variance (several tests reveal variance is often not a good signal, more detailed experimentation on the github writeup), instead it perturbs each direction and runs a forward pass with a few thousand tokens, it takes the KL divergence between the original model and the model with the perturbed direction, then uses this measured KL divergence to determine how important the direction is and assign the real empirical cost of damaging/quantizing it. The method, as well as my experimentation, does have real limitations, to quantize all layers, activations and KV cache you must make one forward pass per direction per matrix per layer which can amount to hundreds of thousands of forward passes to quantize a model. This makes the method very compute-intensive (This probing process took 5 hours for Qwen 2.5 0.5B on a 3090 and 10 hours for Llama 3.2 1B on that same hardware.). It also deliberately uses two simple techniques to actually quantize the models: a simple additive vector codebook and round-to-nearest (RTN), these could be swapped with other methods readily. Posting here I'm looking for feedback and to make these results known. Feel free to ask any questions or to contribute to the github repo. Here's a sample of the result's table for Llama 3.2 1B quantized fully at 4-bits. Method W4A4KV4 Llama 3.2 1B Wikitext-2 PPL FP16 9.75 QuaRot (training free) 14.59 SpinQuant (trained + GPTQ) 13.52 KLQ (training-free, VQ) 13.36 ReSpinQuant (trained + GPTQ) 13.09 submitted by /u/Federal-Setting-3014 [link] [comments]

Related

• [[paper-statistically-lossless-quantization-of-large-language-|[Paper] Statistically-Lossless Quantization of Large Language Models]]
• [[release-wintermix-qwen35-122b-a10b-in-native-mlx-an-82-gib-b|[Release] WinterMix — Qwen3.5-122B-A10B in native MLX: an 82 GiB build that beats 94–95 GiB quants, plus a 68 GiB build for agent swarms]]
• llama.cpp's Preliminary SM120 Native NVFP4 MMQ Is Merged

Source: r/LocalLLaMA | 2026-08-09