Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant

arXiv:2605.08114v1 Announce Type: new Abstract: We analyse three KV cache quantization schemes under a fair bit budget: extbf{KV} (scalar MSE baseline), extbf{KQV} (WHT + MSE on K; WHT + MSE + QJL on V), and extbf{QKQV} (WHT + MSE + QJL on both). Starting from the Beta distribution on the hypersphere, we trace how QJL on K inflates inner product variance by pi/2, which softmax amplifies nonlinearly via Jensen's inequality, and we present statistical inference and information metrics to highlight practical differences. Three empirical findings emerge. (1)~At n=4 (the practically dominant budget), KQV wins on every measure -- KL divergence, geometric K error, and 6D distance -- across all distributions and ranks tested. (2)~The K--V asymmetry is unconditional: QKQV is consistently worse than KQV in KL divergence at every budget and distribution. (3)~A budget-dependent crossover exists: QKQV achieves better geometric K reconstruction at n in {2,3,5}, KQV at n in {4,6}, invariant to rank and tail weight -- an open rate-distortion problem. KL(p_{ref} | p_{quant}), K-only by construction, bridges K direction error to routing corruption and output collapse. We present a sufficient condition when the Jensen mechanism amplifies superlinearly through the softmax. At n in {2,3,5}, QKQV wins geometrically because this assumption does not bind. At n=4, elevated K error and KL divergence for QKQV strongly suggest the Jensen mechanism is the operative cause of the crossover, providing a new perspective and explanation.

Source: arXiv cs.LG | 2026-05-12