Applications
Low-Discrepancy Dither for Quantized Recurrent State Caches
arXiv:2609.39185v1 Announce Type: new Abstract: Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this st
arXiv:2609.39185v1 Announce Type: new Abstract: Mamba-style and hybrid language models compress their past into a fixed-size recurrent state that is rewritten at every generated token. Storing this state in low precision saves memory bandwidth, but every rounding error is fed back into the next update and can accumulate over long generations. Production systems round the state stochastically; we ask which rounding rule such caches should use. We find that a deterministic golden-ratio Weyl dither, which needs no random numbers, consistently brings the quantized model closer to the full-precision one than stochastic rounding, across pure and hybrid models, storage formats, and long decoding horizons, at no extra cost. Round-to-nearest behaves differently: because it discards small updates, its error keeps growing, so it can look best in short evaluations yet falls far behind over long generations. A discrepancy analysis explains this ordering, and we document implementation pitfalls that silently remove the benefit.
Related
- Perturbation is All You Need for Extrapolating Language Models
- Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
- HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
- Calibration Bets on the Past: Post-Training Quantization for Financial Time-Series Forecasting
Source: arXiv cs.LG | 2026-10-01