Model Releases
LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators
arXiv:2609.39361v1 Announce Type: new Abstract: While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this pheno
arXiv:2609.39361v1 Announce Type: new Abstract: While most attention logits can be computed in low precision without degrading numerical stability, current attention kernels fail to exploit this phenomenon. We introduce a novel hardware-algorithm co-design in the form of mixed-precision FlashAttention. Our method accumulates key-query products and evaluates their exponentials in 8-bit formats, then adaptively identifies sensitive sub-blocks and recomputes them in 16-bit formats. We propose the specifications for a dedicated accelerator capable of executing this pipeline efficiently. Simulated experiments with Qwen3 and Gemma 3 show that rerouting a selective minority of sub-blocks to high precision is sufficient to recover the baseline model performance.
Related
- From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
- Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training
- GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
- CoQuant: Joint Weight-Activation Subspace Projection for Mixed-Precision LLMs
- One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
Source: arXiv cs.LG | 2026-10-01