Model Releases
Hardware-Aware FP4 FlashAttention-4
arXiv:2609.04105v1 Announce Type: new Abstract: Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies domin
arXiv:2609.04105v1 Announce Type: new Abstract: Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with Direct-P for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13imes the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14imes. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Related
- UE5M3 FP4 Block Scaling for Stable Language Model Pretraining
- Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
- KernelFoundry: Hardware-aware evolutionary GPU kernel optimization
Source: arXiv cs.LG | 2026-09-04