Hardware
b11576: graph: pad the KV dimension of the no-cache attention to 256 (#30416)
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels for head size 512 require a multiple of FATTN_KQ_STRIDE,
The no-cache path builds a [n_tokens, n_tokens] mask and passes K and V unpadded, so FLASH_ATTN_EXT sees K->ne[1] == n_tokens. The CUDA kernels for head size 512 require a multiple of FATTN_KQ_STRIDE, so the op falls back to the CPU for most lengths, and the -fa auto probe (1 token) disables flash attention for such models. Pad the mask like llama_kv_cache::get_n_kv and pad K and V with zeros up to the KV dimension of the mask. clef builds its own causal mask and pads it the same way.
Related
- FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward
- CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels
- Great collab with @SakanaAILabs on an #ICML26 paper about sparse transformer kernels + formats optimized for modern NVIDIA GPU execution. • …
- The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough
- M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification
Source: llama.cpp Releases | 2026-10-12