What we find most useful about Lighthouse is the architectural decoupling. The selection logic is fully separated from the attention kernel,…

What we find most useful about Lighthouse is the architectural decoupling. The selection logic is fully separated from the attention kernel, which means every upstream software or hardware attention improvements are inherited for free, and the same kernel runs at training and inference. Most sparse-attention methods require a custom kernel, and we think more should be designed without one. Paper erratum: In the paper, we say FA3/FA4, but the comparison experiments were done fairly by using cuDNN+Lighthouse and comparing against cuDNN+SPDA, not FA3/FA4. This is reflected in the fully reproducible code we release along with the paper.

Source: Nous Research (X) | 2026-05-15