I released a softmax-free attention model at GPT-2 Medium scale (~354M params, 11.5B tokens): structural sparsity + tile-skipping kernels for long-context VRAM savings. Open weights + custom Triton kernels [R]

A researcher released an open-source softmax-free attention model at GPT-2 Medium scale (354M parameters trained on 11.5B tokens) that uses structural sparsity and tile-skipping kernels to reduce VRAM requirements for long-context processing. The release includes custom Triton kernel implementations to optimize the efficiency of this alternative attention mechanism. This work demonstrates a practical approach to scaling transformer models with reduced memory overhead through architectural and computational innovations.

Source: r/MachineLearning | 2026-06-21