At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

arXiv:2607.25504v1 Announce Type: cross Abstract: Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by 6.9ext{--}7.4imes over optimized RVV baselines, with only 3.1% area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large 4imes4 multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical 40ext{--}60% dual sparsity, Ventaglio achieves 2.40ext{--}5.25imes and 2.06ext{--}3.16imes speedup over dense baselines during prefill and autoregressive decoding, respectively.

Source: arXiv cs.AI | 2026-07-29