ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

ParallelKernelBench is a benchmark from Together AI that evaluates frontier large language models' ability to write optimized multi-GPU CUDA kernels, revealing significant limitations in current LLMs' capacity to generate high-performance parallel computing code. The benchmark demonstrates that even state-of-the-art models struggle with the complex task of creating efficient kernels for distributed GPU computation, highlighting a gap between LLMs' coding capabilities in typical domains and specialized high-performance computing scenarios.

Source: Together AI Blog | 2026-06-23