laya.cpp: Optimized laya near-instant decision making

After seeing u/Nandakishor_ml’s post introducing Laya, I wanted to see how fast it could run in a standalone C++ implementation. Credit to u/Nandakishor_ml for the architecture, training and open-source release. My contribution is the inference implementation: laya.cpp, built on ggml with custom CUDA kernels. It supports all three checkpoints—English, multilingual and typed-decisions—with native tokenization, model execution and output formatting. There’s also an HTTP server with a JEV-compatible endpoint. No Python or PyTorch is required for inference. Some English-model results on an RTX PRO 6000 Blackwell, capped at 450 W: Batch Python BF16 C++ BF16 Python FP32 C++ FP32 1 149 366 148 342 2 268 586 202 421 4 460 761 233 437 8 663 810 232 386 These are questions per second over a fixed 250-question corpus containing choices, scores and booleans. Each precision has paired Python/C++ timings with alternating execution order. Loading and JSON transport are excluded. The README has the full three-model results. Most of the optimization came from removing unnecessary conversions and copies, fusing operations while preserving rounding, and improving attention memory access. The code is MIT-licensed. BF16 currently needs the documented CUDA 13.0/cuBLAS 13.1.0 build profile. Implemented using Codex Astra. submitted by /u/lkarlslund [link] [comments]

Related

• Headlong: An open source agent microharness featuring persistent agency and recursive LLMs
• I literally built the Jev architecture one year back and completely open-sourced it with model, dataset and paper

Source: r/LocalLLaMA | 2026-09-20