Qwen3.8-27B on a single 3090: 140 tok/s on code with a custom megakernel

I've been using Claude Opus 5.5 to speed up Qwen3.8-27B on my PC (rtx 3090), it wrote a CUDA megakernel that is 1.4-1.9x faster than llama.cpp depending on the task/context length. The results and code are below: Results (same hardware, my megakernel vs llama.cpp with MTP): - Writing code: 140 tok/s vs 73 (1.9x faster) - Coding with a 4K-token file in context: 95 tok/s vs 56 (1.7x faster) - Prompt processing (prefill): ~1,600 tok/s vs ~1,100 (1.4x faster) The speedup comes from running each whole speculative-decoding cycle in one kernel launch, so checking 4-5 drafted tokens costs about the same as checking one. It's an OpenAI-compatible server, so it works as a drop-in replacement for the llama server. Caveats and limitations: it's built for one model (unsloth's Q4_K_M) on 3090s only. The outputs match llama.cpp's apart from rare rounding near-ties. KVCache is fp16 up to ~80k context, then q8. Code and full benchmarks: https://github.com/L-Forster/open-jet/tree/master/megakernel submitted by /u/Adorable_Weakness_39 [link] [comments]

BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
[[benchmark-optimal-dflash2-quants-for-speed-and-context-size-|[Benchmark] Optimal DFlash2 quants for speed and context size, 5090 RTX, llama.cpp, Qwen 3.8 27B Dynamic3 Unsloth. Comparison with MTP]]
CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token by ynankani · Pull Request #27621 · ggml-org/llama.cpp

Source: r/LocalLLaMA | 2026-10-10

Related