I ran all the popular Qwen 3.8 FN engines on a single 5090 + 128GB of RAM, here are the numbers

Before you read: I'm the developer of Basalt, one of the engines benchmarked below. I tried to keep this as fair as possible: same checkpoint, GPU and requests, and every engine set up the way its own docs recommend. How it was run is at the bottom. This post has had some AI cleanup from my own wording. I wanted to get a comparison on all of the Qwen 3.8 FN engines I hear people talking about, so after 5 or so excruciating hours of setup (remind me not to do this ever again), I benchmarked all of the ones I know of: Strata, Infernix, capy-flash, ninfer-ext, ninfer, llama.cpp and Basalt. Each one converted NVIDIA's NVFP4 with its own tool, then got the same seven greedy requests on an RTX 5090 at 400 W, one cold start per request. Anyway, here are the results: Decode, short context Decode, short context Decode at context depth Decode at context depth Prefill at context depth Prefill at context depth All the numbers Best value per column in bold. Decode, tokens per second Engine Struct Conv Prose 25K 50K 100K 200K Basalt 354 188 180 190 187 184 160 Infernix 349 170 163 163 174 176 158 capy-flash 292 165 152 152 143 152 135 Strata 204 146 123 130 123 121 114 ninfer-ext 140 101 95 92 86 79 66 ninfer 54 77 45 53 46 47 44 llama.cpp 39 39 39 37 36 35 32 Prefill, tokens per second Engine 25K 50K 100K 200K Basalt 6,492 7,662 8,666 8,609 Infernix 6,493 6,544 6,630 6,436 capy-flash 4,026 4,272 5,206 5,458 Strata 3,530 3,738 4,129 4,245 ninfer-ext 3,552 4,049 4,683 4,588 ninfer 2,080 2,236 2,308 2,248 llama.cpp 1,354 1,420 1,394 1,257 Time to first token, seconds Engine Struct Conv Prose 25K 50K 100K 200K Basalt 0.35 0.23 0.17 3.94 6.65 11.81 23.79 Infernix 0.42 0.24 0.16 3.94 7.79 15.43 31.81 capy-flash 0.88 0.50 0.29 6.35 11.93 19.65 37.52 Strata 0.45 0.29 0.20 7.24 13.64 24.78 48.24 ninfer-ext 1.05 0.47 0.21 7.19 12.59 21.85 44.63 ninfer 1.78 0.19 0.17 12.28 22.80 44.32 91.07 llama.cpp 1.09 0.75 0.31 18.87 35.91 73.39 162.86 Takeaways Infernix and Basalt are ahead of the rest. Against Infernix, Basalt decodes 10% faster on prose and 10% on conversation, struct is roughly a tie (354 vs 349). Prefill at 25K is a tie (3.94 s vs 3.94 s to the first token). From 50K up Basalt pulls away. capy-flash was interesting, was the closest of the rest on decode, it is built mostly for two GPUs, so this single GPU run is not its best case. llama.cpp ran with MTP off, so it's decode is not comparable with the others. This was just a reference point. How it was run Model: Qwen3.8 FN, NVIDIA's NVFP4 converted for each engine (table below). Hardware: RTX 5090, PL 400 W, 128 GB RAM, Linux. Sampling: greedy (temperature 0), thinking off, int8 KV cache for every engine (llama ran f16 kv). Cold: a fresh server per request, no warm up, one run each. Decode varies between runs, so any gaps under 5% are ties. Depth tests: a wikitext prefix of about 25K / 50K / 100K / 200K tokens, then "Write an essay on the Odyssey.", 512 tokens out. What each engine ran Engine Commit Weights Drafter Note Basalt 9c3692e7 NVIDIA NVFP4 experts bit-exact, dense and PLE table 8-bit (Q8_0) own MTP drafter (spec 8, shipped default) Fair pack built with Basalt's converter Infernix 7dbe99a3 Dense8 conversion: experts and FP8 PLE bit-exact, dense 8-bit own DFlash2 drafter Its README's best setup capy-flash 4e45286a Own convert: indexer BF16, GDN 8-bit own re-distilled MTP head Single-GPU profile, built mainly for two-GPU tensor parallel Strata 011c9e60 Experts bit-exact, dense Q8_0, PLE from an IQ4_NL GGUF Qwen BF16 MTP head repacked Basalt's upstream ninfer-ext bc168661 Own convert: MTP NVFP4, PLE re-scaled FP8, projections 8-bit own MTP ninfer 64d10fbf Patched to read NVIDIA's experts and fix the PLE scale own MTP Stock dev head did not build, stock converter needs the BF16 base llama.cpp 36cbf415 magnitudedev NVFP4 GGUF: experts bit-exact, dense and PLE Q8_0 none submitted by /u/jesdga95 [link] [comments]

Source: r/LocalLLaMA | 2026-10-23

Related