Qwen-3.8-Flash-Next on 1x RTX 5090: TG=50 t/s, PP=2300 t/s - with FreeToken

I was somewhat disappointed with the performance of Qwen 3.8 Flash Next on my single RTX 5090 using llama.cpp. One issue is that llama.cpp still has no Expert Caching implemented for MoE models. There are various PRs and discussions (see here, for example), but nothing is merged yet. Then I stumbled upon FreeToken which claims to "unlock datacenter-class intelligence on the hardware you already own". With that I am able to run 3.8 Flash Next with TG ~ 50 t/s (around 40-60 depending on how well the cache works), pretty stable even over longer context. PP ~ 2300 t/s which is actually useable, thanks to expert caching. (Stats are from FreeToken logs) Launched with: --model --served-model-name qwen3.8-flash-next --moe-strategy auto --moe-cache-auto --max-running-requests 2 --expert-load parallel --max-seq-len-override 229376 --kv-reserve-tokens 229376 --reasoning-parser qwen3 --tool-call-parser qwen3_coder Specs: RTX 5090, 128 GB DDR5 6000 (96 GB should work fine as well), Intel 270K Plus Model used: nvidia/Qwen3.8-Flash-Next-NVFP4 (one of the suggestions from here - Be sure to convert it to FreeToken format otherwise each start is slow) It works with other MoE models as well. So even if you can only run something like Qwen 3.6 35B-A3B and have some spare VRAM which can be used to cache experts, give it a try. Still an early project - vision just got implemented. There is not KV quantization yet, nor MTP. submitted by /u/dir3ctly [link] [comments]

Related

• Qwen3.8-Flash-Next on 2x3090 + DDR4: 17 → 25-29 t/s decode with the expert cache PR
• UPDATE: Qwen3.8-Flash-Next on 2x3090 + DDR4 (Part 2): 25-29 -> 37-41 t/s decode (UD-Q4_K_XL + expert cache + MTP), plus a branch you can build
• What settings do you use for running Qwen3.8-Flash-Next in llama.cpp?
• 50% tg increase with offloading 'hot' experts to VRAM
• 10%+ performance improvement on MoE ssd-streaming with expert-lookahead

Source: r/LocalLLaMA | 2026-09-19