Lean-QwenFlash: Qwen3.8-Flash-Next (125B) running at ~20 tok/s on 16gb vram and 8gb ram via direct nvme SSD offloading

Hey everyone, Most current setups for Qwen3.8-Flash-Next (like Strata) keep all 24k experts pinned in system RAM (~48–64 GB) and compute cache misses on the CPU. That works great on high-spec workstations, but not on normal machines or laptops with limited RAM. A few weeks ago, I noticed that https://github.com/sybil-solutions/glm-flash-lite was running GLM-5.3 with surprisingly low memory footprints considering the model, way below anything I'd seen at those speeds, which made me realized that something about the design was likely really efficient and being slept on. So I adapted that to the much lighter Qwen3.8 Flash Next model, and the results were pretty impressive. Benchmarks (3.05bpw_h5_ng5, Q4 KV cache) RTX 4090 (with only 8 GB system RAM allocated and 16GB vram): Short context (8k): 28–33 tok/s prose, 19–22 tok/s code | ~1,080 tok/s prefill 100k context: 20.8 tok/s decode | ~1,536 tok/s prefill 262k context (full window): 20.0 tok/s decode | ~1,627 tok/s prefill RTX 3080 Laptop (16 GB VRAM, 32 GB RAM): 100k context: ~18–21 tok/s decode | ~650–700 tok/s prefill | Peak VRAM: 15.5 GB Setup instructions, model downloaders, and packing scripts are up on GitHub: https://github.com/MaxDevv/lean-qwenflash Big thanks to turboderp for ExLlamaV3, sybil-solutions for the original GLM offload implementation, and Alibaba for Flash-Next. submitted by /u/MaxDev0 [link] [comments]

Source: r/LocalLLaMA | 2026-10-25

Related