capy-flash v0.7.1: Qwen3.8-Flash-Next (125B MoE) now runs in 64 GB of RAM (234 tok/s on two 16 GB cards), and Qwen3.8-27B 3-bit fits on a single 16 GB card (174 tok/s)

I've been building capy-flash, an open-source inference engine for running Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on consumer RTX 50 cards. Until now it needed my setup (two 16 GB cards and 96 GB of RAM) or better. v0.7 is about smaller machines: start with what you have, add a second card later, same engine and same model files. What's new Flash-Next on 64 GB of RAM, one or two GPUs. The routed experts are ISTA-DASLab's GSQ-RCO i-quants (~3 bit, copied as they are, not re-quantized): 48 GiB in RAM instead of 65 for NVFP4. The hot experts sit in VRAM, the rest are computed on the CPU. Peak RAM ~57 GB. Qwen3.8-27B on one 16 GB card, 3-bit GSQ weights (ISTA-DASLab), 64k context, vision, MTP. 13.9 GB of VRAM. AVX2 CPUs (Zen 2/3, Intel client) now work, and Windows/WSL2's page-locked memory limit no longer stops the server from starting. Guards against empty replies (the model stopping inside its reasoning), on by default. Speed (LocalMaxxing reasoning-v1, one request, MTP, verified runs; RTX 5070 Ti 16 GB, Ryzen 9 9900X) 1× 16 GB GPU 2× 16 GB GPUs Qwen3.8-Flash-Next, 3-bit experts 121 tok/s Qwen3.8-27B, 3-bit 174 tok/s One GPU or two? If you have a single 16 GB card, Strata is probably already the better choice for Flash-Next today. capy-flash is at its best with two cards, where it adds tensor parallelism and a VRAM expert cache shared by both GPUs. Quality: 3 bits cost something on the 27B: GSM8K is on par with the NVFP4 model, MMLU-Pro drops from 0.78 to about 0.72. With two cards, the NVFP4 27B is still the better choice. Flash-Next's 3-bit experts are on par with NVFP4 on both. Try it curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash -s -- --check curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash -s -- --model flash-next-64gb # or --model 27b-16gb for the 27B on one card Repo: https://github.com/ValerioDolci/capy-flash · release notes: https://github.com/ValerioDolci/capy-flash/releases (v0.7.0 and v0.7.1) Weights: Flash-Next 64 GB · 27B 3-bit Thanks to everyone who opened issues and PRs (several v0.7 features came from them), and to ISTA-DASLab for the quantized weights. Questions and numbers from other machines are very welcome. submitted by /u/FeydRowan [link] [comments]

Source: r/LocalLLaMA | 2026-10-22

Related