Qwen3.8-Flash-Next on 12GB VRAM - 65 tokens per second

A while ago I posted 15 tok/s output and 100-120 tok/s prompt processing with the IQ3_XXS quant on a 12GB RTX 5070 using llama.cpp. Since then I built my own inference engine for this one model and this kind of PC. The same IQ3_XXS now runs at ~65 tok/s output and ~430 tok/s prompt processing, and the 2-bit quants run faster still using RCO-GSQ quantization. Using: 64GB DDR5 (5600) 12GB RTX 5070 SFF (Gigabyte) Ryzen 5 7600 CPU Windows Output (tokens/s) on 128K context: Q2_0 (equivalent to unsloth Q3): 65.1 IQ2_XS (equivalent to unsloth Q4): 52.0 IQ3_XXS (equivalent to unsloth Q5): 44.8 Prompt processing (tokens/s) on 128K: Q2_0: 543 IQ2_XS: 472 IQ3_XXS: 414 Requirements: Q2_0 = 37.6GB minimum in RAM+VRAM IQ2_XS = 39.2GB minimum in RAM+VRAM IQ3_XXS = 47GB minimum in RAM+VRAM Vision encoder = 0.91GB additionally You can now one click install and run the engine with low cost hardware (currently only optimized for CUDA). GitHub: https://github.com/Niko1221/Strata Model: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF submitted by /u/KnownAd4832 [link] [comments]

Related

• Qwen3.8 Flash on 12GB VRAM - 15 tokens/s
• What settings do you use for running Qwen3.8-Flash-Next in llama.cpp?
• Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage
• Qwen 3.8 Flash Next q4_k_m, 130k context, q8 cache on 16GB VRAM ann 64GB RAM, 15-20 t/s on 4080
• Qwen3.8 Flash Next llama.cpp config tuning

Source: r/LocalLLaMA | 2026-09-24