Model Releases
My Qwen3.8-Flash-Next recipe for single GB10/DGX Spark, uses Intel AutoRound int4 quant and vLLM, fp8 ngram table offloaded to local SSD or external RDMA server. At mtp=3 c=1, code is ~47.5t/s, json is ~60t/s. Prefix cache is ON.
Repo: https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound I forked from blazux/qwen3.8-Flash-DGX and stole all the ideas from previous qwen 3.5 122b recipes, and created a hybrid model with
Repo: https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound I forked from blazux/qwen3.8-Flash-DGX and stole all the ideas from previous qwen 3.5 122b recipes, and created a hybrid model with (caveat: uncalibrated) int8 quant of lm_head, and also (caveat: uncalibrated) fp8 quant of GDN in/out projections, QSA q/k/v/o, shared expert. Did not see noticeable drop in quality, and currently I have been battle testing it for a day or 2, without crashes or model going haywire. vLLM c=1 pp is around 2000t/s in API call, dividing prompt tokens by wall time, and llama-benchy report does not reflect that well. llama-benchy peak mtp=3 tg (treat this as lower bound): c=1: 41.33t/s ± 1.89. c=8: 152.67t/s ± 7.32. c=16: 239.33t/s ± 3.30. At d=32768, c=1: 45.33t/s ± 5.44, c=8: 122.33t/s ± 7.41, c=16: 138.33t/s ± 10.62 Beware that technical info inside the repo might be AI slop, as I don't really have the ability to modify vLLM myself nor know what's going on, the same goes to the model quality, too. But the startup script is verified and is likely usable. Also, MTP is currently introducing a lot of extra TTFT in concurrency case, which might be a problem for you. I am looking forward to DSpark/DFlash2 models. If you have a NAS with 100G+ connection and 64GB+ ram sitting next to your GB10 device, the branch "magi" may be interesting to you, since it uses external RDMA server that basically eliminates the extra latency caused by ngram/PLE lookup IO, adding 3t/s tg to everywhere. https://preview.redd.it/k184cfrpypmh1.png?width=812&format=png&auto=webp&s=9e2d60338633e573f6d3d06015ac2c79d073bfb2 Repo: https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound submitted by /u/Saren-WTAKO [link] [comments]
Related
- Qwen3.8-Next-Flash up to 240t/s on single rtx 6000 pro
- What is the best intelligence/stable model currently for a single GB10/DGX spark?
- [[megathread-qwen38-flash-next---release-day|[Megathread] Qwen3.8-Flash-Next - Release Day]]
- Experience report - Qwen 3.8 Flash Next on memory rich, GPU poor setup
Source: r/LocalLLaMA | 2026-08-31