Model Releases
Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.
Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users
Write-up, serve command and raw data: https://github.com/lukolszewski/vllm This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my earlier llama-multigpu performance fixes https://www.reddit.com/r/LocalLLaMA/comments/1x023z4/qwen38flashnext_on_6x3090_6x4090_without_nvlink/ Upstream vLLM does not start this model on that machine at all: the model refuses --pipeline-parallel-size > 1, TP has to be 1/2/4/8 so six cards are useless for it, and a 24 GB card cannot hold it at TP=1. Putting the 102 GB n-gram (PLE) table on SSD has just showed up as PR on upstream today (keeping it in RAM is the typical setup), and fp8 KV crashes on Ampere. Mashine is a consumer PC, no epyc/threadripper, no P2P, still pretty usable (6x3090, 7950X, 187 GB RAM, four cards behind Thunderbolt), GPTQ-INT4 checkpoint (nvfp4 was slower), one streaming request per cell, tokens/s: workload ctx llama.cpp-multigpu vLLM, this branch (int8 KV) prefill, 1 session 250k 2111 6574 time to first token, 250k prompt 250k ~118 s 37.8 s decode, 1 session 250k 33.7 57.6 decode, 5 sessions, each / sum 250k ~30 / ~150 (at 200k) 37.9 / 190 decode, 7 sessions, each / sum 250k n/a (5 slots) 29.4 / 206 262k contexts resident in GPU KV 5 7.33 host RAM free with the server idle 154 GiB 141 GiB Things that did not pan out: NVFP4 checkpoints load with no changes but are 10-12 % slower than GPTQ on these cards, both go through Marlin weight-only, there is nothing for FP4 to accelerate on Ampere. And per-user decode under pipeline parallelism falls with the user count (57 -> 44 -> 38 -> 29 at 1/3/5/7 users) while llama.cpp's grouped decode stays nearly flat (34 -> 30 at 5); the sums are similar at five users. So: vLLM for prefill, single-user speed and seat count; llama.cpp if you want every user to see the same speed (I posted a similar set of fixes for llama earlier). The AMD side exists too: u/alanoo81 ran the model at PP=3 on Radeon V620s earlier and that is where the PP idea came from; on current upstream only the guard removal is still needed and the branch keeps a partition check instead. If you are on AMD start there. Caveats: one model family, NVIDIA only, tested on exactly one machine (plus kernel unit tests on a 5060 Ti). Batched outputs are not byte-identical to solo runs, vLLM's usual batch nondeterminism. Load to healthy is 4-5 minutes. As with the llama.cpp work, the code was written with an LLM and validated by measurement and output checks (needle tests, kernel outputs against oracles), not by review; each change is one commit, take what you need. This is a fixing branch, not a fork I intend to maintain. submitted by /u/flynth92 [link] [comments]
Source: r/LocalLLaMA | 2026-10-12