I pushed Qwen3.8-27B limits again... Dflash2 - 134 tps on a RTX 3090

Edit: Title says 134 tps, it's actually 138 -- keep in mind my 3090 is power limited to 250w. Three days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak), and yesterday's update took it to ~114 tps single-user / ~1,000 tps at 64 concurrent. Today it's ~138 tps at default sampling on real chat prompts (up from ~124), 942 tps at 64 concurrent (re-measured today on the current stack), and the thing I'm actually happy about: a follow-up turn in a long chat now costs ~1 second instead of ~23. What we had: - fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with an own-output 40k draft head, GPTQ-int4 lm_head/MTP, split-KV verify attention, sampler patch, KVarN for 262k context Now added: - DFlash2 drafting. Inco published a block drafter for this exact model (5 layers, predicts 7 tokens in one non-autoregressive pass + a path selector). vLLM support is an unmerged PR on main, so I backported it to 0.27.1 and fixed what it silently relies on - including one real bug: 0.27.1 caches temperature-applied draft logits while main caches raw ones, so at 0<T≠1 the verify would have used the wrong proposal distribution. 2.8 → 3.3 tokens per step. - The drafter requantized to W4A16. It's 3.85 GB in bf16, which on a 24 GB card is a net loss (106 tps). GPTQ int4 with Hessians captured from the drafter's own inputs on real traffic: 1.19 GB, no greedy acceptance loss, and that's what turns it into a win. Ships as python fetch_dflash2.py. - Lookup-augmented drafting (my own idea - really happy about this one). A block drafter sees a 2,048-token window, but a long-context assistant spends much of its output reproducing what it was given... quoting a doc, repeating commands, rewriting a paragraph while keeping the code. Those tokens sit verbatim in the prompt, 20k tokens beyond what the drafter can see. So: one Triton kernel scans the request's own token history for the most recent occurrence of the last 6-12 generated tokens and proposes what followed. +29% tokens/step and 105 → 131 tps on "reproduce every command" work, +5% on ordinary chat, 0.075 ms per step. Stays exact and greedy never reads the draft distribution, and sampled positions get a point-mass q, which is a legal proposal for the rejection sampler. - Prefix caching for a hybrid model. vLLM keeps it opt-in for mamba/GDN hybrids, so by default every chat turn re-prefills the whole conversation. Turned on with --mamba-cache-mode align (the recurrent state resumes from the last cached block boundary): 24k-token document, turn 2+ goes 23 s → 0.85-1.35 s, same answers token for token. In batch mode it's just as big: 64 requests sharing a 5,820-token system prompt take 222 s → 16.9 s (median latency 95 s → 8 s). Costs ~14-16% of the KV pool. - 64k context with DFlash2, which needed an allocator fix: vLLM sizes a hybrid model's KV groups by the smallest layer bucket, so the drafter's 5 sliding-window layers made it pad the target's 16 attention layers to 20 and its 48 GDN layers to 50... 25% more memory per token, to pad the layers that weren't the problem. Padding the window group instead: 105 → 78 KB per token. Also made the V2 runner's CUDA-graph memory explicit; upstream it returns 0, so ~1.2 GB lands on top of whatever --gpu-memory-utilization you asked for. - Docker. docker compose --profile single up -d - image pins vLLM 0.27.1 + all patches, a prepare step downloads and requantizes the model, and verify.sh runs at build. Quality unchanged throughout (perplexity 8.09, GSM8K 96.5%) and speculative decoding is exact by construction and the state resume is exact too. Caveats worth stating: DFlash2 is best for 1-4 concurrent users (each request reserves 8 recurrent-state slots, so MTP wins again at 8+ concurrent), and its 2,048-token window means MTP is still slightly ahead on long-context free-form prose. Both modes are one env var apart. Repo: https://github.com/syv-ai/qwen38-27b-rtx3090 W4A16 DFlash2 drafter: https://huggingface.co/syvai/Qwen3.8-27B-DFlash2-W4A16 Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant I said last time that was probably the last update. Then someone released a better drafter and I found two features that were switched off by default, so here we are. Lets see what happens next... submitted by /u/iamMess [link] [comments]

Source: r/LocalLLaMA | 2026-08-19

Related