I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases.

Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFlash 1 numbers on Qwen 3.6 27B for 3 days. One RTX PRO 6000, concurrency 1, about three days of runs. The interesting result isn't the biggest number I measured. It's where n-gram actually helps and where it doesn't. Short version: DFlash 2 alone: 2.26x on 100 real LiveCodeBench problems (67.97 → 153.91 tok/s, inter-token latency 14.27 → 6.02 ms), natural stop, nothing forced. That is the headline. Costs +2.7 GB VRAM. DFlash 2 + one n-gram lookup table (ngram-map-k4v): 4.68x on the build phase of an 18-turn coding session (65.1 → 304.9 tok/s). Adding the second table (ngram-mod) made it slower, 3.77x. In July, with DFlash 1, stacking both was the winner. I did not expect that to flip. The same n-gram flag is +52% on a synthetic benchmark, +1% on LiveCodeBench and -30% on prose. The +52% is the harness degenerating, do not quote it. The recommended --spec-draft-n-max 7 is past the peak. 5 gave roughly 11% more on 8K coding prompts. 7 is also a hard cap (block_size 8), anything above is silently clamped. --spec-draft-p-min does nothing on DFlash 2. The DFlash 2 code path in common/speculative.cpp never reads it. I also measured 8.47x in a synthetic test. I nearly used that as the headline. It was mostly benchmark garbage caused by the model falling into a repetitive loop. Setup (the parts that matter for reproducing) Target ggml-org/Qwen3.8-27B-GGUF:Q4_K_M (18 GB). Drafter incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (1.1 GB). MTP sidecar mtp-Qwen3.8-27B-Q8_0.gguf (3.0 GB). Reasoning off. llama.cpp b10498 built from PR #27342 (commit 5ecbe1ac), CUDA 13.3. The PR build matched upstream b10499 within 0.3% on a non-speculative baseline (checked at 512 and 4K only). RTX PRO 6000 Blackwell 96 GB, Ryzen 9 9950X. -c 262144, f16 KV, -fa on, -ngl -1, drafter fully on GPU. Concurrency 1 everywhere. Greedy (temperature 0) for everything except the multi-turn coding harness, which runs model-default sampling with no seed (more on that below). One server on the GPU at a time (flock), fresh container per config, card cooled to 45 °C between configs and 60 °C between context sizes. 11.6 hours of telemetry, zero throttle events, so the card sits on its power limit, not a thermal one. Full context 1. DFlash 2 more than doubles real coding throughput, and beats DFlash 1 at the same draft width for half the VRAM 100 LiveCodeBench problem statements replayed in the same order, streaming, no ignore_eos, no min_tokens, no max_tokens. Every answer ends where the model ends it. https://preview.redd.it/ina7l4wsfzkh1.png?width=1700&format=png&auto=webp&s=f803f3328a146063e8f19944b265838e63c5c546 tok/s vs own base ITL wall clock Qwen 3.8 27B, no speculation 67.97 1.00x 14.27 ms + DFlash 2 (n=7) 153.91 2.26x 6.02 ms + DFlash 2 + both lookups 155.83 2.29x 6.11 ms Qwen 3.6 27B, no speculation 67.75 1.00x 14.34 ms + DFlash 1 (n=7, matched) 135.34 2.00x 6.93 ms DFlash 1 was re-run at n=7 because comparing it at its own maximum of 15 would measure the cap, not the drafter. At matched width DFlash 2 is ahead, 2.26x vs 2.00x against each model's own baseline, with probe acceptance of 60% vs 48%, and it costs +2,720 MiB where DFlash 1 cost +5,554 MiB in July. Part of that memory gap is a quant choice (Q4_K_M 1.1 GB drafter vs Q8_0 1.8 GB), not architecture. Two things to be careful with. The cross-generation rows are not a controlled A/B: different model, different target quant, different drafter quant (and the drafter quant works against DFlash 2, not for it). Compare the speedups, never the absolute tok/s; the two baselines landing 0.3% apart is luck. Claim vs measured: Inco AI quote 2.7x to 3.4x at batch size 1 on SGLang for this model. I got 2.26x on llama.cpp on single-turn coding so it depends on task and engine it will probably get better soon with updates to engines. 2. One lookup table on top of DFlash 2 is the best stack. Two is worse. That is the opposite of DFlash 1. The n-gram drafters copy spans that already exist in context, so single-turn prompts are their worst case (+1.2% above, and the median actually says -2.7%). The case that matters is working on a code base, so I drive 18 fixed prompts as one cumulative conversation: turns 1-9 build a Gradio chat client for llama.cpp feature by feature, turns 10-18 maintain it (re-emit the file, docstrings, renames, a bug, a refactor, tests, README). https://preview.redd.it/v106usk5gzkh1.png?width=1700&format=png&auto=webp&s=04d49cda970411e65b000870e3c8bb718ea580bf stack --spec-type build 1-9 tok/s vs base all 18 accept (build) drafts/tok no speculation - 65.14 1.00x 56.95 - - DFlash 2 alone draft-dflash 181.89 2.79x 177.53 66.4% 1.24 DFlash 2 + k4v draft-dflash,ngram-map-k4v 304.92 4.68x 343.52 64.2% 1.41 DFlash 2 + both lookups draft-dflash,ngram-mod,ngram-map-k4v 245.84 3.77x 306.04 55.6% 1.59 DFlash 2 + mod draft-dflash,ngram-mod 229.37 3.52x 313.46 58.6% 1.48 lookups only, no drafter model, 0 VRAM ngram-mod,ngram-map-k4v 133.00 2.04x 170.54 59.5% 1.00 Read the build column. Turn 10 is "show me the complete final app.py", which is ~99% draftable and inflates every speculative method. Over all 18 turns the k4v stack reads as 6.03x, a real number about the easiest thing you can ask a copying drafter to do. I expected the July result to repeat: with DFlash 1, draft-dflash,ngram-mod,ngram-map-k4v was the winner at 6.01x and ngram-mod did almost all of the n-gram work. Instead, on DFlash 2 the k4v table alone wins, mod alone is the weakest stack, and both together are slower than k4v alone. It could be draft tokens number or early implementation we will see. DFlash 1 had max 15 draft slots, DFlash 2 has 7, and two lookup drafters crowd each other out of them. 3. The same one-line change gives four different answers, and the synthetic one is wrong Same DFlash 2 server, same weights, append ngram-mod,ngram-map-k4v to --spec-type, run everything again: workload DFlash 2 alone + both lookups change editing code, 18-turn session, turns 1-9 181.89 245.84 +35% forced-length synthetic, 4K in / 4K out (medians) 176.57 267.82 +52% one-shot coding, LiveCodeBench x100 153.91 155.83 +1.2% writing fresh prose, one request 158.9 111.6 -30% https://preview.redd.it/vhr4stn8lzkh1.png?width=1700&format=png&auto=webp&s=920e0c8137a88f6f74ffd597823a67e8fe0e2d12 The synthetic bench from aiperf is inflated by its own harness. It passes ignore_eos and min_tokens, forces the model past its natural stop until it loops, and a lookup drafter copies loops perfectly. Carried to 36K the same harness says DFlash 2 + lookup is 8.39x (498 tok/s). On 100 real prompts that stack was worth +1.2%. 8.39x is the kind of number that you could get but in very specific usecase. Prose is the opposite corner: "Write a very long story", nothing in context to copy, the tables burn draft slots on guesses that never land, acceptance 54% → 32%. That row is a single instrumented request, a probe, not a run. Practical consequence: turn the lookup drafters on for iterative coding and anything that re-emits its own context, leave them off for one-shot prompts and creative writing. They cost zero VRAM and zero prefill, so this acceptance loss is their only cost. 4. The recommended draft width is past the peak, and 7 is a hard cap anyway https://preview.redd.it/usnvqpyalzkh1.png?width=1920&format=png&auto=webp&s=035c2ef4985f7e132daeb2d47f9eca182587ed2b 16 coding prompts per width at 8K tokens from livecodebench, cache_prompt false so every request pays a cold prefill: I use livecodebench and cut it to the size to measure worst case here. n_max DFlash 2 tok/s accept MTP tok/s accept 2 140.56 82.6% 133.42 79.5% 3 158.06 72.6% 154.53 77.9% 4 174.60 72.0% 159.07 71.7% 5 187.13 70.4% - - 6 184.60 67.0% 154.64 62.9% 7 168.06 59.7% - - Running the model card's 7 leaves roughly 11% on the table. An earlier 8-prompt sweep put the optimum at 6 rather than 5, so call it 5-6; both sweeps agree 7 is past the peak. And you cannot go above 7: the draft GGUF carries dflash.block_size=8, llama.cpp clamps n_draft_max = block_size - 1, logs a warning and uses 7. Some cells rest on only 3-7 valid generations of 16 (the truncated prompts sometimes make the model emit EOS immediately), so treat the exact peak as soft. MTP on this model peaks at n=4 and flattens near 2.5x across context. Qwen 3.8's sidecar declares nextn_predict_layers=1, one trained head, against DFlash 2 reading five target layers. That is a property of this sidecar, not of MTP as a method; Qwen 3.6's had eight heads. 5. Long context: the drafter gets relatively cheaper and absolutely more expensive https://preview.redd.it/wnunzgxelzkh1.png?width=1700&format=png&auto=webp&s=de18ce5fa2f47faec167d414685160a335f2be9b The usual complaint is that speculative decoding falls apart at long context. Two costs hide in that sentence. Prefill, where the drafter has to read the prompt too, I could measure. Decode at those depths I could not (see caveats). Cold prefill, 12 prompts per depth: prompt depth prefill tok/s, none prefill tok/s, DFlash 2 speed kept extra wait 1K 3,506 2,656 0.76 +0.09 s 4K 3,845 3,164 0.82 +0.23 s 16K 3,639 3,162 0.87 +0.68 s 64K 2,867 2,588 0.90 +2.46 s 128K 2,239 2,056 0.92 +5.20 s Relative to baseline the tax shrinks with depth (24% down to 8%). In seconds it grows, +0.09 s to +5.20 s. Both readings are true; quoting only the first is the flattering half. The prefill cost is repaid in 13 / 30 / 69 output tokens at 1K / 4K / 16K, so any real answer clears it, but someone on a 128K prompt does wait five seconds longer for the first token. The lookup drafters cost nearly nothing here (0.994-0.997 of baseline), which doubles as the control that the gap is the drafter and not drift. MTP's tax is smaller (0.83 at 1K vs 0.75). On the forced-length synthetic decode sweep DFlash 2 goes 1.59x → 2.62x → 2.96x → 3.55x at 512 / 4K / 12K / 36K while the baseline falls 67.6 → 59.3 tok/s. DFlash 1 at its own max of 15 did 4.44x at 36K on that harness in July (higher still when re-measured this month), and at matched width 7 it did 3.71x. I expected the new drafter to win everywhere. It does not: it wins on real prompts at equal width, and loses the synthetic long-context race to the old drafter with more slots, because it is capped at 7. 6. --spec-draft-p-min is a no-op on DFlash 2, and buys nothing on MTP either Adaptive draft truncation should let the drafter stop a block early when it is unsure to save resources. There are more advance method form DeepSeek Dspark paper but they just landed on vLLM. I logged draft width and cycles per second, not just tok/s: drafter p_min tok/s accept draft width cycles/s DFlash 2 0.00 195.3 71.6% 6.998 32.51 DFlash 2 0.85 171.1 60.5% 6.998 32.69 MTP 0.00 161.7 90.4% 3.001 43.54 MTP 0.85 155.7 97.6% 2.642 43.52 Draft width is identical at 0.00 and 0.85 on DFlash 2. common/speculative.cpp has four drafter implementations: draft_simple, draft_eagle3, the DFlash 1 branch and draft_mtp honour p_min; the is_dflash2 selector branch never consults it, because it reads a selector lattice rather than a probability. The server still prints the flag in its startup banner, so a log-based check passes while nothing happens. The 12.4% throughput drop in that row is the text, not the flag: the server did identical work (cycles/s within 1.5%), the sampled output just accepted fewer of the same seven tokens. I nearly published "p_min costs 12%". On MTP the flag works exactly as documented (width 3.00 → 2.64, acceptance 90% → 98%) and throughput goes nowhere, +0.8% at best against a 4.1% noise floor. What I would run Iterative coding, agents, anything that re-emits its own context: --spec-type draft-dflash,ngram-map-k4v --spec-draft-n-max 5 One-shot prompts and Q&A: --spec-type draft-dflash --spec-draft-n-max 5 Prose: DFlash 2 alone, no lookups. Keep the KV cache at f16 for now or test it it will be probably stable soon but on last version there were issues and I use default. Ignore --spec-draft-p-min. git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp git fetch origin pull/27342/head:pr-27342 git switch pr-27342 # NVIDIA CUDA cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build -j # Apple Silicon cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON cmake --build build -j # Best measured config: iterative coding, agents, anything that re-emits its own context # (4.68x on the multi-turn coding session vs 2.79x for DFlash 2 alone) ./build/bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M --spec-type draft-dflash,ngram-map-k4v --spec-draft-n-max 5 -ngl -1 --spec-draft-ngl all -fa on -c 262144 --parallel 1 --jinja --reasoning off --no-mmproj --host 0.0.0.0 --port 8000 --alias qwen38-dflash2-k4v Caveats, all of them No accuracy measurement this time. Greedy speculative decoding is output-lossless by construction and the July study measured it (MATH-500: 87 vs 86 of 100, then 440 vs 435 of 500), but that was Qwen 3.6 with DFlash 1 and I did not rerun it here. Only a LiveCodeBench smoke test. No deep-context decode. At 64K and 128K every generation returned one token and stopped, so prefill at those depths is valid and decode does not exist. Only one experiment was ever repeated (the p_min controls). Everything else is one sample. The spreads from those repeats, 4.1% and 14.7%, are the noise floor for this whole post. The multi-turn harness can measure the wrong thing. It declares no tools, but under default sampling the model sometimes answers with a <tool_call> block and waits for a result that never comes, and that session comes out fast because tool-call XML is predictable. One run did exactly that (1,134 tokens where its siblings produced 50K-73K), got caught by its token count and was rerun. This is one workload family (coding) on one machine at concurrency 1. A 96 GB card is not what most of you run. The drafter is 1.1 GB and nothing in the KV math depends on the card, so I expect the shape to hold on a 24-32 GB card with a smaller context, but I have not measured it. DFlash 2 is a PR build. Numbers can move when it merges. Resources Repo (both studies, this one on top): https://github.com/lukaLLM/DFlash2_Qwen3.8_3.6_27B_LlamaCPP Video walkthrough (the first half explains the mechanism, path selector and the convolution the rest go even more deeper into the scores etc. ): https://youtu.be/RBlRTUwJMI4 One-click setup, builds the PR image, downloads the models, smoke tests and leaves a server running: ./scripts/setup_dflash2.sh --arm dflash2_ngram (arms: base, dflash2, mtp, ngram, dflash2_ngram). Compose file docker/docker-compose-qwen38-dflash2.yaml; ablate with LLAMA_SPEC_TYPE=... and LLAMA_SPEC_N=5. Reproduce the whole study in order: ./scripts/run_all_benchmarks.sh, then run_matched_n.sh, run_context_scaling.sh, run_bench_ngram.sh, run_nmax_redo.sh, run_pmin_agentic.sh. Every number in one machine-readable file: benchmark/results_summary.csv (TABLE 8-14 are this study). Raw artifacts under artifacts/q38_*/, the thermal log in artifacts/thermal/, quarantined runs and the reasons in artifacts/_suspect/README.md. Long-form write-up with the charts: report/dflash2-report.html in the repo. https://inco.ai/blog/dflash2/ the blog Previous posts: DFlash 1 in July https://www.reddit.com/r/LocalLLaMA/comments/1uq0h4o/i_tested_freshly_merged_dflash_in_llamacpp_on/ and the n-gram stack https://youtu.be/zNUoHONUHGk AI was abused in editing this post. Questions: Has anyone run DFlash 2 on SGLang or vLLM at concurrency 1 with this model? I want to know whether the 2.7-3.4x claim holds there and how much of the gap to my 2.26x is the engine. Anyone on a 4090 or 5090 with a 24-32 GB budget: does n=5 still beat 7 for you, and where does the k4v-only stack land on your own multi-turn coding? Has anyone tried some other combinations that I didn't think of? submitted by /u/FantasticNature7590 [link] [comments]

Source: r/LocalLLaMA | 2026-08-22

Related