I ran DeepSeek V4 Flash 284B + DSpark on one RTX PRO 6000. The drafter was faster in RAM than VRAM.

Hey guys, Just finished benchmarking DeepSeek V4 Flash 284B + DSpark on a single RTX PRO 6000 96GB. Short version: DSpark: ~15–17% faster generation on my coding workload On this setup, the DSpark drafter was faster in system RAM than VRAM q8_0 KV cache: 256K → 768K context with basically no decode-speed loss Best 9-turn coding run: 31.16 tok/s The 144.4GB model obviously does not fit in 96GB VRAM Hardware: Ryzen 9 9950X | RTX PRO 6000 Blackwell 96GB | 96GB DDR5 | Ubuntu Model: DeepSeek-V4-Flash-0731 UD-Q4_K_XL — 144.4GB My final target-model split: 21 expert layers → GPU 19 expert layers → system RAM Those RAM-resident experts have to be streamed during generation, so this setup is largely memory-bandwidth bound rather than compute-bound. 1. DSpark still helped with heavy CPU offload I compared the configurations at roughly the same VRAM usage: No drafter 26.52 tok/s DSpark in VRAM 29.86 tok/s +12.6% DSpark experts in RAM 31.16 tok/s +17.5% Because layers can only move in whole units, I think the honest result is ~15–17%, rather than treating 17.5% as an exact/general number. Matching VRAM usage matters here. My first comparison gave the DSpark configuration significantly more target-model weights in VRAM than the baseline, which exaggerated the gain. I reran it with memory usage matched as closely as possible. These are the rerun numbers. 2. The result I didn't expect: put the drafter in RAM This was probably the most interesting result. The DSpark drafter is about 10.15GB in Q8_0. Keeping it in VRAM means that VRAM can't be used for DeepSeek's expert layers. Moving the drafter experts into system RAM freed enough VRAM to keep three additional target-model expert layers on the GPU. I expected this configuration to lose. It didn't: Drafter in VRAM 29.86 tok/s Drafter experts in RAM 31.16 tok/s That's about 4.4% faster with the drafter experts in RAM. My interpretation is that what matters isn't only how many GB you move, but how often those GB are read. The drafter is used to generate a few speculative tokens per verification step. The target model's expert layers are needed continuously. So on this machine, spending that VRAM on more of the 284B target model was more valuable than spending it on the drafter. I'm very curious whether this flips on multi-GPU systems. 3. Three draft tokens was the sweet spot I also swept the number of speculative guesses: 2 guesses 32.72 tok/s mean 82.9% accepted 3 guesses 33.40 tok/s mean 74.7% accepted 4 guesses 31.32 tok/s mean 72.0% accepted 5 guesses 30.16 tok/s mean 67.2% accepted Interesting part: Higher acceptance rate != higher generation speed. Two guesses are accepted more often, but three guesses produce more useful accepted tokens per verification round. For this workload, 3 was the sweet spot. Going to 4 or 5 made things slower. 4. q8_0 KV cache seems basically free for decode here With the same n_cpu_moe 19 target split: 256K 89.5 GiB 32.36 tok/s 512K 91.6 GiB 32.24 tok/s 768K 94.2 GiB 32.12 tok/s 1M 93.3 GiB 29.22 tok/s So I could go from 256K → 768K without moving another expert layer out of VRAM, and decode speed barely changed. At 1M I finally had to move another expert layer, which is where generation speed dropped. I didn't want to assume the quantized KV cache was fine just because decode speed looked good, so I tested retrieval too. Needle retrieval worked at ~240K, and I also successfully retrieved a hidden fact from a document at around 900K tokens. So at least in these tests, q8_0 KV wasn't obviously damaging long-context retrieval. I will run more tests and upadte repo 5. Real coding workload, not only synthetic prompts For the main result I used a cumulative multi-turn coding session rather than repeatedly benchmarking fresh short prompts. The model keeps extending its own project across turns, so context grows naturally. I used turns 1–9. I intentionally excluded a later rewrite-style turn because it produced extremely high speculative acceptance and inflated the overall result. Final setup: 31.16 tok/s generation with: 19 target expert layers in RAM DSpark experts in RAM 3 speculative guesses q8_0 target KV q8_0 draft KV 6. Quick quality sanity check I also ran 30 LiveCodeBench problems. Result: 28/30 = 93.3% But please don't quote that as a general DeepSeek capability score. The problems were older and skewed easier than a recent hard-heavy slice. Both failures also hit the generation-length limit before outputting the final code block rather than simply producing an incorrect solution. I mainly used this as a sanity check that the quant/cache configuration wasn't obviously breaking the model. Final config --n-cpu-moe 19 --n-cpu-moe-draft 99 --spec-type draft-dspark --spec-draft-n-max 3 --cache-type-k q8_0 --cache-type-v q8_0 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --fit off -c 524288 --fit off matters because I wanted to control the split manually rather than let llama.cpp change it. Everything reproducible GitHub — Docker deploy, benchmark scripts, CSVs and raw per-turn results: https://github.com/lukaLLM/deepseek-v4-flash-dspark-rtx6000pro Video walkthrough + DSpark explanation/animations: https://youtu.be/EDls1Popv1o What I'd like to test next Has anyone tested drafter-in-RAM vs drafter-in-VRAM on a multi-GPU setup? Especially dual RTX PRO 6000s or RTX PRO 6000 + 5090. Has anyone found a better layer/drafter split for V4 Flash around 96GB VRAM? Any other engines or tuned builds worth benchmarking against llama.cpp? I'm also curious about any issues with quants? If anyone has a configuration you want compared, post and I could run it. I need to upgrade benchmarks too. Abused AI to make it more readable here from my draft. submitted by /u/FantasticNature7590 [link] [comments]

Source: r/LocalLLaMA | 2026-08-12

Related