DeepSeek-V4-Flash-0731 on Bosgame M5 with RTX PRO 6000 Max-Q eGPU

Here are my numbers: Quant Size Layout Decode Prefill Draft acceptance UD-Q8_K_XL 150.8 GiB 20 layers CUDA0 / 23 ROCm0 + drafter 44.0 t/s 564 t/s 0.535 UD-Q4_K_XL 144.4 GiB 22 / 21 + drafter 48.4 t/s 585 t/s 0.532 UD-Q2_K_XL 90.2 GiB entirely on CUDA0, no drafter 59.5 t/s 1513 t/s — I let claude port the DSpark drafter from the closed PR to current main. https://github.com/haraldh/llama.cpp/tree/dspark-dsv4 UD-Q2_K_XL sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q2_K_XL --host 0.0.0.0 --port 8000 --temp 1.0 --top-p 1.0 --min-p 0.0 --no-mmap -fa on -np 1 --device CUDA0 -ub 2048 -b 4096 -c 200000 --cache-ram 65536 UD-Q4_K_XL sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL --host 0.0.0.0 --port 8000 --temp 1.0 --top-p 1.0 --min-p 0.0 --no-mmap -fa on -np 1 --device CUDA0,ROCm0 --split-mode layer --tensor-split 100,0 -ot 'blk.(2[2-9]|3[0-9]|4[0-2])ffn_(gate|up|down)expsweight=ROCm0' -md DSV4-Flash-0731-DSpark-draft-bf16.gguf --spec-type draft-dspark --spec-draft-n-max 5 --device-draft CUDA0 --spec-draft-p-min 0.3 -ub 2048 -b 4096 -c 200000 --cache-ram 32768 UD-Q8_K_XL As above with :UD-Q8_K_XL and the expert boundary two layers lower, since its dense part is 6.3 GiB larger: -ot 'blk.(2[0-9]|3[0-9]|4[0-2])ffn(gate|up|down)_expsweight=ROCm0' submitted by /u/backslashHH [link] [comments]

Source: r/LocalLLaMA | 2026-08-01

Related