DeepSeek-V4-Flash-0731-UD-Q3_K_XL 3x3090 test results

For anyone interested, here are the llama-bench results on 3 bit K_XL quantization. I think this could be pushed further but no luck so far. CURRENT RESULTS: full moe offloading Prefill suffers 116 --> 72 t/s , generation 8-->14 t/s compared to previous case with no moe offlloading. ./llama-bench -m /home/ckitapp/llamacpp/modelsmain/unsloth/ds4/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -ngl 99 --split-mode layer -p 512 -n 128 -r 5 --n-cpu-moe 99 ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72364 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24116 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB | model | size | params | backend | ngl | n_cpu_moe | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | --------------: | -------------------: | | deepseek4 ?B Q3_K - Medium | 119.40 GiB | 284.33 B | CUDA | 99 | 99 | pp512 | 72.20 ± 14.26 | | deepseek4 ?B Q3_K - Medium | 119.40 GiB | 284.33 B | CUDA | 99 | 99 | tg128 | 13.94 ± 0.43 | PREVIOUS RESULTS: fit 21 layers to gpus first, dump the rest to ram Command ./llama-bench -m /home/user/llamacpp/modelsmain/unsloth/ds4/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -ngl 21 --split-mode layer -p 512 -n 128 -r 5 Output CUDA Initialization ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72364 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24116 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Benchmark Results Model Size Params Backend NGL Test T/S deepseek4 ?B Q3_K - Medium 119.40 GiB 284.33 B CUDA 21 pp512 116.04 ± 27.64 deepseek4 ?B Q3_K - Medium 119.40 GiB 284.33 B CUDA 21 tg128 7.71 ± 0.09 Build: e3546c794 (9976) System Memory Total Used Free Shared Buff/Cache Available Mem 122Gi 7.9Gi 1.2Gi 165Mi 114Gi Swap 0B 0B 0B NVIDIA-SMI Status +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A | | 58% 55C P2 151W / 390W | 22696MiB / 24576MiB | 35% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3090 Off | 00000000:31:00.0 On | N/A | | 30% 53C P2 120W / 350W | 20193MiB / 24576MiB | 9% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA GeForce RTX 3090 Off | 00000000:6C:00.0 Off | N/A | | 32% 55C P2 135W / 420W | 17811MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ submitted by /u/consultkitapp [link] [comments]

Source: r/LocalLLaMA | 2026-08-01

Related