3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed

The post describes some experiments I had while trying to desperately run deepseek-v4-flash-0731 4 bit+ quants on my machine which is supposed to support only q2 quants of the model, a or 2.xx bpw quants at best. Long story short , I wanted to have my tgs in the high twenties and my prompt processing at least in the 300s with 156K context to consider running it locally as my daily driver (hermes, coding and so on) First I describe my machine so you are in the picture - people seem to ignore the importance of putting your exact hw config but a small difference there can give huge performance variation - : intel gen 14 i5 with 20 usable pcie5 channels, ddr 5 128 GB total = 2 x 48 + 2 x 16 at 4400 , 2 RTX3090 + 1 RTX3060, 2 DRAM-less SSD's that can in theory read at 4.5 Gb/s The best I could get with the initial 4bit+ quants with the sidecar models was 7 tgs and around 20 pp, after pinning some layers to GPU's in the most optimal way I could and after implementing a redundant sidecar so cpu can read in parallel from my 2 ssd's at the same time , but it was not really helpful I cloned after that leloch's llama.cpp and I could get in the lower teen's tgs with AtomicChat 3bit quants But I wanted to run the 4 bit quants as they have mostly the original bit-identical experts. The issue was that they are bigger than my RAM (140+ GB). So , with the way llama.cpp is designed, running them would cause quite some cache misses reading from my not so fast SSD's . and I was back to less than 10 tgs. For me it was a bit "strange" that I have to go fetch from the SSD every token when my RAM + VRAM >> total model weight. So I was telling myself , even if i keep some space for cache and the scratch memory used for temporary ops and such, I should still be able to squeeze the total model in RAM + VRAM , and not have to go back to the SSD. I just would need to mlock the memory of the experts, so they are always in a RAM kind of memory, and no SSD read is ever needed after initial model load. Except it was not that simple (hint: kernel page caching) So what I ended up doing is just getting rid of the redundant expert caching between RAM and VRAM : i.e. if a hot expert is promoted to VRAM , its memory cache is unlocked, so kernel can load something else in its place. And when an expert is demoted from VRAM, it will not be immediately read from SSD, but the first time it is needed, it is read from the SSD and mlocked. This means that the same expect is never in RAM and VRAM at the same time. After this (2 patches) , and adding the dflash drafter AND pinning the dflash into host RAM, I was able to get low to mid twenties of tgs , especially if generation is more than 1000 tokens. This involved quite some tuning of different params, including VRAM cache budget. It was not bad, at least for interactive sessions. BUT, the prompt processing was low : less than 60 tokens per second. You can imagine how long it would take to start with a 30K initial prompt ... I tried playing with batch sizes, cache size .. the prompt processing never moved. Than I tried something I believe is novel : loading a lower quant just for the prompt processing phase, if the prompt is long enough that what we gain from speed of processing by a lower quant model is much more than what we loose when unloading-original-model + loading lower quant + reloading original-model + initial not so hot expert cache because 2 different models are used in the 2 phases. Studies showed that even starting from a lower quality initial cache, smart models recover quality as the decode becomes longer. (I read the title and introduction of one such study but do not have it in front of me now) So I tried with the IQ_2M from AtomicChat and in some configurations it could give me near 200 prompt processing, but even with all the optimization and "stitching" I added the overall prompt handling (processing + decode) did not improve that much in the end unless the prompt was 30K or more, because the decode was always starting with very low tgs for the first 1000 tokens or so after a prompt processing done by the IQ_2M . I tried to "transfer" the hot expert cache (just the ID's though) between the 2 modes but the initial tokens from decode were always slow, because the cache actually needed to be rebuilt from scratch. May be the next idea is just to start a prompt processing remote service (should be much cheaper than normal api, as you only send the prompt if it is long enough, get the cache continue decode locally) anyway, I share the llama.cpp clone, with my 2 branches on top of leloch's work https://github.com/oussemah/llama.cpp/tree/moe-cache-ousemma - moe-cache-ousemma branch does not have the prompt processing specifi model logic, that s the one that gives 20 tgs and aroudn 45 pp - moe-cache-ppswap branch has the prompt processing model logic hopefully someone can be inspired to try some new ideas or just use it on a better hardware and get better results The main model is : unsloth UD-Q4_K_XL The prompt processing I used with the second branch is : AtomicChat/AD-IQ2_M Sample command for first branch sudo 'ulimit -l unlimited && GGML_CUDA_MOE_CACHE_RESERVE_MB=512 GGML_CUDA_MOE_CACHE_ADMIT_AFTER=1 GGML_CUDA_MOE_CACHE_INSERTS=256 GGML_CUDA_MOE_CACHE_QUEUE_MB=2048 GGML_CUDA_MOE_CACHE_MODE=on GGML_CUDA_MOE_CACHE_BUDGET_MB=40000 GGML_CUDA_MOE_CACHE_BUDGET_MB_DEVICES=0:11800 GGML_CUDA_MOE_CACHE_STATS=1024 GGML_CUDA_MOE_CACHE_MLOCK=1 GGML_CUDA_MOE_CACHE_ELITE_PCT=60 GGML_CUDA_MOE_CACHE_DEMAND_DECAY=4096 ./llama.cpp/build/bin/llama-server --host 0.0.0.0 --port 8080 -m /home/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/fbbb5b93fb787c21338159b0af3318bb3f4d9768/UD-Q4_K_XL/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf -md /home/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf --spec-type draft-dspark -ngld 0 -td 20 --spec-draft-n-max 5 -c 167936 --parallel 1 --split-mode layer -t 16 -tb 20 --cache-type-k q8_0 --cache-type-v q8_0 -b 4096 -ub 4096 --flash-attn on --moe-cache auto --jinja --temp 1.0 --top-p 0.95 --reasoning on -lv 4 --reasoning-format deepseek --slot-save-path /home/data/ --alias DeepSkee-v4-Flash-0731-UD-Q4_K_XL -lv 4 ' Sample command for the prompt-processing-model branch : sudo 'ulimit -l unlimited && GGML_CUDA_MOE_CACHE_RESERVE_MB=512 GGML_CUDA_MOE_CACHE_ADMIT_AFTER=1 GGML_CUDA_MOE_CACHE_INSERTS=256 GGML_CUDA_MOE_CACHE_QUEUE_MB=2048 GGML_CUDA_MOE_CACHE_MODE=on GGML_CUDA_MOE_CACHE_BUDGET_MB=40000 GGML_CUDA_MOE_CACHE_BUDGET_MB_DEVICES=0:11800 GGML_CUDA_MOE_CACHE_STATS=1024 GGML_CUDA_MOE_CACHE_MLOCK=1 GGML_CUDA_MOE_CACHE_ELITE_PCT=60 GGML_CUDA_MOE_CACHE_DEMAND_DECAY=4096 LLAMA_EXPERT_SWAP_NO_PRELOAD=0 LLAMA_EXPERT_SWAP_PREFETCH=1 LLAMA_EXPERT_SWAP_MLOCK=1 /home/ous/infra/llama.cpp/build/bin/llama-server --host 0.0.0.0 --port 8080 -m /home/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/fbbb5b93fb787c21338159b0af3318bb3f4d9768/UD-Q4_K_XL/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf --prompt-processing-model /home/.cache/huggingface/hub/models--AtomicChat--DeepSeek-V4-Flash-0731-GGUF/snapshots/5f8e5b74544ad821d71aedf658c2b8acdecd4b2b/AD-IQ2_M/DeepSeek-V4-Flash-0731-AD-IQ2_M-00001-of-00004.gguf --prompt-processing-min-tokens 8192 -md /home/dspark-DeepSeek-V4-Flash-0731-Q8_0.gguf --spec-type draft-dspark -ngld 0 -td 20 --spec-draft-n-max 5 -c 167936 --parallel 1 --split-mode layer -t 16 -tb 20 --cache-type-k q8_0 --cache-type-v q8_0 -b 4096 -ub 4096 --flash-attn on --moe-cache auto --jinja --temp 1.0 --top-p 0.95 --reasoning on -lv 4 --reasoning-format deepseek --slot-save-path /home/data/ --alias DeepSkee-v4-Flash-0731-UD-Q4_K_XL -lv 4 --prompt-processing-gpu-moe 0 ' submitted by /u/Similar_Can_3143 [link] [comments]

Source: r/LocalLLaMA | 2026-08-22

Related