[Deepseek-V4-Flash-0731] Full 1M context on a single RTX5090 + DDR5 Desktop Setup with VLLM CPU/Ram Offloading, ~800 tps pp & 15+ tps decode [Agentic Coding]

First of all, obviously I took some help from AI to type this post and this is the topic that enabled me to accomplish all that: https://old.reddit.com/r/LocalLLaMA/comments/1veow4b/deepseek_v4flash_284b_moe_at_33_toks_single_68/ This post of mine is based on the link above. My Hardware: RTX 5090 32GB Ryzen 9 9950X3D 256GB DDR5-5600 Single NUMA node Linux Mint NVIDIA driver 595.71.05 CUDA 13.2 Software guqiong96/Lvllmds4-x vLLM 2.3.9 lk_moe 2.3.2 PyTorch 2.11.0+cu130 native DeepSeek-V4-Flash-0731 safetensors checkpoint 48 safetensors shards ~155.4 GiB checkpoint size One fix I needed During startup, FlashInfer's CUDA IPC helper could accidentally find TileLang's: libcudart_stub.so instead of the real loaded CUDA runtime. That eventually caused: undefined symbol: cudaDeviceReset The problem was FlashInfer's find_loaded_library("libcudart") doing a substring search over /proc/self/maps. I patched: flashinfer/comm/cuda_ipc.py so it checks the actual filename instead: def find_loaded_library(lib_name): with open("/proc/self/maps") as f: for line in f: if "/" not in line: continue start = line.index("/") path = line[start:].strip() filename = path.split("/")[-1] if ( filename.startswith(lib_name + ".so") or filename.startswith(lib_name + "-") ): return path return None After that, FlashInfer correctly resolves the real libcudart instead of the TileLang stub. This is a local patch and obviously needs to be reapplied if the package gets replaced. Current launch configuration This is the configuration I ended up using: source ~/ds4x-venv/bin/activate MODEL="/home/blackbeard/models/DeepSeek-V4-Flash-0731" export CUDA_DEVICE_ORDER=PCI_BUS_ID export CUDA_VISIBLE_DEVICES=0 export LVLLM_MOE_NUMA_ENABLED=1 export LK_THREADS=12 export OMP_NUM_THREADS=12 export LK_THREAD_BINDING=CPU_CORE # Keep two complete routed MoE layers GPU-resident on the GPU. export LVLLM_GPU_RESIDENT_MOE_LAYERS=0,1 # CPU/hybrid prefill path for now. export LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0 export FLASHINFER_DISABLE_VERSION_CHECK=1 export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True vllm serve "$MODEL" --host 0.0.0.0 --port 8070 --tensor-parallel-size 1 --max-model-len 1048576 --gpu-memory-utilization 0.92 --trust-remote-code --served-model-name DeepSeek-V4-Flash-0731 --compilation_config.cudagraph_mode FULL_DECODE_ONLY --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 --dtype bfloat16 --max-num-seqs 2 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --kv-cache-dtype fp8_ds_mla --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "max"}' --speculative-config '{"method":"dspark","num_speculative_tokens":2,"draft_sample_method":"greedy"}' --disable-custom-all-reduce Full native 1M context fits on the 32GB GPU even with two complete routed MoE layers resident on the GPU. The rest of the experts remain in system RAM. DSpark behaves very differently during reasoning During long reasoning sections, draft acceptance can collapse. I observed extended periods around: Draft acceptance: ~30-50% Generation: ~11-13 tok/s There was one ~6 minute section averaging roughly: Draft acceptance: ~40% Generation: ~11.9 tok/s Then the model transitioned into a much more predictable generation phase and the numbers jumped to roughly: Draft acceptance: ~87-88% Generation: ~17.4-17.6 tok/s The relationship is extremely strong: throughput basically tracks DSpark acceptance. Some high-acceptance windows look like: Avg Draft acceptance rate: 89.8% Avg generation throughput: 17.9 tokens/s while low-acceptance reasoning windows look like: Avg Draft acceptance rate: 38% Avg generation throughput: ~12 tokens/s This suggests an obvious optimization. Dynamic DSpark depth For this workload I suspect the ideal behavior would be approximately: reasoning/thinking: 1 speculative token normal/final decoding: 2 speculative tokens The second draft token often isn't worth computing while the model is doing difficult reasoning, but becomes very valuable when it transitions into more predictable code/text generation. vLLM does not currently give me a simple runtime switch for this, so I may patch the speculative decoding path later and experiment with changing the draft depth based on whether the model is currently emitting reasoning or final output. That looks like one of the biggest remaining decode optimizations. ---non AI comment section begins--- Stay tuned, I am working on a if/else block to fix that stupid behavior slowing down during reasoning and squeeze even more tps out of this stack. ---non AI comment section ends--- submitted by /u/BlackBeardAI [link] [comments]

Source: r/LocalLLaMA | 2026-08-04