DeepSeek V4-Flash (284B MoE) at 33 tok/s single / 68 tok/s aggregate on 2× RTX 3090 + a used quad-Xeon DDR4 server — full config

Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine. Why bother with a 2018 server The model is 156 GB. That number decides everything before speed matters: Platform Memory Bandwidth Price Runs DS4-Flash? Mac Studio M3 Ultra 96 GB max¹ 819 GB/s 3,999+ ❌ won't load DGX Spark 128 GB 273 GB/s 4,699² ⚠️ 4-bit re-quant only, ~10 GB headroom AMD Ryzen AI Halo 128 GB ~256 GB/s 3,999 ⚠️ same RTX PRO 6000 Blackwell 96 GB 1,792 GB/s ~9,000 ❌ won't load 6× RTX 3090 144 GB 936 GB/s ~6,600 cards alone ✅ (+ a chassis that takes 6 cards) Used R940 + 2× 3090 512–768 GB 141 GB/s × 4 nodes ~6K ✅ full checkpoint ¹ Apple pulled the 512 GB M3 Ultra option in March 2026 and the 256 GB in May — 96 GB is the current ceiling. ² Up from 3,999 at launch, explicitly attributed to DRAM costs. Unified-memory boxes give you bandwidth in a small pool. A 4-socket server gives you a huge pool at lower per-node bandwidth — but four independent memory controllers running in parallel. For sparse MoE, where only ~13B of 284B params activate per token, capacity wins. Inference platform Lvllmds4-x v2.3.8 — guqiong96's SM80+ DeepSeek V4 specialization. A vLLM fork (base: yhfgyyf/vllm-deepseek-v4-sm89) with the lk_moe v2.3.1 CPU-GPU hybrid MoE engine doing NUMA-aware expert compute in system RAM. Prebuilt cp312 wheel from the GitHub release, no compiling. Model DeepSeek V4-Flash-0731 · 284B total / 13B active MoE · official safetensors, 156 GB (48 shards) Quantization-aware trained — routed experts (~96% of params) ship natively in MXFP4. Nothing re-quantized. FP8 linears run weight-only, activations BF16, KV cache fp8_ds_mla. The sm_86 trick: no native FP8/FP4 compute on Ampere, so the fork routes everything through Marlin weight-only kernels (MXFP4 MoE backend + MarlinFP8 linears). That's how a Blackwell-era checkpoint runs on 2020 GPUs. DSpark speculative decoding (built into the checkpoint, 5 draft tokens) — where most of the single-stream speed comes from. Hardware (all used/eBay-class) Dell PowerEdge R940 · 4× Xeon Platinum 8268 (96C/192T, Cascade Lake, AVX512-VNNI, no AMX) 768 GB DDR4-2933 (24× 32 GB, 6 channels/socket, 4 NUMA nodes) 2× RTX 3090 24 GB (sm_86), both PCIe x16, TP=2 NVMe + SATA SSD for model storage Current eBay pricing (Aug 2026): 96-core R940 with 128 GB runs 2,000–2,800; 512 GB around 3,800; 768 GB around 7,600. Add ~2,200–2,600 for a pair of used 3090s. You don't need 768 GB to run it. One instance needs ~170 GB, and with --membind pinning that has to fit on a single NUMA node — so 512 GB (128 GB/node) is roughly the entry point at ~6K all-in. The extra RAM buys instances, not speed: going 22→24 DIMMs moved throughput ~5%, within noise. Resource footprint while serving VRAM: 6.6 GB weights + KV per card (21.6/24 GB used) — GPUs sit at ~25% util System RAM: ~170 GB per instance (experts live in DRAM, streamed by CPU via lk_moe AVX512-VNNI kernels) Power (iDRAC/Redfish + nvidia-smi measured): ~1,000 W chassis under decode, 435 W idle. GPUs draw only 136–145 W avg (189 W peak). I power-capped both 3090s 350 W → 250 W and throughput didn't move a single tok/s — the cap never engages. ~95% of the load delta is 96 Xeon cores streaming experts from DRAM. At 0.13/kWh that's ~94/month worst-case 24/7, far less at realistic duty cycle. Results 128-token completions, temp 0, 22K max context, max-num-seqs 4, spec depth 5. Concurrent Aggregate Per user 1 33 tok/s 33 4 53–68 tok/s 13–17 8 47–63 tok/s 6–8 (Ranges = cold first pass → warm steady state with prefix cache.) For scale: the same box running the same model on ik_llama.cpp hybrid does 12.2 tok/s single-stream. The spec-decode + Marlin path is a 2.6× single / ~3× aggregate jump on identical hardware. What didn't matter Three separate things I expected to help and didn't: +2 DIMMs (22→24, symmetric 192 GB/node): ~5%, within noise GPU power cap 350→250 W: zero effect More GPUs: wouldn't help — they're at 25% util and 6.6 GB of 24 All three point the same way: the bottleneck is CPU-side DRAM bandwidth. This workload wants DDR5 and AMX (Sapphire Rapids), not more Ampere. If you're planning a build around this, spend on memory channels, not cards. Gotchas that cost me hours TileLang JIT-compiles kernels at runtime with whatever nvcc it finds — system CUDA 12.0 fails with cryptic lambda syntax errors. Point CUDA_HOME at the pip-bundled toolkit inside the venv (site-packages/nvidia/cu13). No system CUDA install needed. The wheel's pip CUDA packages ship internally mismatched (nvcc 13.2 vs runtime headers 13.0) → CCCL "compiler and toolkit headers are incompatible". Fix: pip install nvidia-cuda-runtime==13.2.86 nvidia-cuda-nvrtc==13.2.86. Undocumented DSpark constraint, found the hard way: max_num_seqs × (spec_tokens + 1) must be ≤ 32 or engine warmup dies with a tensor-size mismatch. seqs=4 × spec=5 is the sweet spot — wider batches with shallower spec were slower everywhere. MiniMax and other non-DeepSeek MoE on this fork still hit the sm_86 vectorized_gather_kernel assert from generic LvLLM. The Ampere fixes are DS4-path only. Happy to share the full launch command / venv recipe in comments. submitted by /u/AbbreviationsSad5582 [link] [comments]

Source: r/LocalLLaMA | 2026-08-03