MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp

TensorSharp's MoE CPU-offload feature has been merged into main. Here is the parameters description of this feature: Mixture-of-Experts CPU offload: --n-cpu-moe | -ncmoe Keep the routed MoE expert weights of the first N layers in system RAM and multiply them on the CPU; attention, norms, the router and the shared expert stay on the accelerator. This is what makes a 35B-A3B MoE fit beside a long-context KV cache on a 12-16 GB card. Pass 'all' for every layer. Default: 0 (everything on the accelerator; TS_N_CPU_MOE env var overrides). Example: --n-cpu-moe 32 --cpu-moe | -cmoe Shorthand for --n-cpu-moe all: every routed expert stays in system RAM. Default: off (TS_CPU_MOE env var overrides). Example: --cpu-moe To measure its performance, I ran benchmark to compare TensorSharp with llama.cpp, and here is the result. The completed benchmark report has been checked-in: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/moe_cpu_offload_benchmark.md Host and software Component Detail GPU 2 x NVIDIA RTX PRO 6000 Blackwell Server Edition, 97,887 MiB each, driver 580.126.20, PCIe 5.0 x16 CPU 2 x Intel Xeon 6952P (384 threads, 6 NUMA nodes), cgroup quota 81.6 CPUs RAM 1,511 GiB Storage Models on a MooseFS network mount (page-cache warm for every measured run) OS Ubuntu 24.04.3 LTS, CUDA 12.8 TensorSharp branch feature/support_moe_offload_to_cpu, .NET 10.0.110, backend ggml_cuda llama.cpp llama-bench build 4308a4f, CUDA backend, default -t 192 Results by model Ratios are TensorSharp / llama.cpp: >1.0x means TensorSharp is faster, and for VRAM >1.0x means TensorSharp is heavier. Gemma 4 26B-A4B it (UD-IQ4_XS, 30 MoE layers) --n-cpu-moe TS VRAM (MiB) TS pp4096 TS pp8192 TS tg128 llama VRAM (MiB) llama pp4096 llama pp8192 llama tg128 0 (baseline) 16,822 11,173 11,274 161.4 14,602 10,843 10,628 206.7 8 15,724 7,063 6,500 80.2 11,874 1,459 1,459 32.7 16 14,128 4,183 4,888 54.5 9,122 833 854 21.9 24 12,346 3,500 3,958 49.1 6,368 667 689 16.7 30 (--cpu-moe) 11,038 3,035 3,072 39.7 4,134 543 495 12.9 --n-cpu-moe VRAM pp4096 pp8192 tg128 0 1.15x 1.03x 1.06x 0.78x 8 1.32x 4.84x 4.46x 2.45x 16 1.55x 5.02x 5.72x 2.49x 24 1.94x 5.25x 5.74x 2.93x 30 2.67x 5.59x 6.21x 3.07x Qwen 3.5 35B-A3B (UD-IQ4_XS, 48 MoE layers) --n-cpu-moe TS VRAM (MiB) TS pp4096 TS pp8192 TS tg128 llama VRAM (MiB) llama pp4096 llama pp8192 llama tg128 0 (baseline) 19,862 9,538 9,405 160.0 17,522 8,149 8,073 228.4 12 18,148 6,755 6,648 75.4 13,282 988 954 27.5 24 15,414 4,412 5,259 52.3 9,010 498 484 15.8 36 12,684 3,772 4,223 50.7 4,738 523 517 11.3 48 (--cpu-moe) 11,606 3,917 3,709 38.6 3,314 477 457 10.2 --n-cpu-moe VRAM pp4096 pp8192 tg128 0 1.13x 1.17x 1.16x 0.70x 12 1.37x 6.84x 6.97x 2.74x 24 1.71x 8.85x 10.86x 3.31x 36 2.68x 7.21x 8.17x 4.50x 48 3.50x 8.21x 8.11x 3.77x GPT-OSS 20B (Q8_0 / MXFP4, 24 MoE layers) --n-cpu-moe TS VRAM (MiB) TS pp4096 TS pp8192 TS tg128 llama VRAM (MiB) llama pp4096 llama pp8192 llama tg128 0 (baseline) 13,186 13,964 12,925 212.8 12,204 17,856 17,642 344.2 6 11,560 8,975 7,617 85.8 9,812 1,747 1,666 32.2 12 9,378 6,470 6,394 51.7 7,386 1,176 1,188 18.3 18 7,192 4,315 4,393 30.7 4,962 807 751 12.1 24 (--cpu-moe) 4,762 4,277 3,798 27.7 2,536 568 548 9.4 --n-cpu-moe VRAM pp4096 pp8192 tg128 0 1.08x 0.78x 0.73x 0.62x 6 1.18x 5.14x 4.57x 2.67x 12 1.27x 5.50x 5.38x 2.83x 18 1.45x 5.35x 5.85x 2.54x 24 1.88x 7.53x 6.93x 2.95x DeepSeek V4 Flash (UD-Q8_K_XL, 5 shards / 150.7 GiB, 43 layers, both GPUs) --n-cpu-moe TS VRAM (MiB) TS pp4096 TS pp8192 TS tg128 llama VRAM (MiB) llama pp4096 llama pp8192 llama tg128 0 (baseline, both GPUs) 169,132 3,448 4,387 51.1 155,608 2,398 2,232 49.6 12 131,818 392 428 10.3 117,150 126 124 13.7 24 79,742 218 236 5.3 78,954 64 63 7.2 --n-cpu-moe VRAM pp4096 pp8192 tg128 0 1.09x 1.44x 1.97x 1.03x 12 1.13x 3.11x 3.46x 0.75x 24 1.01x 3.42x 3.72x 0.74x TensorSharp is a native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support. Github repo: https://github.com/zhongkaifu/TensorSharp Thank you for checking out it and starring the project! Any feedback is really appreicated. submitted by /u/fuzhongkai [link] [comments]

Related

• Minimum VRAM GPU to run DeepSeek-V4-Flash-0731 Q4_K_XL at around 30 t/s ?
• I tried running a 1.56TB MoE model on a 6GB RTX 4050 Laptop, Here’s the result
• Benchmarks: TensorSharp vs. llama.cpp

Source: r/LocalLLaMA | 2026-08-05