DeepSeek V4.1 Flash on 8× A40: ~40 tok/s Q2_K and ~32 tok/s Q4_K_M with TensorSharp

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend. Setup: 8× NVIDIA A40, layer split, F16 KV cache, 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts—not the full context window. Final optimized results — all speeds in tokens/sec: Metric |Q2_K |Q4_K_M Prefill |533–539 |451.8–492.1 Single-request decode |40.31–40.72 |31.0–32.5 Decode, 2 concurrent requests — total |— |39.3 Decode, 4 concurrent requests — total |— |48.9 Decode, 8 concurrent requests — total |— |48.5 — = not reported in the cited final Q2_K retest. Concurrent figures are aggregate throughput, not per-request speed. What made the difference: Q2_K: Keeping the ~60 GiB quantized Engram tables on GPUs removes storage-dependent lookups. Unifying the backend per GPU also cuts decode graph splits from roughly 570 to 8, reducing synchronization overhead. Q4_K_M: The larger Engram tables stay in host memory. Automatic warming, tighter VRAM budgeting and token-batched decode deliver ~1.9× prefill and ~2× four-request throughput versus the previous behavior. Only 1 of 40 layers keeps its routed experts on the CPU. More parallelism isn’t automatically faster: On this no-NVLink system, Q4_K_M layer split beats experimental routed-MoE tensor parallelism: 31–32.5 vs. 21.4–22 tok/s single-stream decode. These are project-reported throughput measurements, not cold-start timings or a head-to-head win over another engine. Numerical/model-quality parity remains unproven, and batching can change generated output. submitted by /u/fuzhongkai [link] [comments]

Source: r/LocalLLaMA | 2026-09-12

Related