Model Releases
DeepSeek-V4-Flash-0731 (284B MoE) at 75 tok/s on 2× DGX Spark — full recipe, 11 gotchas, reboot-proof cluster, Codex CLI integration
Spent two nights getting deepseek-ai/DeepSeek-V4-Flash-0731 (284B MoE, 13B active, native FP4/FP8, 1M context) running production-grade on two DGX Sparks connected by one QSFP DAC cable. Everything —
Spent two nights getting deepseek-ai/DeepSeek-V4-Flash-0731 (284B MoE, 13B active, native FP4/FP8, 1M context) running production-grade on two DGX Sparks connected by one QSFP DAC cable. Everything — scripts, tuning data, raw benchmarks — is in the repo: https://github.com/raullenchai/twinspark Headline numbers (TP=2 over RoCE, vLLM 0.25 + DSpark speculative decoding + NVFP4 MLA KV cache): 74.8 tok/s single-stream on real coding prompts (spec decode = 2.74× over the 27.3 tok/s bandwidth floor) ~1.7–1.9k tok/s prefill, 42 s TTFT on a 64k prompt, decode holds 50 ms/tok at long context KV capacity 2.74M tokens → 1M context actually usable Dual power-cycle → back to serving in ~9 min, zero manual steps (self-healing containers, actually tested by rebooting both nodes) Things that surprised us (full gotcha table in the README): The single QSFP cage is wired to TWO ConnectX-7 controllers. One 400G cable lights up 2× 200G links — and each controller sits on PCIe Gen5 x4, so per-rail RDMA caps at ~109 Gb/s. Give NCCL both rails (NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0) → 21.7 GB/s bus bandwidth. Ray's memory monitor kills your workers on unified memory. vLLM's "GPU" allocation counts as host RAM on GB10; Ray OOM-kills the worker while 117 GiB sits free. RAY_memory_monitor_refresh_ms=0. Garbled chat output with random Chinese = missing DSpark env vars, not a broken model. VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1 + VLLM_USE_BREAKABLE_CUDAGRAPH=0. (/v1/completions works fine the whole time, which makes it extra confusing.) Speculative decoding inverts under concurrency. spec7 wins single-stream (74.8), spec3 wins at 4 concurrent streams (105.7 agg). And on real agent traffic (tool outputs everywhere) draft acceptance drops from 75% → 40%. We ship both launch profiles. Random-token benchmarks sandbag spec decode by ~2×. vllm bench serve --dataset-name random says 33.8 tok/s; real prompts do 74.8. If two posts disagree about Spark numbers, this is probably why. Codex CLI 0.147 dropped wire_api="chat" — you need the Responses API (vLLM 0.25 ships /v1/responses) plus the undocumented model_catalog_json to register a custom model. Config examples in the repo. Also ran an honest agentic eval: Codex + this model landed a real multi-file concurrency feature (lazy-load model pool) in a 1,831-file / 17k-test codebase — one CI-report iteration needed. Details in the README. Stack: DGX OS 7.2.3, CUDA 13.0, ghcr.io/anemll/dspark-vllm-gx10:0.1.1 (vLLM 0.25.2 + sm_121 kernels) + Ray 2.57. Versions pinned in the README — this is a snapshot, expect drift. Happy to answer questions. Raw benchmark JSONs are in bench/results/ if you want to poke at the data. submitted by /u/Striking-Swim6702 [link] [comments]
Related
- Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM?
- Running DeepSeek-V4-Flash-0731 (155 GB MoE) on a DGX Spark with vLLM-Moet 2-bit quantization - AI's narrative
- DeepSeek V4 Flash 0731 appreciation post
Source: r/LocalLLaMA | 2026-08-11