Qwen3.8-Flash-Next NVFP4 2xDGX Spark config: 50t/s decode, 2,900t/s prefill

After a couple of days of faffing around, here's my current config in case it helps anyone out. Some of this is also valuable for a single Spark. AI slop below for you to read and paste into your own agent :) Stack vLLM PR #53896 (release/qwen38next branch) — NOT main; main doesn't have the model, and the recipe image's vLLM commit isn't in the public repo sm_121 support is a 2-file patch (details below) — GB10 is arch 12.1 and the NVFP4 E2M1 conversion needs a software fallback PLE n-gram table served from internal NVMe via mmap:(48 GB, MADV_RANDOM is essential — it's a 30× read-amplification difference on hash-scattered row lookups), with a node-local CPU-worker process doing gathers over ZMQ + pinned buffers + CUDA-IPC outputs so the gather is graph-safe and TP2-safe TP2 across both Sparks: native venv (NOT docker — see dead ends), eager + MTP k=3 Numbers (TP2, warmed medians, count-20 structured / 3-paragraph prose) config structured prose prefill (11k tokens) eager + MTP k=3 49.7 34.8 ~2,875 Dead ends PIECEWISE cudagraphs cost ~28% decode under MTP+TP2. Same tree, same everything, graphs on vs off: 36 vs 50 tok/s. Graphs help single-node no-MTP decode (+80%, 9→16.5), but combined with MTP at TP2 they're a straight loss — MTP already amortizes the launch overhead the graphs were eliminating. We chased a "tree-level MTP regression" for a day before realizing the config was the regression. Docker silently degrades NCCL to TCP sockets. Default containers get no IB device → NET/Socket → TP2 prefill at ~40% of native. You need --privileged (or IB device passthrough) AND the right HCA name — see next point. Native venv runs just get RoCE. RoCE device names are not stable across reboots, and can differ between two identical machines (ours: rocep1s0f1 on one, rocep1s0f0 on the other, resolved dynamically). Hardcoded NCCL_IB_HCA will silently fail on one node. Enumerate with ibdev2netdev, match port-ACTIVE to your rail netdev, and verify with NCCL_DEBUG=INFO that you see NET/IB, not NET/Socket. flashinfer must be 0.6.18 on GB10 — 0.6.17 crashes the NVFP4 MoE fallback kernel. vLLM's MoE shuffleInputRowsKernel has an uninitialized-permutation OOB read that shows up as a fake CUTLASS status=7 GEMM failure — a two-line bounds guard fixes it (already in PR #53896). The 2-file sm_121 patch (vs PR #53896 base) CMakeLists.txt: add 12.1 to CUDA_SUPPORTED_ARCHS, SCALED_MM_ARCHS, CUTLASS_MOE_DATA_ARCHS, FP4_SM120_ARCHS csrc/libtorch_stable/quantization/fp4/nvfp4_utils.cuh: ~90-line software E2M1 conversion guarded by CUDA_ARCH == 1210 (GB10 has no cvt.rn.satfinite.e2m1x2.f32) That's the whole delta. Build offline with the usual dependency cache; CUDA arch 121. Startup command(s) Head node (rank 0): bash export VLLM_PLE_MMAP=1 VLLM_PLE_MMAP_WORKERS=64 VLLM_PLE_MMAP_PREWARM=1 export VLLM_PLE_MMAP_DIR=$HOME/ple-table # internal NVMe copy of the table export VLLM_PLE_CPU_OFFLOAD=1 # activates the IPC CPU-worker gather path export NCCL_SOCKET_IFNAME=<your-rail-if> # bootstrap only; NCCL finds RoCE itself export GLOO_SOCKET_IFNAME=<your-rail-if> python -m vllm.entrypoints.openai.api_server --model <path-to-qwen38-flashnext-nvfp4> --host 0.0.0.0 --port 8086 --load-format safetensors --tensor-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr <head-rail-ip> --master-port 29511 --distributed-executor-backend mp --max-model-len 262144 --max-num-seqs 13 --gpu-memory-utilization 0.85 --no-enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens 8192 --long-prefill-token-threshold 4096 --enforce-eager --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":3}' Worker node (rank 1): identical, but --node-rank 1 --headless, launched ~30s after the head. Note --enforce-eager — that's not a compromise, it's the fast path (see dead end #1). submitted by /u/-dysangel- [link] [comments]

Source: r/LocalLLaMA | 2026-08-30

Related