Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM

If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you! It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single request, using tcclaviger's MXFP4-FP8 quant and custom vLLM image docker.io/tcclaviger/vllm:DevQwenNextFlash optimized for R9700. Total context (shared across all parallel requests) in this setup is 700k tokens. Here is the full command: podman run --rm -it --init --network host -v /models:/models:ro -v ~/.vllm-cache:/cache -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_ROCM_USE_AITER=0 -e ROCR_VISIBLE_DEVICES=0,1,2,3 -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor -e TRITON_CACHE_DIR=/cache/triton --device /dev/kfd --device /dev/dri --group-add keep-groups --annotation run.oci.keep_original_groups=1 --security-opt label=disable --security-opt seccomp=unconfined --shm-size 8g docker.io/tcclaviger/vllm:DevQwenNextFlash /models/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8 --served-model-name Qwen3.8-Flash-Next --tensor-parallel-size 4 --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --max-num-seqs 16 --enable-prefix-caching --enable-chunked-prefill --kv-cache-dtype fp8 --max-num-batched-tokens 4096 --gpu-memory-utilization 0.96 --mm-processor-cache-gb 4.0 --override-generation-config '{"max_tokens": 65536, "temperature": 1.0, "top_p": 0.95, "top_k": 40, "presence_penalty": 1}' --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}' --compilation-config '{"cudagraph_capture_sizes": [5,10,15,20,25,30,35,40], "max_cudagraph_capture_size": 40}' --host 0.0.0.0 --port 8080 submitted by /u/sloptimizer [link] [comments]

Related

• Qwen3.8-Flash-Next NVFP4 2xDGX Spark config: 50t/s decode, 2,900t/s prefill
• Qwen3.8-Flash-Next at 170K context on a single 96 GB card. ~110 tok/s.

Source: r/LocalLLaMA | 2026-08-30