Qwen3.8-27B on a 24GB RTX PRO 4000 Blackwell: 128K real context, 785 tok/s prefill, 67 tok/s MTP3 decode

I wanted to see whether NInfer could run Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB. Hardware/software: NVIDIA RTX PRO 4000 Blackwell, 24GB 145W power limit CUDA 13.3 NVIDIA driver 610.x Ubuntu LXC under Proxmox Qwen3.8-27B NInfer artifact ruwwww/ninfer-5060ti, branch rtx-5060ti I used the 5060 Ti fork rather than the 4090 fork because the PRO 4000 is Blackwell and supports sm_120a. The 5060 Ti fork also has changes that calculate cooperative scheduling from the GPU's actual SM count instead of assuming a 5090. What I had to do Install the missing build dependencies: sudo apt install -y build-essential ninja-build pkg-config libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libcurl4-openssl-dev I then hit this during CMake: nvcc fatal: Unsupported gpu architecture 'compute_120a' The machine had several CUDA toolchains installed. /usr/bin/nvcc was being selected and did not support the Blackwell architecture-specific target properly. The actual CUDA 13.3 compiler was here: /usr/local/cuda-13.3/bin/nvcc I verified it directly: /usr/local/cuda-13.3/bin/nvcc -arch=sm_120a /tmp/test120a.cu -o /tmp/test120a Compile and execution both returned 0, confirming that sm_120a worked on the RTX PRO 4000. Then I rebuilt NInfer explicitly against CUDA 13.3: export CUDA_HOME=/usr/local/cuda-13.3 export PATH=/usr/local/cuda-13.3/bin:PATH export LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:{LD_LIBRARY_PATH:-} cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.3/bin/nvcc -DCUDAToolkit_ROOT=/usr/local/cuda-13.3 cmake --build build --parallel Performance First test was normal autoregressive decoding, no MTP: Config Prefill Decode Overall 4K, BF16 KV, no MTP 417.7 tok/s 31.1 tok/s 28.0 tok/s GPU memory: Weights: 15.92 GiB Free after startup: 5.61 GiB Planned device total: 16.65 GiB Then I enabled Qwen3.8's MTP speculative decoding: --spec mtp --draft-tokens 3 --lm-head-draft At 8K BF16: Prefill: 413.1 tok/s Decode: 59.62 tok/s Overall: 58.27 tok/s MTP acceptance: 63.91% Accepted length: 2.92 tokens/round Fallback steps: 0 Free after startup: 4.56 GiB So MTP3 took decode from roughly 31 tok/s to 60 tok/s, about a 1.9x improvement on this workload. Then I switched the KV cache to INT8 group-64 and increased the reserved context. 32K INT8 KV + MTP3 Prefill: 413.95 tok/s Decode: 62.23 tok/s Overall: 60.33 tok/s KV payload: 1.10 GiB MTP acceptance: 68.37% Accepted length: 3.05 tokens/round Free after startup: 4.00 GiB 64K INT8 KV + MTP3 Prefill: 419.50 tok/s Decode: 62.28 tok/s Overall: 60.41 tok/s KV payload: 2.19 GiB Free after startup: 2.90 GiB 128K INT8 KV + MTP3 Prefill: 412.94 tok/s Decode: 62.17 tok/s Overall: 60.27 tok/s KV payload: 4.38 GiB Free after startup: 727.06 MiB Planned device total: 21.61 GiB Summary: Reserved context KV Decode Free VRAM after startup 4K BF16, no MTP 31.06 tok/s 5.61 GiB 8K BF16 + MTP3 59.62 tok/s 4.56 GiB 32K INT8 + MTP3 62.23 tok/s 4.00 GiB 64K INT8 + MTP3 62.28 tok/s 2.90 GiB 128K INT8 + MTP3 62.17 tok/s 727 MiB The MTP3 tests consistently had around 64–68% draft-token acceptance, roughly 3 accepted tokens per speculative round, and zero fallback steps. One important caveat: these were context-capacity tests, not actual 32K/64K/128K prompt benchmarks. The test prompt was only ~69 tokens. So this proves that the KV allocations fit and that normal short-context generation remains stable with those reservations; it does NOT mean I'm getting ~62 tok/s while attending to an actual 128K-token prompt. 128K INT8 appears to be about the practical maximum for this exact configuration on 24GB because there's only ~727MB left after startup. The interesting next step would be getting the newer E8 4-bit KV cache work onto the Blackwell branch. That could potentially make the full 262K context feasible on 24GB. But as it stands: Qwen3.8-27B, MTP3, ~62 tok/s decode, and 128K INT8 KV capacity on a 24GB/145W RTX PRO 4000 Blackwell. 128K NIAH benchmark: 130,048 prompt tokens, INT8 KV, MTP off, thinking off. Prefill took 164.7s at 789.6 tok/s; decode after the full context was 24.4 tok/s. KV payload was 4.12 GiB with 1.73 GiB VRAM remaining. The model successfully retrieved the planted values (ORCHID=493817; COLOR=COBALT), so this was a real long-context inference test, not just KV allocation. 128K benchmark: Qwen3.8-27B on an RTX PRO 4000 Blackwell 24GB, 130,048 actual prompt tokens, INT8 KV. Prefill: 785 tok/s (~166s). With MTP3, post-128K decode reached 67 tok/s, versus 24.4 tok/s without MTP (~2.74× speedup). MTP acceptance was 100% on the 17-token deterministic NIAH answer, and the model correctly retrieved ORCHID=493817; COLOR=COBALT. Total planned VRAM was 21.61 GiB with ~727 MiB left. submitted by /u/mmkaywhatevers [link] [comments]

Source: r/LocalLLaMA | 2026-08-28

Related