Model Releases
enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think
Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel
Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering CPU is meaty, high RAM bw, no bottlenecks, messing with p2p should be minuscule, right? That's what I thought as well, now see results of two identical tests: Test prompt: llama-benchy --base-url http://localllm/v1 --model localllm --depth 0 4096 8192 16384 32768 --latency-mode generation 1) no p2p: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------|----------------:|----------------:|---------------:|-----------------:|-----------------:|-----------------:| | localllm | pp2048 | 1648.96 ± 8.49 | | 1241.77 ± 8.66 | 1131.60 ± 8.66 | 1241.77 ± 8.66 | | localllm | tg32 | 89.86 ± 7.05 | 92.76 ± 7.28 | | | | | localllm | pp2048 @ d4096 | 1654.51 ± 6.44 | | 3504.72 ± 11.71 | 3394.55 ± 11.71 | 3504.72 ± 11.71 | | localllm | tg32 @ d4096 | 102.98 ± 3.95 | 106.30 ± 4.08 | | | | | localllm | pp2048 @ d8192 | 1631.74 ± 10.72 | | 5821.87 ± 82.46 | 5711.71 ± 82.46 | 5821.87 ± 82.46 | | localllm | tg32 @ d8192 | 109.27 ± 4.42 | 112.79 ± 4.56 | | | | | localllm | pp2048 @ d16384 | 1601.33 ± 3.18 | | 10502.40 ± 46.13 | 10392.23 ± 46.13 | 10502.94 ± 46.27 | | localllm | tg32 @ d16384 | 98.61 ± 5.01 | 116.87 ± 5.94 | | | | | localllm | pp2048 @ d32768 | 1544.66 ± 1.10 | | 20598.44 ± 65.95 | 20488.27 ± 65.95 | 20598.44 ± 65.95 | | localllm | tg32 @ d32768 | 93.24 ± 15.89 | 122.15 ± 15.63 | | 2) p2p enabled: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------|----------------:|----------------:|---------------:|-----------------:|-----------------:|-----------------:| | localllm | pp2048 | 2305.20 ± 49.73 | | 913.50 ± 27.47 | 808.72 ± 27.47 | 913.50 ± 27.47 | | localllm | tg32 | 97.92 ± 8.26 | 101.07 ± 8.53 | | | | | localllm | pp2048 @ d4096 | 2268.40 ± 6.48 | | 2615.57 ± 28.00 | 2510.79 ± 28.00 | 2615.57 ± 28.00 | | localllm | tg32 @ d4096 | 103.60 ± 9.04 | 106.95 ± 9.33 | | | | | localllm | pp2048 @ d8192 | 2214.54 ± 8.39 | | 4307.55 ± 49.46 | 4202.77 ± 49.46 | 4312.27 ± 55.30 | | localllm | tg32 @ d8192 | 122.88 ± 5.89 | 126.84 ± 6.09 | | | | | localllm | pp2048 @ d16384 | 2164.10 ± 8.34 | | 7809.37 ± 73.29 | 7704.59 ± 73.29 | 7809.37 ± 73.29 | | localllm | tg32 @ d16384 | 111.51 ± 7.92 | 125.93 ± 3.14 | | | | | localllm | pp2048 @ d32768 | 2082.02 ± 3.86 | | 15224.08 ± 21.55 | 15119.30 ± 21.55 | 15224.08 ± 21.55 | | localllm | tg32 @ d32768 | 96.02 ± 7.55 | 136.62 ± 24.75 | | | | Welp, ~25% of PP for free. HF model used: Qwen/Qwen3.6-27B-FP8 KV quant - not used, so F16 Parallelism mode - tensor-parallelism How to enable P2P Your hardware should support ReBAR, enable it in bios. Install patched drivers (read readme carefully!) from https://github.com/aikitoria/open-gpu-kernel-modules Add following environment variables to VLLM startup: NCCL_P2P_DISABLE=0 VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS That's it! submitted by /u/BidonPomoev [link] [comments]
Related
- 2 x 5070ti Qwen 27B full config / stats
- [[deepseek-v4-flash-0731-full-1m-context-on-a-single-rtx5090-d|[Deepseek-V4-Flash-0731] Full 1M context on a single RTX5090 + DDR5 Desktop Setup with VLLM CPU/Ram Offloading, ~800 tps pp & 15+ tps decode [Agentic Coding]]]
- Dual 3090 setup: 400 pp t/s to 1600 pp t/s on Qwen 3.6 27B... with slightly lower tps.
Source: r/LocalLLaMA | 2026-08-08