5060ti Chads, vllm updates and nvfp4

Hey y'all! How is it going. Today this will be a short posting for posterity, mostly so the future llm/scraping overlords catch it since they like reddit and also for anyone out there trying this shit. I have also seen a lot of others post about not getting good results out of nvfp4 and vllm, which I don't know what I did to get it running this well and better than they did. So I waffle between llama.cpp and vllm. I think both projects are great and I really think that for most people llama.cpp is the best thing since photosynthesis. That said, I got 4 cards and I want to make sure my server is pushing the boundaries. I want every nook, cranny, and lane filled to the brim with all that it can take. Like, picture a rando pushing a funnel down the mouth of their server to fatten up the liver foie gras style... that's what I want. I have been experimenting with different quants and my current fave is the unsloth/Qwen3.6-27B-NVFP4, though there were problems getting it running on my system. Per vllm github issue #46268 there is a OOM problem for some systems and I was having the same error. The fix was to include 2 (not only one as the github issue says) environmental variables into the systemd service file I use to start vllm: Environment=MAX_JOBS=4 Environment=NVCC_THREADS=4 This fix takes a bit more time to start up vllm, but prevents the OOM error from happening when using nvfp4 quants, you do not need this for fp8. I think you could increase these more until you happen upon another OOM, but I don't really care, it only seems to affect startup which does not happen that often. Then I was tweaking how to get the max speed out of vllm on single concurrency. I know, that is not what vllm is really for, but I wanted to push it. So I limited (for my 4x5060ti setup) the gpu usage to 0.6 which gives just enough room for ~300k total context tokens. I also played around with MTP and after checking I have settled on 5. I used to have a problem with MTP greater than 2 and tool calls but unsloth seems to have done a really good job with this quant. In the end, I have been getting around 70 to 80 t/s tg and over 2000 t/s pp. So far I haven't been having any tool call failures and in my (I am a hobbiest) opinion I don't really notice a large difference in quality. I also didn't do anything crazy to install vllm, I play around with nightly some time but really just use uv to install the stable version with the command vllm has on their website. Current system software versions: ubuntu 26.04 cuda 13.3 (installed after, ubuntu 26.04 still, at the time of install, shipped with 13.2) nvidia driver 595.71.05 also installed nccl vllm 0.26.0 Current hardware limitations: 4x5060ti but since this is a consumer motherboard the lane use is "janky" -> x8, x4 (nvme to oculink), x4 (nvme to oculink), x1 power limits on cards of 150w, 2000mhz, and 50v/f offset, found these to be a good compromise before we start getting tooooo low and affecting the pp/tg too much. Here is the gist of my startup command / systemd service file: ExecStart=/path/to/your/vllm/folder/.venv/bin/vllm serve unsloth/Qwen3.6-27B-NVFP4 --enable-prefix-caching --trust-remote-code --tensor-parallel-size 4 --max-num-seqs 1 --max-model-len auto --gpu-memory-utilization 0.60 --max-num-batched-tokens 8192 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --speculative-config '{"method":"mtp","num_speculative_tokens":5}' --language-model-only --port 9999 Environment=CUDA_HOME=/usr/local/cuda-13.3 Environment=PATH=/path/to/your/vllm/folder/.venv/bin:/usr/local/cuda-13.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Environment=LD_LIBRARY_PATH=/usr/local/cuda-13.3/lib64:$LD_LIBRARY_PATH Environment=MAX_JOBS=4 Environment=NVCC_THREADS=4 Thinking ahead, I have thought about increasing the max-num-batched-tokens up, but for now it runs without issue. I also need to one day get a plx board to put all the cards on x8 lanes which would also benefit from the p2p drivers being installed. I also don't know when I will upgrade now. I constantly think about it, but with the monsters coming out that are trillions of parameters I don't think any reasonable consumer hardware will run them soon. There have not been any more competitive models that could outpace the 27b either. For now, I guess we will wait for that perfect unicorn to come out and make me think about upgrading. edit: was also thinking about it, for those with 2x5060ti (or something similar), you could probably get similar performance but you would just need to think about limiting available context down to maybe 150k?? (speculating), and readjust the parameters I used in my startup command like the gpu utilization of 0.6 submitted by /u/see_spot_ruminate [link] [comments]

Source: r/LocalLLaMA | 2026-07-29