Don't ignore llama.cpp RPC with old hardware. Results of a 5070 Ti and 1080 Ti over gigabit ethernet: it's actually functional.

Results up front: I had to prioritize prefill or token generation - there was no happy medium. Using UD-Q4_K_XL, q8 kv cache, and 96k max context: focus on generation (MTP = 2): 350 pp and 36 tg @ 12k context focus on prefill (disabled MTP): 560 pp and 19 tg @ 12k context Focus on quality: using UD-Q5_K_XL, full kv cache, 96k max context, and no MTP: 380 pp and 15 tg @ 12k context Goals: I really wanted to be able to try the new Qwen 3.8 27B model, but I didn't want to have to compromise the kv cache or the model. That meant, q4 kv was off the table, as was any model quant under 4-bit. Since my primary GPU only had 16GB VRAM and offloading to CPU was just horrendous, I decided to see what was possible using the GPU in my NAS, but without messing with hardware. Don't need real-time chat. Forum-style interactions and asynchronous work fits my expectations and workstyle. Lessons learned: enabling MTP absolutely tanks prefill speeds. Like, a 30% hit immediately. From what I've read, that was supposed to be something that a future PR would fix but the last ticket I looked at about it was 3 months old. It's probably not an easy fix. obvious but worth mentioning: MTP uses VRAM. So, if you need more context and don't mind waiting, then disable MTP. enabling kv quant for the draft MTP model will lower your available context - the complete opposite of expectations. No, seriously, every single time I tested it that was the result. There's even a github issue about it and it was explained that that's expected behaviour. Be aware! (https://github.com/ggml-org/llama.cpp/discussions/24102) MTP applies at the end/tail of each generation loop. That means, you actually want your strongest GPU to be the last card in the RPC chain, not the first. This also means it's going to impact your prompt processing as conventional wisdom had you putting the strong GPU up front to help with prefill speeds, but now you gotta move it. when I ordered my cards CUDA0,RPC0 then the max text generation speed I could get was around 22-25. With RPC0,CUDA0 I was able to get over 36 tokens per second. A massive increase by having the 5070 Ti handle the MTP step instead of the 1080 Ti. it was impossible to balance prefill and text generation. The jumps in either direction were huge and mutually exclusive. Best option was to optimize each and let the situation/usecase dictate what should be used. Think of the functionality graph as U-shaped. enable jumbo frames on your network. You'll have to do this along the whole chain to avoid fragmentation (NIC, switch, host/hypervisor). batch sizes make a big difference (old news, I know). Focusing solely on prefill, best speeds were with 512/64. But, when GPU arrangement was switched and MTP was enabled, the best batch sizes were 1536/256. when I was focusing on speed and not output quality, I was using a bit over 14GB of VRAM on the 5070 Ti and only 7.2GB on the 1080 Ti. So, an 8GB card with similar bandwidth would work just as well. Future steps: still lots of room for tweaks. I was running KDE while doing all of this, so if llama.cpp was running headless then I'd be able to squeeze out at least another 1-1.5 GBs of VRAM on the 5070 Ti. each leg of the sweep was only run 4 or 5 times and only at 12k context. That was enough for me to get a feel of what was successful without spending the entire evening waiting. Should be enough to help others find success with their own hardware combinations. many ways to increase context size, if that's desired. For me, speeds drop too much by that point that it's not even worth letting it run overnight. Better to find a more efficient solution. For now, I'll just have the model keep breaking down steps into managable chunks. This also aligns with how I'd plan/write things if I was doing it all myself. So, it makes things easier for me to review/proof. Things to note: llama.cpp b10362 n-gram speculation was always enabled (match 16, min 32, max 64) but not optimized fit was not used as I found better success with manual tuning of layer splitting generation (RPC,CUDA was 10,19) and prefill (CUDA,RPC was 25,10) 5070 Ti is on CUDA 13.3 with driver 610.57.04 running under Fedora 44 with KDE 1080 Ti in on CUDA 12.9 with driver 580.126.18 running in headless Fedora 44 LXC on Proxmox 9.2 with PCI passthrough Pascal card, so no Tensor cores and crappy F16 support gigabit ethernet with a single managed switch between PCs I'll post the llama-swap config entries in a comment below: https://www.reddit.com/r/LocalLLaMA/comments/1vrkxdt/dont_ignore_llamacpp_rpc_with_old_hardware/p4e1fd4/ submitted by /u/BigPoppaK78 [link] [comments]

Source: r/LocalLLaMA | 2026-08-18

Related