PSA for anyone with multiple V620's or other gfx1030 cards having problems making llama.cpp tensor split work -- set '-ub 384' and -b to a multiple of that depending on number of GPUs

Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or higher. I will be opening an issue report on the llama.cpp GitHub if there isn't already one related to this. I'm using ROCm but Vulkan gave me issues too. But yeah, set -ub 384 (I haven't tried between 384 and 512, but this is working) and -b to a multiple of it, multiplied by at least the number of GPUs you're splitting the model across. I have 3x V620 in an R740. I just did two quick tests with Qwen3.6 using two of the cards. 27B Q8_0 + Q8_0 KV with MTP = 40 to 50+ t/s gen, 500 to 800+ prefill. 35B-A3B Q8_0 + Q8_0 KV with MTP = 80 to 110+ t/s gen, 1400 to 1800+ prefill. I'm finally starting to feel like I didn't waste money buying older GPUs! lol It seems stable, I've been using 27B heavy in Claude Code like this for hours with zero issues whatsoever. I found that jumping to three cards did not help performance, and it even slightly slowed it down but that may be due to the PCIe architecture of the R740 server. Two of the PCIe risers are connected to the same CPU, the third is on the other CPU, and communication to/from the third has to travel through a slower interconnect... tensor split = heavy PCIe bandwidth usage if you don't have a direct interconnect like NVLink. I ordered a mining rig frame and am going to be setting up a 6x V620 system without an interconnect like that, will see how that changes things. Now, hopefully they get DSV4 tensor split mode support added soon... And if this memory bug gets fixed, we can crank the ub higher and get even faster prefill. submitted by /u/TheWolfOfWalmart [link] [comments]

Source: r/LocalLLaMA | 2026-08-08

Related