Calculate --override-tensor for llama.cpp using QWEN models and Pi for 2 GPU

When using two GPU with llama.cpp with --split-mode layer you usually provide a --tensor-split value to fit the proportion of the allocation on each GPU, say you have 16GB + 12GB something like --tensor-split 0.6,0.4 may do. That is a rough division of the models, as vRAM counts and hundreds of MB translate into tens of thousands of ctx tokens you wanna fine tune with the --override-tensor in order to move the tensor in such a combination, think like lego bricks in a box, that allow the best usage of space. For example for this Qwen3.8-27B-UD-Q6_K_M.gguf : 110848 ctx with normal split 136704 ctx with 17 graphs splits 139776 ctx with 23 graphs swaps Result like: -ot '^blk43.w[w.]=ROCm0,^blk40.w[w.]=ROCm0' The tricky part is that to test this you have to restart the model, so after the initial evaluation the skill gives you a script with all the probes it needs to run, you stop the model, run the scripts and then when it's done you relaunch your original model (tip: save the KV cache with --slot-save-path ) and the model evaluates the results of the probes giving you the final result. As said I tested for both ROCm and Vulkan, should work for CUDA too yet I did not test it, the script are meant to run on Linux yet I guess that your model can adapt those for Windows if you ask. Link to see it: https://store.piffa.net/lm/dual-gpu-tuner/ archive to dwl in single file: https://store.piffa.net/lm/dual-gpu-ot-tuner.tgz submitted by /u/ea_man [link] [comments]

Related

• Dual 3090 setup: 400 pp t/s to 1600 pp t/s on Qwen 3.6 27B... with slightly lower tps.
• NCCL-Free Tensor Parallelism on Dual Blackwell PCIe llama.cpp b9095 released!
• (NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s
• PSA for anyone with multiple V620's or other gfx1030 cards having problems making llama.cpp tensor split work -- set '-ub 384' and -b to a multiple of that depending on number of GPUs

Source: r/LocalLLaMA | 2026-08-30