[Benchmark] llama.cpp batch/ubatch impacts on PP and TG

My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To improve the speed, weights have to be loaded as little as possible, so I explored the impact of batch and ubatch parameters. PP ubatch/batch 128 256 512 1024 2048 4096 8192 128 1.70 256 1.78 2.31 512 1.70 2.33 3.32 1024 1.64 2.18 3.21 6.62 2048 1.65 2.16 3.59 6.37 10.33 4096 1.65 2.29 3.40 6.70 9.99 15.78 8192 ? ? ? ? ? ? 18.61 Input prompt is 4143 tokens. TG batch/ubatch 128 256 512 1024 2048 4096 8192 128 3.11 256 3.41 3.07 512 3.37 3.37 3.02 1024 3.50 3.10 2.95 2.61 2048 3.26 3.20 2.91 2.82 2.38 4096 3.04 3.51 3.19 2.59 2.32 1.81 8192 ? ? ? ? ? ? 1.15 Note: 256 generated tokens, after the input prompt. MTP=2, this add little randomness to the generation speed. Conclusion: - in this config, batch size != ubatch size is useless for PP - increasing ubatch size increases PP speed as expected - increasing ubatch size decreases TG speed. This is a surprise for me. Can someone explain why bigger ubatch values has impact on TG? I thought ubatch would only change the PP, so now I am not sure to understand ubatch parameter usage in the code. Is there a way to have PP speed from high ubatch values and TG speed from small ubatch values? submitted by /u/PhilippeEiffel [link] [comments]

Related

• GLM-5.2 local inference: ubatch size made a much bigger difference than I expected
• DeepSeek-V4-Flash-0731 (284B MoE) at 75 tok/s on 2× DGX Spark — full recipe, 11 gotchas, reboot-proof cluster, Codex CLI integration
• 3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed
• Extremely slow DSpark draft model performance (1-2 t/s) with DeepSeek-V4-Flash on llama-server compared to MTP?

Source: r/LocalLLaMA | 2026-08-24