Memory bandwidth, not VRAM size, sets your tokens/sec — here's the arithmetic

Every week someone asks which card to buy and the thread turns into people naming GPUs they happen to own. There's an actual calculation behind it, it takes two numbers off the spec sheet, and it predicts your tokens/sec surprisingly well. Posting it because I keep writing it out in comments. Bandwidth, not bus width Token generation is bandwidth bound. For every single token, the model streams its weights out of memory. So the number that matters is GB/s, and it's the product of two specs, not either one alone: GB/s ≈ (bus width in bits / 8) × per-pin speed in Gbps That's why "128-bit" on its own tells you nothing: GPU bus memory bandwidth RTX 4060 Ti 16GB 128-bit GDDR6 18 Gbps ~288 GB/s RX 9060 XT 16GB 128-bit GDDR6 20 Gbps ~320 GB/s RTX 3060 12GB 192-bit GDDR6 15 Gbps ~360 GB/s RTX 5060 Ti 16GB 128-bit GDDR7 28 Gbps ~448 GB/s RTX 4070 12GB 192-bit GDDR6X 21 Gbps ~504 GB/s RX 6800 16GB 256-bit GDDR6 16 Gbps ~512 GB/s RTX 3090 24GB 384-bit GDDR6X 19.5 Gbps ~936 GB/s RTX 4090 24GB 384-bit GDDR6X 21 Gbps ~1008 GB/s RTX 5090 32GB 512-bit GDDR7 28 Gbps ~1792 GB/s DDR5-6000 dual channel 128-bit system RAM ~96 GB/s A narrow bus with fast memory beats a wide bus with slow memory: the 128-bit 5060 Ti has more bandwidth than the 192-bit 3060. And a five year old 3090 still doubles it, which is why that card refuses to die. Turning that into tokens/sec For a dense model, one token means reading every weight once. So: tokens/sec ≈ bandwidth / size of the model in memory Real efficiency lands around 60-80% of theoretical, so multiply by ~0.7 for a usable estimate. An 8B at Q4_K_M is roughly 4.9GB: on 320 GB/s: 320 / 4.9 = 65 theoretical → ~45 t/s realistic on 448 GB/s: 448 / 4.9 = 91 theoretical → ~64 t/s realistic on 936 GB/s: 936 / 4.9 = 191 theoretical → ~134 t/s realistic Check it against your own setup, it usually lands close. For MoE models, only the active experts get read per token, so use the active parameter count rather than the total. That's the whole reason a big MoE can feel faster than a much smaller dense model. The part that outranks all of it Bandwidth only applies to the weights that actually fit. The moment part of the model spills into system RAM, those layers run at DDR and PCIe speed, and you can see from the table that's an order of magnitude down. Say you have a 10GB model and 8GB fits in VRAM at 448 GB/s, with 2GB in system RAM at ~96 GB/s: 8GB / 448 = 17.9ms 2GB / 96 = 20.8ms total ≈ 38.7ms per token → ~26 t/s versus 10GB / 448 = 22.3ms → ~45 t/s if it all fits. 20% of the model spilling costs you about 40% of your speed, and the last 20% is doing more work than the first 80%. This is why "just add more system RAM" doesn't rescue a card that's too small, and why capacity comes first: pick the quant that fits with headroom for context, then chase GB/s. Practical order of operations Decide the model class you want to run, and get its size at the quant you'd actually use. Pick a card where that fits in VRAM with room for context on top. Among the cards that clear step 2, take the highest GB/s you can afford. Only then argue about CUDA versus ROCm versus Vulkan. It's a real difference in setup friction, but it isn't what sets your ceiling. Happy to be corrected on the efficiency factor, ~0.7 is a rule of thumb from my own boxes and I'd be interested in what other people measure. submitted by /u/Firm-Luck2062 [link] [comments]

Source: r/ollama | 2026-07-30