For MoE models the arithmetic splits in two: capacity follows total params, speed follows active

A few people asked for this after the bandwidth thread, so here it is on its own instead of buried in a comment. The dense rule was simple: every token reads every weight, so tokens/sec ≈ bandwidth ÷ model size, times about 0.7 for real efficiency. For a mixture of experts that breaks, and it breaks in a more interesting way than "it's faster." The two numbers stop being the same number In an MoE only the routed experts are read per token. So: capacity is set by total parameters — you still have to hold the whole thing speed is set by active parameters — you only stream a slice of it per token That's the part everyone repeats. It's right, and it's why a 120B-class MoE can feel quicker than a 30B dense model. But it's where the useful detail starts rather than ends. It's not just the active experts Attention, embeddings, and — on the architectures that have one — the always-on shared expert get read on every single token regardless of routing. So per-token traffic is the routed experts plus that fixed part. Say a 122B model with 10B active carries roughly 2-3B of always-on weights. You're streaming 12-13B per token, not 10B. That's 20-30% slower than the naive number, which is exactly the gap that makes people think their setup is misconfigured when it isn't. Check it on your own box with --verbose and compare the eval rate against bandwidth ÷ (active + always-on). Spilling hurts an MoE more, not less This is the one that surprises people, because "smaller working set" sounds like it should be more forgiving. In a dense model the part that doesn't fit is a fixed slice. You can compute the cost exactly: 20% of the model at DDR speed instead of VRAM speed cost about 40% of the throughput in the example from the last thread. Painful, but bounded and predictable. In an MoE there is no stable hot set. Routing changes per token and per layer, so the experts you paged out are the ones that get picked often enough to matter. You aren't paying the penalty on a fixed 20%, you're paying it on whatever the router happens to want, and the average collapses toward the slow number rather than landing between the two. So the bandwidth requirement drops but the capacity requirement doesn't budge at all. Fitting the whole thing stays mandatory. Batch size quietly gives the advantage back Concurrent requests don't activate the same experts. Two users hit different ones, four users more, and the union of what has to be read per batch creeps back toward the full model. Serve enough parallel requests and an MoE degrades toward its dense-equivalent cost. Single user, this never happens and you can ignore it. Serving a team, it's the difference between the number you benchmarked alone and the number people actually get. What it changes when you're buying The order of operations from the dense case still holds, but the gap between the steps widens: fit the whole model, with context headroom — non-negotiable, and set by total params then chase GB/s — but you need far less of it than the file size suggests Which is why large unified-memory machines make sense for MoE in a way they don't for dense. 128GB at ~270 GB/s is a bad deal for a 70B dense model and a genuinely good one for a 100B-plus MoE with a few billion active — capacity is what you can't buy your way around, and the bandwidth you need is low enough that the slower memory stops being the binding constraint. It also softens the usual "buy the fastest 24GB card" advice. For MoE that card is fast at something it can't hold. Where I'd like to be corrected The always-on fraction is the number I'm least sure of and it varies by architecture. If you're running one of the big MoEs, what does your measured eval rate imply for it? Take bandwidth ÷ measured t/s ÷ 0.7 and you get the bytes actually streamed per token — subtract your active params and what's left is the fixed part. I'd be curious how close that lands to the counts on the model card. submitted by /u/Firm-Luck2062 [link] [comments]

Source: r/ollama | 2026-08-01