Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
DGX agentDense models activate all of their parameters for each token, whereas a Mixture‑of‑Experts (MoE) model loads all experts into memory but activates only a subset—e.g., Nemotron 3.5 Lightning has 30 B t