Banger compression paper from NVIDIA. (bookmark it) Bigger MoE models keep winning on quality, but serving them at interactive latency is st…

Banger compression paper from NVIDIA. (bookmark it) Bigger MoE models keep winning on quality, but serving them at interactive latency is still hard. NVIDIA compresses the hybrid MoE Nemotron-3-Super into Puzzle-75B-A9B and roughly doubles interactive server throughput while holding quality. Pay attention to the joint structural search. Heterogeneous MoE pruning, active-parameter budget, and Mamba pruning get optimized together rather than one at a time, wrapped in an iterative pipeline with distillation, RL, quantization, and a Multi-Token Prediction head. Why does it matter? On a single 8xB200 node it hits about 2x the parent's server throughput at matched user-throughput, and 1M-token concurrency on a single H100 climbs from 1 request to 8. Accuracy holds across reasoning, coding, long-context, and agentic benchmarks. Cheaper serving with agentic capability intact changes what you can afford to run with these models. Paper: https://arxiv.org/abs/2607.04371 Learn to build effective AI agents in our academy: https://academy.dair.ai/

Source: DAIR.AI (X) | 2026-07-07