BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batching
DGX agentarXiv:2411.16102v2 Announce Type: replace Abstract: Offline batch inference, which leverages the flexibility of request batching to achieve higher throughput and lower costs, is becoming more popular