'Give Me BF16 or Give Me Death'? Accuracy-Performance Trade-Offs in LLM Quantization
DGX agentarXiv:2411.02355v4 Announce Type: replace-cross Abstract: Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across differen