The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving

arXiv:2610.15629v1 Announce Type: new Abstract: Choosing a mixed-precision configuration for large language model (LLM) serving trades speed against accuracy. Allocators that use average bit-width or a memory budget as a speed proxy can miss changes in the resource that limits execution. We argue that served speed can be limited by compute, weight loading, or key-value (KV) cache access, depending on the phase, batch size, and context length. A byte-based proxy does not capture all of these costs, so fewer bits do not always mean faster serving. To address this gap, we build a cost model that accounts for the binding resource and use it in two ways. First, as a predictor: given a configuration and operating point, it estimates serving latency using constants calibrated for the model, GPU, and serving stack. We evaluate its accuracy across three hardware classes. Second, as an allocator: we combine modeled time costs with measured per-tensor accuracy costs and use a dynamic program to select a configuration under a latency or accuracy budget. We compare its configurations against byte-aware and sensitivity-only allocation.

Related

• MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
• SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving
• CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters
• Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
• DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

Source: arXiv cs.LG | 2026-10-14