When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

arXiv:2610.24993v2 Announce Type: replace Abstract: The throughput gains of speculative decoding depend on the trade-off between acceptance length and execution latency. We ask when additional acceptance is worth the cost of obtaining it. We establish a comparison protocol spanning training, inference, and evaluation, remeasuring released checkpoints and retraining six drafter recipes on shared target-generated data across ten benchmarks and two target models. Alignment changes the picture: gaps that appear large in published comparisons shrink substantially under shared training and inference conditions. This narrowing motivates two deeper questions: how is acceptance improved at different costs, and when do these gains improve throughput? We find that acceptance can be improved through nearly free objective and context changes or more expensive token-to-token conditioning. DCore illustrates the former route as a strong baseline with little additional measured drafting overhead. The payoff is conditional: task uncertainty and serving conditions affect acceptance returns, execution costs, and the preferred drafter and budget. These findings inform future drafter design and deployment choices.

Related

• Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inference
• ConFu: Contemplate the Future for Better Speculative Sampling
• Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
• TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairs
• Acceptance Dynamics Across Cognitive Domains in Speculative Decoding

Source: arXiv cs.CL | 2026-10-21