Achieve state-of-the-art inference latencies with speculative decoding

Speculative decoding is a technique for reducing inference latency in large language models by using a smaller, faster model to generate candidate tokens that are then verified by a larger model in parallel, allowing multiple tokens to be accepted per forward pass. This approach achieves state-of-the-art inference speeds by combining the efficiency of smaller models with the quality of larger models, particularly useful for serving LLMs in production environments with strict latency requirements.

Source: Modal Blog | 2026-06-24