More info about speculative decoding with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md

Speculative decoding is a technique implemented in llama.cpp that speeds up inference by using a smaller, faster model to predict multiple tokens ahead, which a larger model then verifies in parallel, reducing latency while maintaining output quality. The documentation on the llama.cpp GitHub repository provides technical details on how to configure and use this optimization strategy. This approach represents an advanced inference optimization for running large language models more efficiently on resource-constrained systems.

Source: Georgi Gerganov (X) | 2026-07-08