Hardware
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
Speculative decoding speeds LLM inference by letting a lightweight draft model propose multiple tokens that the larger target model verifies in parallel, reducing decoding iterations while preserving
Speculative decoding speeds LLM inference by letting a lightweight draft model propose multiple tokens that the larger target model verifies in parallel, reducing decoding iterations while preserving accuracy. The authors outline five guidelines for selecting optimal draft length and mechanism—such as aligning draft sizes with GEMM compute‑bound regions or setting them to 128/G – 1 when attention dominates—and compare several draft strategies (external models, EAGLE‑3, MTP, DFlash, DSpark, suffix or n‑gram methods) in terms of training cost, memory usage, and speculation overhead. NVIDIA’s SPEED‑Bench benchmark and the Model‑Optimizer repository provide realistic workloads, acceptance‑length metrics, and ready-to-run fine‑tuning/quantization examples for models like Nemotron 3.5 Lightning.
Related
- Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
- BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
- AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
- PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
- More info about speculative decoding with llama.cpp: https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md
Source: NVIDIA Developer | 2026-09-02