Qwen3.6-27B speculative decoding gets better on heavier quants

I finished the speed leg of my spec-decode benchmarking for Qwen3.6-27B, main algorithms across quants. Overall: the heavier the quant, the more spec-decode buys you (10 of 10 speculative configs rank Q8 > Q6 > Q4 by multiplier). Acceptance is quant-independent at matched depth, the base step slows with weight bytes, draft+verify overhead does not. Exception is nvfp4/SGLang which is the fastest quant/engine pair overall. DFlash is the fastest algo overall. Weaver beats it but is the outlier: fork-only, and the head is per-target, so that is one model/quant combo, not a general option. Recipe, patches, and versions for Weaver: https://gist.github.com/thavoc/a9f3a37c082e7a8bbcf2b8efebfada25 MTP is a solid second everywhere. EAGLE3 is dominated by both, useful only as a fallback when no DFlash drafter exists. ngram I would skip: ~1.03× regardless of quant, and net-negative under concurrency. Interesting anomaly: llama.cpp's MTP path on UD-Q4 is pathologically slow. Q4 MTP-3 is slower in absolute tok/s than Q6 MTP-3 at identical acceptance, which bandwidth cannot explain. Output is correct, mechanism unknown. Caveat: a single, narrow data point on one hardware config. Spec-Bench with short outputs, greedy, batch 1 is close to a best case for speculative decoding. Under concurrency the per-stream gains shrink and longer contexts will compress them further, so treat these as an upper bound for this workload shape, not a general speedup. No accuracy A/B yet either, so this says nothing about whether the heavier quant's quality is worth its remaining speed gap. submitted by /u/thavoc77 [link] [comments]

Source: r/LocalLLaMA | 2026-07-27

Related