Model Releases
Confirmed bolting Q8 NGram into IQ4 Qwen no speed degradation
This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version. Someone running a 5090 r
This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version. Someone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16. Since I'm already running IQ4_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead. I'm still testing whether it improves the actual model output, but in terms of inference speed, the difference appears to be very small. Before bolting on the Q8 N-gram: n_gen = 2588, tg = 8.82 t/s, tg_3s = 8.81 t/s n_gen = 2616, tg = 8.82 t/s, tg_3s = 9.19 t/s n_gen = 2645, tg = 8.83 t/s, tg_3s = 9.37 t/s n_gen = 2675, tg = 8.84 t/s, tg_3s = 9.78 t/s n_gen = 2704, tg = 8.84 t/s, tg_3s = 9.50 t/s n_gen = 2732, tg = 8.85 t/s, tg_3s = 9.06 t/s n_gen = 2761, tg = 8.85 t/s, tg_3s = 9.32 t/s n_gen = 2790, tg = 8.86 t/s, tg_3s = 9.50 t/s n_gen = 2819, tg = 8.86 t/s, tg_3s = 9.39 t/s n_gen = 2847, tg = 8.87 t/s, tg_3s = 9.22 t/s n_gen = 2875, tg = 8.87 t/s, tg_3s = 9.15 t/s With the Q8 N-gram bolted onto IQ4_XS: n_gen = 263, tg = 10.76 t/s, tg_3s = 11.04 t/s n_gen = 294, tg = 10.70 t/s, tg_3s = 10.18 t/s n_gen = 329, tg = 10.76 t/s, tg_3s = 11.32 t/s n_gen = 361, tg = 10.73 t/s, tg_3s = 10.46 t/s n_gen = 395, tg = 10.76 t/s, tg_3s = 11.11 t/s n_gen = 430, tg = 10.81 t/s, tg_3s = 11.37 t/s n_gen = 460, tg = 10.75 t/s, tg_3s = 9.91 t/s n_gen = 494, tg = 10.77 t/s, tg_3s = 11.16 t/s n_gen = 531, tg = 10.86 t/s, tg_3s = 12.15 t/s So far, there doesn't seem to be any meaningful speed penalty from replacing the lower-precision N-gram layer with Q8. The actual output quality is still being tested. It looks like Q8 Ngram is faster, but at the end both are steady state at 10.1~ tok/s -ish. And without MTP since i am using earlier merge from unsloth, RAM 96GB DDR4 (32x3 Three channel active 2400MHZ) Xeon e5 2690v4 RTX 3090 capped at 250 W Edit: the state_dict grew from 90~ GB to 115GB Edit 2: Fck me i forgot to put the code, for stitching it, wait 8 hours from now i'll upload it since it is already very very late at night submitted by /u/Altruistic_Heat_9531 [link] [comments]
Related
- Are models with N-Gram tables going to completely change the AI race?
- N-gram vs Experts explained
- My Qwen3.8-Flash-Next recipe for single GB10/DGX Spark, uses Intel AutoRound int4 quant and vLLM, fp8 ngram table offloaded to local SSD or external RDMA server. At mtp=3 c=1, code is ~47.5t/s, json is ~60t/s. Prefix cache is ON.
- Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)
Source: r/LocalLLaMA | 2026-09-02