Qwen3.8-27B KLDs

I have been lurking on reddit for like 6 years, and this is the first time posting, so apologies for the mistakes. I'm not a native English speaker, so sorry for the bad English too. I used LLM to translate / refine my post, including images. This is my attempt to compare the Qwen3.8-27B quants. Qwen is a coding and STEM model so I used codeparrot/github-code-clean, EleutherAI/proof-pile-2, allenai/peS2o, equal parts of each. I used 24 sequences of 8k tokens and 12 sequences of 32k tokens. The code was filtered for minified stuff and permissive licenses. It was measured with 24 sequences of 8,192 tokens and 12 sequences of 32,768 tokens, since it generates many tokens and is aimed for long-horizon tasks. Every format used the same compute path for the evaluation. All models are dequantized to BF16, and run through the same kernel. Also, checkpoint size is kind of a bad x-axis, since some models don't have MTP, which has zero performance impact. Still, all of them needs to be loaded on VRAM, so at least it's something useful, and that's why there are two graphs. Findings: The gap between 8bit and 4bit is huge. The best 4 bit score is 0.00835 and the worst 8 bit is 0.00071. I mean everyone knows that but yeah it's still something. vLLM can actually use INT5-7 weights, but I'm sure most people don't even know about that. The KLD of 4bit quantization varies a lot. Six checkpoints are at 11.7-11.8 GiB of transformer weights, all group-size 128, quantizing 400-496 modules and they range from 0.01364 to 0.02976. I believe the calibration data and the rounding algorithm takes a big impact. You should not pick a 4-bit quantization by it's file size. group_size affects a lot of kld. Checkpoints with group size 32 did better than ones that used group size 128 but kept some modules in BF16. Selective BF16 really only helps if you add it on top of a fine group size but that makes the file so big you might as well run a 6 bit model. The quantization damage mostly happens at the start of the context. This surprised me (very) but it holds true for all 24 models. The KL is much, much higher for the first 500 tokens. https://preview.redd.it/01gkqgnx93lh1.png?width=2369&format=png&auto=webp&s=ef83076427b93c659befd026a241a45990f0dec8 Quantizing lm_head, embed_tokens and linear_attn doesn't affect the performance at least for this test. submitted by /u/sadnessdevil [link] [comments]

Source: r/LocalLLaMA | 2026-08-23

Related