We quantized DeepSeek V4 0731 and benchmarked it against popular quants on 8× RTX 5090

We converted the model from the original safetensors and found two issues. The first one made our quantization fail several times, the second one does not fail at all, it just quietly ruins the base 1) You must use the --no-lazy option, otherwise token_embd.weight will take on the value NaN. 2) By default, the converter downconverts FP8 tensors to Q8_0, hard-coded in the file conversion/deepseek.py. This causes the model to deviate from the original weights by 0.219 on average KLD even before quantization begins. Our 118 GB quant has a value of 0.2065, so the "lossless" 162 GB baseline model generated by default is farther from the original than the model with 3-bit quantization. We corrected these tensors by replacing them with BF16, and the base model became bit-exact. Then we applied imatrix to 1.87 million tokens, divided into blocks of 8,192 tokens and built 13 quants with per-tensor overrides, setting the expert bits based on the measured activation energy for each layer While we were making these we wanted to know if we were going in the right direction, so we started comparing our numbers against other people's quants. That does not work. Everyone publishes numbers from their own machine and on this model the same file gives different numbers on different GPUs. llama.cpp has a fast path for MXFP4 weights that only switches on for consumer Blackwell, so a 5090 and an H100 run the same file differently. We measured 4.5381 PPL on a 5090 and 4.3406 on an H100. So we stopped reading published numbers and measured all 38 files on one machine ourselves Harness: • 8x RTX 5090 • wikitext-2 at ctx 5632, 51 chunks • reference is our own bit-exact BF16 base (top-1 with unsloth's lossless quant is 100%) The whole lineup of quants 154 GB+ is similar, because the model is QAT. The range where we can see the diff is 85 to 135 GB. For the main part of the quants our curve is lower than anyone else's, which means at the same file size our quant drifts less from the full model and picks the same next token more often But there are a few points where community quants are better, for example prometheusAIR and bullerwins All of this is matched by file size, not quant name, because the names don't line up between publishers. Our AD-IQ2_M is 2.79 bits per expert weight, which others call IQ3_XXS That was the other thing we found while benchmarking. There is no standard for naming quants on HF. We've already talked it over with people who publish a lot of them, so we're not the first to notice and we'll put up a separate thread on how quants should be named Based on our quantization the best quant for DeepSeek on 128 GB hardware is our AD-IQ2_M (104 GB) with 83.6% top-1 Collection on HF with the imatrix, the per-tensor layout and everything else https://huggingface.co/collections/AtomicChat/deepseek-v4-flash-0731-6a708cf546ed69ee944fd3b1 Our app is an open source https://atomic.chat (I'm founder). Feel free to ask any questions and share your feedback! submitted by /u/gladkos [link] [comments]

Source: r/LocalLLaMA | 2026-08-11

Related