Model Releases
ggml-zendnn : add Q8_0 quantization support by z-sachin · Pull Request #23414 · ggml-org/llama.cpp
Benchmark Results Benchmark configuration: threads = 96 type_k = bf16 type_v = bf16 Llama-3.1-8B-Instruct Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 472.28 730.87 54.75% 512 450.86 83
Benchmark Results Benchmark configuration: threads = 96 type_k = bf16 type_v = bf16 Llama-3.1-8B-Instruct Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 472.28 730.87 54.75% 512 450.86 832.48 84.64% 768 446.81 864.52 93.49% 1024 439.58 800.15 82.03% 2048 405.07 778.34 92.15% tg128 33.08 33.14 0.18% Mixtral-8x7B Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 156.09 297.67 90.70% 512 156.63 389.44 148.64% 768 156.76 417.38 166.25% 1024 154.70 438.73 183.60% 2048 150.11 470.41 213.38% tg128 20.95 20.92 -0.14% gemma4 31B Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 116.05 195.02 68.05% 512 112.53 229.12 103.61% 768 111.96 239.02 113.49% 1024 110.93 238.03 114.58% 2048 106.37 222.32 109.01% tg128 8.50 8.47 -0.35% gemma-4-26B-A4B-it Q8_0 Prompt Size GGML_CPU_Q8_0 t/s ZenDNN_Q8_0 t/s Gain 256 570.87 597.84 4.72% 512 581.80 666.18 14.50% 768 588.67 683.91 16.18% 1024 574.79 684.13 19.02% 2048 562.26 642.08 14.20% tg128 33.96 33.83 -0.38% Observations Significant prompt-processing gains are observed for larger prompt sizes Decoding (tg128) performance remains comparable to ggml-cpu This is only for AMD EPYC™ CPUs. For more info https://www.amd.com/en/developer/zendnn.html https://github.com/amd/ZenDNN submitted by /u/pmttyji [link] [comments]
Related
Source: r/LocalLLaMA | 2026-07-15