ROCm 10 + llama.cpp + Qwen3.8 27B Q8 K XL with MTP on 2xR9700

TL;DR - yeah it works, no wild skills or patching required. 37-50 tg/s when generating text, spiking over 60tg/s when writing code and mtp hits hard. Your results can be better if you have a better motherboard than my aging x370. ---- Grabbed the docker image from here: https://hub.docker.com/r/rocm/dev-ubuntu-24.04/tags docker pull rocm/dev-ubuntu-24.04:10.0.0-full Started up the docker image, installed git and cmake on the docker image since they appeared missing. Pulled llama.cpp latest from github. Built llama as per https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip Had to export this to make llama find the linked library export LD_LIBRARY_PATH=/opt/rocm/core-10.0/lib/ Ran llama server with these flags (omitted some that don't matter like host and port, and yeah, i have some legacy flags around here) ./llama-server --no-mmap -kvu --alias qwen -hf unsloth/Qwen3.8-27B-GGUF:Q8_K_XL -sm tensor -c 255000 -cram 0 -ctk f16 -ctv f16 -fa 1 --jinja -t 7 --metrics --temp 0.8 --top-p 0.95 --top-k 20 --min-p 0 --presence_penalty 0.0 --repeat-penalty 1.0 --presence-penalty 0.0 --ctx-checkpoints 4 --checkpoint-min-step 1024 --chat-template-kwargs '{"preserve_thinking": true}' --spec-type draft-mtp --spec-draft-n-max 3 -b 1024 -ub 1024 The amd-smi shows 27GB used on each card, so i could drive context even a bit higher. Told llama.cpp this prompt: Write me a binary tree in javascript with insert, update and delete functionality Produced a solid result, and saw about a 5-10% of performance increase over the stock llama.cpp rocm docker image Llama server output: 7.32.958.983 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, f_sim_best = 1.000 (> 0.100 thold), f_keep = 0.124 7.32.959.115 I slot launch_slot_: id 3 | task 838 | processing task, is_child = 0 7.36.395.172 I slot print_timing: id 3 | task 838 | n_gen = 129, tg = 42.43 t/s, tg_3s = 42.76 t/s 7.39.415.576 I slot print_timing: id 3 | task 838 | n_gen = 243, tg = 40.09 t/s, tg_3s = 37.74 t/s 7.42.464.266 I slot print_timing: id 3 | task 838 | n_gen = 385, tg = 42.26 t/s, tg_3s = 46.58 t/s 7.45.502.452 I slot print_timing: id 3 | task 838 | n_gen = 547, tg = 45.03 t/s, tg_3s = 53.32 t/s 7.48.548.148 I slot print_timing: id 3 | task 838 | n_gen = 669, tg = 44.04 t/s, tg_3s = 40.06 t/s 7.51.576.186 I slot print_timing: id 3 | task 838 | n_gen = 794, tg = 43.58 t/s, tg_3s = 41.28 t/s 7.54.584.276 I slot print_timing: id 3 | task 838 | n_gen = 981, tg = 46.21 t/s, tg_3s = 62.17 t/s 7.57.628.255 I slot print_timing: id 3 | task 838 | n_gen = 1151, tg = 47.42 t/s, tg_3s = 55.85 t/s 8.00.639.680 I slot print_timing: id 3 | task 838 | n_gen = 1334, tg = 48.90 t/s, tg_3s = 60.77 t/s 8.03.700.515 I slot print_timing: id 3 | task 838 | n_gen = 1472, tg = 48.51 t/s, tg_3s = 45.09 t/s 8.06.736.566 I slot print_timing: id 3 | task 838 | n_gen = 1607, tg = 48.14 t/s, tg_3s = 44.47 t/s 8.09.746.428 I slot print_timing: id 3 | task 838 | n_gen = 1766, tg = 48.53 t/s, tg_3s = 52.83 t/s 8.12.778.794 I slot print_timing: id 3 | task 838 | n_gen = 1879, tg = 47.66 t/s, tg_3s = 37.26 t/s 8.15.780.626 I slot print_timing: id 3 | task 838 | n_gen = 2015, tg = 47.50 t/s, tg_3s = 45.31 t/s 8.18.818.898 I slot print_timing: id 3 | task 838 | n_gen = 2139, tg = 47.05 t/s, tg_3s = 40.81 t/s 8.21.839.674 I slot print_timing: id 3 | task 838 | n_gen = 2324, tg = 47.94 t/s, tg_3s = 61.24 t/s 8.24.860.269 I slot print_timing: id 3 | task 838 | n_gen = 2504, tg = 48.62 t/s, tg_3s = 59.59 t/s 8.27.920.200 I slot print_timing: id 3 | task 838 | n_gen = 2687, tg = 49.25 t/s, tg_3s = 59.81 t/s 8.30.944.508 I slot print_timing: id 3 | task 838 | n_gen = 2865, tg = 49.75 t/s, tg_3s = 58.86 t/s 8.33.968.319 I slot print_timing: id 3 | task 838 | n_gen = 3040, tg = 50.16 t/s, tg_3s = 57.87 t/s 8.36.988.505 I slot print_timing: id 3 | task 838 | n_gen = 3214, tg = 50.51 t/s, tg_3s = 57.61 t/s 8.39.991.748 I slot print_timing: id 3 | task 838 | n_gen = 3381, tg = 50.74 t/s, tg_3s = 55.61 t/s 8.43.012.437 I slot print_timing: id 3 | task 838 | n_gen = 3538, tg = 50.79 t/s, tg_3s = 51.97 t/s 8.46.070.439 I slot print_timing: id 3 | task 838 | n_gen = 3682, tg = 50.64 t/s, tg_3s = 47.09 t/s 8.49.084.710 I slot print_timing: id 3 | task 838 | n_gen = 3811, tg = 50.33 t/s, tg_3s = 42.80 t/s 8.52.140.302 I slot print_timing: id 3 | task 838 | n_gen = 3935, tg = 49.95 t/s, tg_3s = 40.58 t/s 8.55.163.498 I slot print_timing: id 3 | task 838 | n_gen = 4081, tg = 49.89 t/s, tg_3s = 48.29 t/s 8.58.202.496 I slot print_timing: id 3 | task 838 | n_gen = 4277, tg = 50.41 t/s, tg_3s = 64.49 t/s 9.01.222.112 I slot print_timing: id 3 | task 838 | n_gen = 4473, tg = 50.91 t/s, tg_3s = 64.91 t/s 9.04.261.976 I slot print_timing: id 3 | task 838 | n_gen = 4668, tg = 51.35 t/s, tg_3s = 64.15 t/s 9.07.303.727 I slot print_timing: id 3 | task 838 | n_gen = 4854, tg = 51.67 t/s, tg_3s = 61.15 t/s 9.10.343.426 I slot print_timing: id 3 | task 838 | n_gen = 5029, tg = 51.85 t/s, tg_3s = 57.57 t/s 9.13.360.418 I slot print_timing: id 3 | task 838 | n_gen = 5197, tg = 51.97 t/s, tg_3s = 55.68 t/s 9.16.387.616 I slot print_timing: id 3 | task 838 | n_gen = 5322, tg = 51.66 t/s, tg_3s = 41.29 t/s 9.17.493.354 I slot print_timing: id 3 | task 838 | prompt eval time = 419.12 ms / 4 tokens ( 104.78 ms per token, 9.54 tokens per second) 9.17.493.357 I slot print_timing: id 3 | task 838 | eval time = 104114.81 ms / 5369 tokens ( 19.40 ms per token, 51.56 tokens per second) 9.17.493.358 I slot print_timing: id 3 | task 838 | total time = 104533.92 ms / 5373 tokens 9.17.493.358 I slot print_timing: id 3 | task 838 | graphs reused = 2498 9.17.493.361 I slot print_timing: id 3 | task 838 | draft acceptance = 0.72147 ( 3673 accepted / 5091 generated), mean len = 3.16 submitted by /u/hurdurdur7 [link] [comments]

Source: r/LocalLLaMA | 2026-08-31