Muse Glimmer on 1/2 AMD v620

Hey. Just tried it on my old ass gpus 😄 Surprisingly Tensor Split is working on 2 gpus almost doubling PP (wonder how it will work with 4 gpus) Q6 — 1 GPU llama-server --model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q6_K_XL.gguf --mmproj <MODEL_DIR>/Muse-Glimmer-30B-GGUF/mmproj-kquant.gguf --spec-draft-model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/dflash-kquant.gguf --spec-type draft-dflash --spec-draft-ngl 999 --spec-draft-n-max 3 --spec-draft-type-k f16 --spec-draft-type-v f16 --ctx-size 65536 --override-kv muse-glimmer.context_length=int:65536,dflash.context_length=int:65536 --n-gpu-layers 999 --device ROCm0 --device-draft ROCm0 --split-mode layer --flash-attn on --fit off --parallel 1 --kv-unified --batch-size 2048 --ubatch-size 512 --threads 32 --threads-batch 32 --cache-type-k f16 --cache-type-v f16 --image-min-tokens 1024 --image-max-tokens 4096 --reasoning-preserve --temp 0.7 --top-p 0.95 --top-k 64 --min-p 0.0 --jinja Q8 — 2 GPUs with tensor split bash llama-server --model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q8_K_XL.gguf --mmproj <MODEL_DIR>/Muse-Glimmer-30B-GGUF/mmproj-kquant.gguf --spec-draft-model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/dflash-kquant.gguf --spec-type draft-dflash --spec-draft-ngl 999 --spec-draft-n-max 3 --spec-draft-type-k f16 --spec-draft-type-v f16 --ctx-size 65536 --override-kv muse-glimmer.context_length=int:65536,dflash.context_length=int:65536 --n-gpu-layers 999 --device ROCm0,ROCm1 --device-draft ROCm1 --split-mode layer --tensor-split 1,1 --flash-attn on --fit off --parallel 1 --batch-size 2048 --ubatch-size 512 --threads 32 --threads-batch 32 --cache-type-k f16 --cache-type-v f16 --image-min-tokens 1024 --image-max-tokens 4096 --reasoning-preserve --temp 0.7 --top-p 0.95 --top-k 64 --min-p 0.0 --jinja --host 127.0.0.1 --port 18088 Q6 — 2 GPUs with tensor split bash llama-server --model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q6_K_XL.gguf --mmproj <MODEL_DIR>/Muse-Glimmer-30B-GGUF/mmproj-kquant.gguf --spec-draft-model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/dflash-kquant.gguf --spec-type draft-dflash --spec-draft-ngl 999 --spec-draft-n-max 3 --spec-draft-type-k f16 --spec-draft-type-v f16 --ctx-size 65536 --override-kv muse-glimmer.context_length=int:65536,dflash.context_length=int:65536 --n-gpu-layers 999 --device ROCm0,ROCm1 --device-draft ROCm1 --split-mode layer --tensor-split 1,1 --flash-attn on --fit off --parallel 1 --batch-size 2048 --ubatch-size 512 --threads 32 --threads-batch 32 --cache-type-k f16 --cache-type-v f16 --image-min-tokens 1024 --image-max-tokens 4096 --reasoning-preserve --temp 0.7 --top-p 0.95 --top-k 64 --min-p 0.0 --jinja --host 127.0.0.1 --port 18090 Benchmark command: bash python3 <BENCH_DIR>/benchmark.py --base-url http://127.0.0.1:18090 --api-key sk-local --profile q6-tensor-2gpu --output <BENCH_DIR>/q6-tensor-2gpu.json Results Benchmark Q6 1 GPU Q6 tensor split, 2 GPUs Q8 tensor split, 2 GPUs 4k prompt processing 355.38 tok/s 472.27 tok/s 550.04 tok/s 16k prompt processing 372.57 tok/s 552.96 tok/s 657.84 tok/s 32k prompt processing 351.37 tok/s 536.88 tok/s 634.82 tok/s 60k prompt processing 320.99 tok/s 503.96 tok/s 590.64 tok/s 256-token generation 35.38 tok/s 36.32 tok/s 26.55 tok/s Vision-chat generation 32.08 tok/s 32.83 tok/s 25.17 tok/s Text DFlash acceptance 176/237, 74.3% 176/237, 74.3% 159/286, 55.6% Vision DFlash acceptance 20/31 20/31 19/33 Any ideas on how to improve that performance? Tbh this already looks like pretty close what I had with Qwen3.6-27B-MTP submitted by /u/Thin_Pollution8843 [link] [comments]

Source: r/LocalLLaMA | 2026-08-10

Related