ROCm 7.2.1 + llama.cpp on mixed Vega GPUs (gfx900 Vega64 + gfx906 MI50) — got 52-57 tok/s on Gemma 3 27B Q4 with patched rocBLAS payloads [repo + Dockerfile]

I'm documenting a reproducible setup for running current llama.cpp on a mixed AMD Vega system with both: Radeon RX Vega 64 / Vega10 / gfx900 (8GB HBM2) Radeon Instinct MI50 (Vega20) / gfx906 (16GB HBM2) presenting as Radeon VII Repo + Dockerfile + test: https://github.com/anng-phtk/rocm-vega-llama.cpp Why this exists Modern ROCm still enumerates these old Vegas via HIP, but prebuilt rocBLAS/Tensile payloads no longer include legacy arches. So: rocminfo shows gfx900 and gfx906 HIP compiles llama-server --list-devices works first real GEMM fails because rocBLAS kernels are missing This setup combines: - ROCm 7.2.1 image patched for gfx906 (mixa3607/rocm-gfx906:7.2.1-complete) - gfx900 rocBLAS/Tensile payloads copied from ROCm 6.3.4 - llama.cpp built for GPU_TARGETS="gfx900;gfx906" Tested hardware Ryzen 5 3600 / 64GB DDR4 Vega 64 8GB (gfx900) MI50 16GB (gfx906) Docker with /dev/kfd + /dev/dri Result: Mixed Vega64 + MI50 runs Gemma 3 27B Q4_K_M under ROCm 7.2.1 once both payloads are present. This is NOT officially supported — experimental, use at own risk. 1. Build the mixed image Create Dockerfile.rocm72-gfx900-gfx906: FROM rocm/dev-ubuntu-22.04:6.3.4 AS rocm6-libs RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends rocblas && rm -rf /var/lib/apt/lists/* FROM mixa3607/rocm-gfx906:7.2.1-complete COPY --from=rocm6-libs /opt/rocm/lib/rocblas/library/ /tmp/rocm6-rocblas/ RUN set -eux; ROCBLAS_LIB="/opt/rocm/lib/rocblas/library"; find /tmp/rocm6-rocblas -maxdepth 1 -type f -name 'gfx900' -exec cp -v {} "ROCBLAS_LIB/" ;; rm -rf /tmp/rocm6-rocblas WORKDIR /workspace CMD ["/bin/bash"] Build: sudo docker build -t rocm72-gfx900-gfx906 -f Dockerfile.rocm72-gfx900-gfx906 . Final image: ~128 gfx900 files + 156 gfx906 files (.co, .hsaco, .dat) 2. Verify enumeration sudo docker run --rm -it --device=/dev/kfd --device=/dev/dri --group-add video --group-add render rocm72-gfx900-gfx906 bash rocminfo | grep -E 'Name:.*gfx(900|906)' Expected: Name: gfx900, Name: gfx906 3. Prove rocBLAS actually runs on gfx900 Device detection is not enough. SGEMM test on device 0 (Vega64): HIP devices: 2 device 0: AMD Radeon RX Vega arch=gfx900 device 1: AMD Radeon VII arch=gfx906 Calling rocblas_sgemm on device 0... rocblas_sgemm returned: 0 hipDeviceSynchronize: no error Test program is test-rocblas.cpp in the repo. 4. Build llama.cpp for both Mount host llama.cpp to keep build persistent: sudo docker run --rm -it -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add video --group-add render -v /home/USER/Development/llama.cpp:/workspace/llama.cpp -v /home/USER/Downloads:/models rocm72-gfx900-gfx906 bash Inside: cd /workspace/llama.cpp cmake -B build-rocm72-dual -DGGML_HIP=ON -DGPU_TARGETS="gfx900;gfx906" -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_SERVER=ON cmake --build build-rocm72-dual --target llama-server -j(nproc) ./build-rocm72-dual/bin/llama-server --list-devices Should show ROCm0: Vega 64 (8176 MiB) + ROCm1: Radeon VII (16368 MiB) 5. Mixed-GPU run — the important part ./build-rocm72-dual/bin/llama-server -m /models/gemma-4-26B-A4B-it-UD-Q4_K_M.gguf --device ROCm0,ROCm1 -ngl 999 --split-mode layer --tensor-split 1,4 -c 8192 -np 1 -b 512 -ub 256 --host 0.0.0.0 --port 8080 Why 1,4 not 1,2: 1,2 overloaded the 8GB Vega64 and caused multi-second stalls — browser unresponsive, then inference resumed. 1,4 left headroom and was smooth. Observed with 1,4: ~57.6 tok/s low context ~56.5 tok/s @ 3.6K ~55.5 tok/s @ 4.4K ~54.3 tok/s @ 6.6K ~53.8 tok/s near 8K ~52.8 tok/s @ 8191/8192 no long stalls 6. Single MI50 baseline Same model on single MI50: ~58-59 tok/s low context ~52-53 tok/s near 8K Way better than Vulkan path for this model. 7. Notes Keep custom ROCm image separate from host Use bind mounts so docker --rm is safe HIP_VISIBLE_DEVICES=1 for single-GPU tests Don't set HSA_OVERRIDE_GFX_VERSION — real gfx900/gfx906 enumerate correctly here Credits: mixa3607/rocm-gfx906:7.2.1-complete, daimonionnn/amd-vega-rocm-vulkan-llm-toolkit, Frigate Vega backport experiments. Full Dockerfile + test.cpp + commands in repo: https://github.com/anng-phtk/rocm-vega-llama.cpp Happy to answer questions about mixed Vega setups. I tested on real Vega64 + MI50 hardware and I have an extra Vega64 if anyone in US is looking to replicate — DM me. submitted by /u/ImageTrick4571 [link] [comments]

Source: r/ollama | 2026-09-13

Related