2x Strix Halo speed-up with an R9700

Forgive the typos and rambling - human actually wrote this post 😂 I've had a Strix Halo board for about a year and been playing around with it for various projects when it's not just being a beefy linux machine. I ordered the 128GB Framework Desktop board pre-panic and I'm very grateful for that. I also grabbed an R9700 Pro AI card late last year for another machine, thinking it would be fun to compare the two. I ended up parting out the machine the R9700 was in for something else and wondered what might be possible with the R9700 in the Strix Halo machine. On the Framework desktop board, there's an x4 4.0 slot hanging out. I already had an x4 extension cable so I could mount a 25G card in it, but a GPU would fit just fine too. I have my board in a Fractal Design case instead of the framework shell (bought the bare board), so I had plenty of room for the card and my power supply had the new 12V connector. Even with today's pricing, a Framework Strix Halo 128GB board and an R9700 is about ~5k all in, so similar price to a DGX spark but with a little more RAM (~160GB, obv with caveats), and it's a regular 16-core ryzen PC instead of the tacky gold box. So, the kicker is that it works. 49 tok/s, 682 tok/s prefill at 32K - double the stock 24 tok/s and 2.5x prefill on Qwen-3.5-122b. Getting this working was a little bit of a mind-bender, so wanted to share with people. Here's how it works. We can't just slap part of the model on the R9700 and expect it to be good though. It's actually worse if you try to do that in most cases. First, we need to place the parts of the model that benefit from the different parts of the hardware. So, with a big MoE model like this, we have a bunch of data that only gets touched for some tokens and those routed experts need to get put on the Strix in the bigger unified memory pool. It works out to about 62GB of the 71GB model, but we might only read 2GB of it per token. The dense parts of the model are about ~4GB and since they get touched for every token, we can put that on the R9700 where we have more compute and memory bandwidth. So we put KV cache, the dense part of the model, and critically, the MTP drafter on the R9700. We can stuff the remaining VRAM on the R9700 with as many layers as fix, which in my setup was 14. This all works because only about 12KB of data per token needs to cross that narrow x4 4.0 link, so as long as the latency isn't bad, it doesn't matter. Trying to do something like Tensor Parallelism across these two would not work well because of that bottleneck. Here's part of the config: llama-server -m Qwen3.5-122B-A10B-Opus-Reasoning-Q4_K_XL.gguf -dev ROCm0,ROCm1 -ts 1,0 --fit off -ngl 999 -fa on --jinja --no-mmap -ot 'blk.(1[4-9]|[2-4][0-9])ffn_(gate|up|down)_exps=ROCm1' -c 32768 -ub 4096 -b 4096 -md mtp-draft-out-q4_K.gguf --spec-type draft-mtp -devd ROCm0 --spec-draft-n-max 4 --spec-draft-p-min 0.5 I kept going on tuning, and tried to reduce the number of kernel launches, which seemed to be holding back performance. I wasn't hitting anywhere near the right numbers per the theoretical bandwidth for each device. I made some updates to llama to make this work, linked on github below. The variant of the model I was using is also linked below, which is a fine tune that I requantized and grafted on an MTP head for my use on a different project. Please let me know if you have questions or ideas on how to improve this! I really feel like this heterogenous setup is on to something and might be how things look going forward. https://github.com/sixvolts/llama-halo-hybrid https://huggingface.co/SixVolts/Qwen3.5-122B-A10B-Opus-Reasoning-MTP-GGUF Epilogue - Funny enough, as I was writing up this project Alex Ziskind drops a video about something similar (sort of) this morning. He used an NVIDIA GPU on an occulink dock, which is worse for a few reasons, but primarily because he's stuck using vulkan for everything with two different GPU vendors. His setup is a $15,000 combo, so it's kind of impractical. Qwen 3.8 flash-next also dropped today, so I need to figure out if I can tune that to run as well. This has been possible for well over a year, but not with plug and play configs. Here the's link to his video if you are curious - https://www.youtube.com/watch?v=Idoif7BzZbg submitted by /u/darklordfireape [link] [comments]

Source: r/LocalLLaMA | 2026-08-26

Related