2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next

I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about 3 times as expensive, I had a "sweet spot", kind of. On the last prime days, I found a X870E mainboard for ~150 € below the standard price, and it got to my head that I can use an upgraded machine for gaming and local inference tinkering. Anyways. Fast forward to this week, I now have the following setup Ryzen 7500F 64 GB DDR5 CL40 6400 MT/s Asus ProArt Creator X870E 2x R9700 32 GB, each running at PCIe 5.0 x8 (Gigagbyte) Currently running ubuntu on an old Samsung EVO 860 1 TB drive; this will become intersting for the ngram / PLE offload; I have Windows and the gaming related stuff on a gen4 NVMe, but will soon add another Gen 5 NVMe with decent random reads The only issue that I can report so far is that one of the cards runs quite hot, so I will definitely implement power limiting to 210 W and some light undervolting. The other card runs 10-15 °C cooler.. Case is a purebase 501 with 4 fans, 2 intake in front, one back and top for output. Now long story short I wanted to give some results of Qwen 3.8 27b FP8 and MXFP4, as well as Qwen 3.8 flash next after the first day tinkering with it. What I found super interesting is that the SATA SSD does not seem to be super terrible when using Qwen 3.8 flash next. Considering the whole build costs ~4k €, or more than 1k less than a single RTX 5090 with 32 GB, I kinda like this setup price/performance wise. Next step is checking context degradation / KV quants. I am using local inference mostly for deep research, summarization, image creation, light coding and non-trivial data analysis Cheers Qwen3.8 benchmarks on 2× Radeon AI PRO R9700 Hardware: 2× AMD Radeon AI PRO R9700 32 GB, 61 GiB system RAM Benchmark: BetterBench 0.2.2, corpus v1.0, single-stream, greedy decoding, 2 warm-ups + 10 measured runs per category, 8k benchmark context. Model Weight format Runtime Server context Max sequences Speculative decoding Weighted decode median ITL 1% low TTFT p50 Prefill ~2k Prefill ~4k Prefill ~7k Qwen3.8-27B Quark AWQ MXFP4 vLLM Radiance, TP2 131,072 1 MTP, up to 8 tokens 111.4 tok/s 77.9 tok/s 81 ms 4,224 tok/s 4,322 tok/s 4,410 tok/s Qwen3.8-27B Native block FP8 vLLM Radiance, TP2 16,384 8 MTP, up to 8 tokens 87.6 tok/s 61.9 tok/s 73 ms 4,134 tok/s 4,329 tok/s 4,305 tok/s Qwen3.8-Flash-Next UD-IQ4_XS GGUF R9V/vLLM, TP2, tiered expert offload 131,072 1 MTP, 2 tokens, FP8 draft 35.4 tok/s 27.3 tok/s 290 ms 1,727 tok/s 1,986 tok/s 1,925 tok/s Qwen 3.8 27b in FP8 and AWQ MXFP4 served with vLLM Radiance Qwen 3.8 Flash next served with vLLM / R9V fork Decode metrics come from the 10-pass standard run. Prefill measurements use cold, nonce-prefixed prompts. Prompt-token medians for the prefill columns were 1,556, 3,024 and 5,226 tokens. No concurrency sweep was included in these results. I expect decode of Flash next to increase a bit when an NVMe is used, and, as I am writing this and checked, I found EXPO was not enabled........oh my god I swear I turned it on when I updated the bios yesterday submitted by /u/smallDeltaBigEffect [link] [comments]

Source: r/LocalLLaMA | 2026-09-06

Related