I spent months pushing my RX 6700 XT into local AI. I think I seriously underestimated this GPU.

My setup: RX 6700 XT 12GB i5-11400F 16GB RAM Ubuntu 26.04 ROCm/TheRock + llama.cpp I've spent the last few months experimenting with different backends, quantization, CPU/GPU split, KV cache, batch sizes, power limits, and increasingly large context windows. Eventually I got a 35B A3B MoE running with up to 256K context. And here's the weird part: The thing I care about most isn't decode speed. At ~20–24 tok/s, the 6700 XT isn't going to win any decode benchmark. 😂 What surprised me is how useful the GPU becomes when you look at prefill, long-context processing, and latency instead. I've managed to get long-context prefill into the hundreds of tok/s, with some runs going substantially higher depending on the configuration. So I'm starting to think we're asking the wrong question about older GPUs. Instead of: “How many tokens per second can it generate?” I'm more interested in: “How much work can I move onto the GPU before the system starts falling apart?” I'm curious: Anyone else here pushing a 6700 XT / 6750 XT hard for local LLMs? I'd love to compare setups. submitted by /u/Full_Director87 [link] [comments]

Related

• I built a Windows GUI launcher to benchmark and manage multiple llama.cpp builds (useful for AMD GPU users juggling Vulkan/ROCm/HIP builds)
• Your model fits in VRAM. Your context doesn't. The KV cache math nobody shows you.
• How we optimized a local Llama 3 agent: From 15s latency and 68% accuracy to 4s and 100% (Full E2E Code & Guide)

Source: r/ollama | 2026-08-23