Model Releases
50% tg increase with offloading 'hot' experts to VRAM
I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next. The idea is simple: instead of offloading entire layers to
I got a 50% performance boost (20 t/s -> 30 t/s) in llama.cpp for MoE models that don’t fit entirely in VRAM—in my case, Qwen 3.8 Flash Next. The idea is simple: instead of offloading entire layers to the GPU, I offload only the “hot” experts. I found that certain groups of experts remain relatively stable across coding, refactoring, and code-review workloads. https://github.com/timadinorth/llama.cpp/pull/1 A couple of important caveats: this llama.cpp fork has been tested only on coding workloads, and it’s useful only when the full model cannot fit in VRAM. Will upstream ever accept it? Probably not. Opus did the low-level implementation, and I don’t feel like showing up to explain every line submitted by /u/nbvehrfr [link] [comments]
Related
- A llama.cpp PR caches “hot” MoE experts on the GPU — 33 → 56 tok/s reported with 8GB VRAM
- MoE CPU-offload benchmark on Deepseek V4/Gemma4/Qwen/GPT-OSS — TensorSharp vs llama.cpp
Source: r/LocalLLaMA | 2026-08-29