Model Releases
10%+ performance improvement on MoE ssd-streaming with expert-lookahead
I implemented expert lookahead, a trick that gets a 10%+ performance improvement on MoE models running on low-memory macs (Qwen 3.8 flash in this case) while using expert-offloading/ssd-streaming (usi
I implemented expert lookahead, a trick that gets a 10%+ performance improvement on MoE models running on low-memory macs (Qwen 3.8 flash in this case) while using expert-offloading/ssd-streaming (using slotstream). This is 10% on top of several optimizations. I initially trained a small model to predict the experts N layers ahead so I could load them into memory in advance, but it turns out the model is its best predictor. After many experiments I ended up with N = 2, so while the current computation runs, slotstream runs qwen's router 2 layers ahead on the current hidden state to predict the next experts to load. Right now I'm training a small correction model on top, and it gives me an additional ~3-4% improvement. expert-lookahead for Qwen 3.8 flash: https://github.com/carloslfu/slotstream/blob/main/docs/EXPERT-LOOKAHEAD.md submitted by /u/carloslfu [link] [comments]
Related
- 50% tg increase with offloading 'hot' experts to VRAM
- Qwen3.8-Flash-Next optimised for Macs
- What settings do you use for running Qwen3.8-Flash-Next in llama.cpp?
Source: r/LocalLLaMA | 2026-09-15