300b on 32gb MoE-streaming findings + optimisations

The past week I've been running DSv4 inference on my laptop by keeping everything RAM-resident except the MXFP4-experts (since expert pool is ~147GB and won't fit) TL;DR - read speed is the limiter more than the kernels; repacking to enable sequential reads rather than random reads works, pipelining to hide reads behind compute is possible at prefill time (less so at decode time), speculative expert prefetch at prefill works and does help, caching can counterintuitively slow things down by double-buffering, and even tiny prompts hit most of the experts so TTFT is always going to be slow -- Read speed is the biggest limiter unsurprisingly, which has meant pipelining has been the biggest thing to attack (as opposed to kernel / compute optimisations). I wanted to share various optimisations I've been working on that have helped or been informative: - Repacking the model to be sequential rather than random read: repacking to a layer-major style means each layer's experts are now one contiguous blob, so a layer's reads are big sequential slabs at full device rate (~7GB/s) instead of slower random reads - Weirdly enough caching was sometimes worse on my laptop than no caching: From my tests page cache actually double-buffers 100GB+ streams because it goes from SSD to CPU, then CPU to GPU, instead of just straight to GPU, which made everything ~3× worse since it chewed up a lot of byte transfer bandwidth - At prefill, attention/router compute is big enough to hide reads under if the prompt is big, say >2k toks. But it's a bit of a catch-22 since you don't know what experts will be needed until the router has run. The solution was largely speculative expert prefetch - using router hints to predict layer l+1's experts and have the SSD read them during the current layer l's compute - If the prompt is small, say ~30toks, the opposite is true - even a 30tok prompt still touches a huge fraction of all experts across 43 layers (union of top-6 routings ≈ most of the pool). Compute tucks 100% into the reads so TTFT is always slow submitted by /u/maddie-lovelace [link] [comments]

Source: r/LocalLLaMA | 2026-08-09

Related