Model Releases
An 80B MoE (Qwen3-Next, 48 GB GGUF) answering in a Chrome tab: llama.cpp's WebGPU backend, experts paged in from disk
Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU ba
Follow-up to last week's post about streaming MoE experts from disk in a browser tab. That ran on hand-written engines for two models. This week the same disk tier sits under llama.cpp's own WebGPU backend, so it works for the MoE models llama.cpp already runs, with no per-model engine. The video is one real recording in Chrome. The sped-up parts say so on screen, with a real-time clock. Measured in that recording (MacBook Pro M4 Pro, 24 GB): Model (GGUF) File On the GPU Load from disk Decode Qwen3-Next 80B-A3B Q4_K_M 48.4 GB 4.89 GB 48 s 2.9 tok/s (proof of concept) OLMoE 1B-7B Q4_K_M 4.2 GB 3.92 GB 5.3 s 22.4 tok/s LFM2 8B-A1B Q4_K_M 5.0 GB 3.89 GB 6.2 s 17.1 tok/s Qwen3-30B-A3B runs too (14.7 GB file). How it works. The GGUF downloads once into the browser's private storage (OPFS). The GPU holds the dense layers, the KV cache and a fixed number of expert slots per layer, sized from a GPU memory budget you pick. At each MoE layer the router's picks are read back, and any expert not in a slot is read from the OPFS file (sync access handles in workers) into the least recently used slot. The paging is a small change to ggml-webgpu: a new buffer type, slot-mapped MUL_MAT_ID shaders, and the residency logic. Correctness. With paging on and off on the same backend, replies are identical (9/9 conversations on OLMoE and LFM2; on the two big ones, which do not fit unpaged, 8 vs 32 or 16 vs 64 slots give identical replies). Against llama.cpp on Metal, 4 to 8 of 9 replies match. The misses are WebGPU-vs-Metal rounding at near-ties: against llama.cpp's CPU backend, the WebGPU path is as close as Metal is. Limits. The 80B is slow: it reads about 359 MB from disk per token. Prompt processing is still slow (last week's fair complaint). Chrome or Edge 137+ only; tested on Apple silicon only. The first load copies the whole file into browser storage. The 80B model's own answer, asked how it could run in a browser tab: "It can't." It was running in a browser tab. Try it: https://diskformer.naklitechie.com · Code: https://github.com/NakliTechie/diskformer.js · submitted by /u/naklitechie [link] [comments]
Related
- Gemma 4 26B-A4B and a 37 GB Qwen3.6 MoE running in a browser tab on a 24 GB Mac — experts streamed from disk, output matches llama.cpp
- 1-bit 27B in the browser: 25–30 tok/s on a 6 GB RTX 3060 Laptop (WebGPU, no install)
- Qwen3.8-Flash-Next 177B NVFP4(119GiB): SSD streaming at 9-10 tok/s on one 16 GB RTX 5060 Ti + 32 GB RAM
Source: r/LocalLLaMA | 2026-10-13