Qwen3.8-Flash-Next on a 96GB Mac Studio (here's my memory math, tell me where it's wrong)

Mac Studio, 96GB unified memory (M3 Ultra). I want the largest usable Qwen3.8-Flash-Next setup, and I'd rather not burn 100GB of bandwidth on the wrong download. Here's my math. Please tell me which parts are wrong. What makes this model weird 177B params total: a 125B MoE trunk + a 51B n-gram embedding table + a 4B MTP head. The table isn't compute, it's a lookup — 20M hashed bigram/trigram entries, 320M rows of 160 values, and each token touches exactly 16 rows, roughly 2.7 KB. That's the one part that can plausibly live on SSD instead of RAM. KV is unusually cheap too. Only 12 of the 48 layers are real attention (2 KV heads, head dim 256); the other 36 are Gated DeltaNet with a fixed-size state. That works out to 24 KiB/token: 262K native: 6.4 GB 1M with YaRN: ~25 GB So on this architecture the KV cache is basically not my problem. The resident weights are. My budget 96 GB total. sudo sysctl iogpu.wired_limit_mb=87040 gives the GPU ~85 GB and leaves ~11 GB for macOS. So resident weights + KV + compute buffers have to stay under roughly 85 GB. The options as I understand them Build File total Resident with n-gram on SSD Headroom for KV unsloth UD-Q4_K_XL 111.3 GB ~73–80 GB (my estimate) 5–12 GB unsloth UD-IQ4_XS 93.7 GB ~65–67 GB (my estimate) ~18 GB AtomicChat AD-4.27bpw 92.9 GB 54.5 GB (published) ~30 GB mlx-serve 4-bit pack 73.4 GB + 32 GB table ~70–75 GB (published) ~10–15 GB mlx-lm-style 4-bit pack ~98 GB ~107 GB — table goes to the GPU doesn't fit Reference points I've found, all on bigger machines: a 128GB M5 Max ran UD-IQ4_XS at the full 262K window with ~106 GB wired, and a DGX Spark ran UD-Q4_K_XL at 128K with the table mmap'd at ~89 GB used. Nothing at 96GB. Where I'm not sure 1. Does the n-gram mmap actually free anything on Metal? AtomicChat's model card claims llama.cpp hands Metal the entire mmap'd region of any shard that contains GPU tensors, so a table interleaved with normal weights gets wired anyway and you die with kIOGPUCommandBufferCallbackErrorOutOfMemory. Their builds isolate the table in its own shard for exactly this reason. Unsloth's UD-Q4_K_XL is split purely by size (10.9 MB / 49.9 / 49.4 / 12.1 GB), so the table is definitely sitting next to weights. If that claim still holds on current master, Q4_K_XL is effectively fully resident and dead on arrival at 96GB. Has anyone actually confirmed this either way on Apple Silicon? 2. How big is per_layer_token_embd in UD-Q4_K_XL**, exactly?** I'm inferring 30–38 GB from the file totals and the fact that the tensor has row width 160, so only block-32 types apply (IQ4_NL / Q4_0 / Q4_1 / Q5_0 / Q5_1 / Q8_0). That's the single biggest error bar in my whole calculation. If someone has it loaded, a tensor dump would settle it. 3. Is re-splitting worth the hassle? llama-gguf-split --merge, then re-split with --split-max-size 2G to isolate the table into its own shard. Needs ~220 GB free temporarily. Has anyone done this on the unsloth files and confirmed it fixes the Metal wiring behaviour? 4. llama.cpp or MLX right now? oMLX still can't load qwen4_exp (issue #3170, open, no PR), and the mlx-lm PR is unmerged, but the mlx-serve packs keep the table in a separate mmap'd .bin and report ~60 tok/s decode, 78 with MTP, on an M4 Max. llama.cpp's MTP is deferred to a follow-up. Is MLX genuinely ahead on Apple Silicon for this model, or is that a snapshot that'll be stale in a week? 5. Quantized KV. I've seen the qwen4exp assert with -ctk q8_0 -ctv q8_0 and the LLAMA_ATTN_ROT_DISABLE=1 workaround. Fixed on master yet, or should I just plan for f16 KV? At 24 KiB/token I can probably afford f16 anyway. 6. Anyone actually on 96GB? Every data point I can find is a 128GB Mac or a DGX Spark. What quant and what context length are you running, and what decode speed do you get? If the answer is "just get UD-IQ4_XS, run it at 262K, stop overthinking it," I'll happily take that too. But I'd like to know whether the offload path works on Metal before I commit, because that's the difference between a 4-bit build and a 2-bit one on this machine. submitted by /u/Mxmtm [link] [comments]

Source: r/LocalLLaMA | 2026-08-31