Model Releases
Qwen3.8-Flash-Next 176B on a 16GB card: yes it works, here's how
Got the new Qwen3.8-Flash-Next (176B total, 6B active MoE) running locally and figured I'd share for anyone else wondering if their setup can handle it. **My hardware:** 192GB DDR5 RAM, RTX 5070 Ti (1
Got the new Qwen3.8-Flash-Next (176B total, 6B active MoE) running locally and figured I'd share for anyone else wondering if their setup can handle it. My hardware: 192GB DDR5 RAM, RTX 5070 Ti (16GB), Linux. The trick is you don't need a big GPU — you need big RAM. This model runs with the experts CPU-offloaded (--cpu-moe) while attention lives on GPU. Setup that works: - llama.cpp built from the PR branch (PR #27742, still unmerged as of writing — GGUF repos like DevQuasar's say "EXPERIMENTAL: use this branch") - Q8_0 GGUF, ~175GB across 12 shards. llama handles the split transparently — point it at shard 1 - 262K context, q8_0 KV cache - Getting ~16 tok/s generation, which is perfectly usable for chat Memory notes: - q8_0 KV at full 262K context = ~14GB — fits my 16GB card with room - The 175GB model sits in system RAM. Tight on 192GB but works. 128GB machines would struggle, 256GB would be comfy All the local GGUF quants are already up (Unsloth, DevQuasar etc.) — no need to convert from safetensors yourself. Save your bandwidth/disk. Happy to answer setup questions in comments. For anyone asking about offloading the n-gram lookup table to SSD in llama.cpp — yes, the flags exist. I didn't mention it in my original post because I have 192GB RAM so there's no need for it in my setup, but here's how to do it if you're RAM-constrained: Enable n-gram speculative decoding: --spec-type ngram-simple Available types (comma-separated, you can stack multiple): • ngram-simple — basic n-gram lookup • ngram-map-k — n-gram map (K cache only) • ngram-map-k4v — n-gram map (K+V cache) • ngram-mod — modular n-gram with configurable N range • ngram-cache — n-gram with persistent cache Persist the lookup table to SSD: --lookup-cache-static /path/to/ssd/cache.bin Loads a static cache file, not updated during generation. --lookup-cache-dynamic /path/to/ssd/ngram_cache.bin Loads AND updates the cache file during generation, so it persists and improves across runs. This is the one you want for SSD offloading. Tuning the n-gram sizes: --spec-ngram-simple-size-n 12 — lookup n-gram length (how many tokens to match on) --spec-ngram-simple-size-m 48 — draft m-gram length (how many tokens to generate as a draft) --spec-ngram-simple-min-hits 1 — minimum hits before accepting a draft For ngram-mod specifically: --spec-ngram-mod-n-min 48 — min n-gram tokens --spec-ngram-mod-n-max 64 — max n-gram tokens --spec-ngram-mod-n-match 24 — lookup match length Full example with SSD cache offload: llama-server --model model.gguf --spec-type ngram-cache --lookup-cache-dynamic /mnt/ssd/ngram_cache.bin --spec-ngram-simple-size-n 12 --spec-ngram-simple-size-m 48 Use a fast NVMe SSD for this — SATA SSDs will likely be too slow to see a real benefit. submitted by /u/Available-Confusion2 [link] [comments]
Related
Source: r/ollama | 2026-08-27