Model Releases
llama.cpp ngram on RAM/SSD?
I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but na
I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but naiive attempts failed because even at Q4 it seems to load God knows what to God knows where. Would appreciate if someone share the recipe, or tell what are the official plans to support this (I can wait, knowing that it is upcoming). Thank you in advance. submitted by /u/NickNau [link] [comments]
Related
- What settings do you use for running Qwen3.8-Flash-Next in llama.cpp?
- Qwen-3.8-Next-Flash Ngram Hot-Swappable Knowledge Injector for llama.cpp
- Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph
- Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage
Source: r/LocalLLaMA | 2026-09-11