Qwen3.8-Flash-Next in llama.cpp from CPU-only to 96GB VRAM: 8.5 to 109 tok/s, max context and parameters test. My findings on RTX 6000 PRO.

Hey guys, I tested Qwen3.8 Flash with llama.cpp from CPU-only to the full 96GB of my RTX PRO 6000. Short version: CPU-only reached 8.34 tok/s at a 2K prompt Full 96GB reached 109.07 tok/s At 245K context, 24GB to 96GB gave 14.89 to 21.61 tok/s The 96GB advantage over 24GB decreased from 2.80x at 2K to 1.45x at 245K Forcing the 27.2 GiB PLE table onto CUDA reduced decode from 108.5 to 1.95 tok/s RAM-resident loading gave 1.87x more prefill than mmap Non-unified KV reached 92.0 tok/s total output at concurrency 16 Setup Model: unsloth/Qwen3.8-Flash-Next-GGUF Quant: UD-IQ4_XS Model size: 87.2 GiB Engine: llama.cpp b10666, revision 4e97ac86e Qwen3.8 merge: 6c84c7d5d, PR #27742 GPU: NVIDIA RTX PRO 6000 Blackwell, 96GB CPU: AMD Ryzen 9 9950X System RAM: 96GB DDR5 OS: Ubuntu CUDA: CUDA 13 I started a fresh server for each configuration. I waited for the previous VRAM allocation to disappear and for the GPU to cool. Each run saved the resolved configuration, server log, output, memory use and GPU telemetry. Important note about the VRAM ranges I used the same RTX PRO 6000 for every GPU test. A helper process reserved GPU memory, so llama.cpp saw a smaller usable VRAM pool. This tests VRAM capacity and CPU offload. It does not simulate the compute power or bandwidth of a real 8GB or 24GB GPU. The 8GB result does not mean that every 8GB card will reach the same speed. VRAM results All numbers below use a 2,048-token prompt. CPU-only: Prefill: 182.64 tok/s Decode: 8.34 tok/s Usable VRAM Expert layers in RAM Prefill Decode 8GB 48 of 48 232 tok/s 35.69 tok/s 16GB 45 of 48 249 tok/s 37.93 tok/s 24GB 42 of 48 260 tok/s 39.01 tok/s 32GB 36 of 48 292 tok/s 42.24 tok/s 48GB 23 of 48 746.7 tok/s 51.73 tok/s 96GB 0 of 48 1,955 tok/s 109.07 tok/s https://preview.redd.it/a5cibex9krmh1.png?width=940&format=png&auto=webp&s=06ae5db9959ccb13dd6cf612cea071fe97e2f3ae All GPU tests use the same RTX PRO 6000. The limits simulate memory capacity, not smaller GPU performance. 1. The model runs on the CPU At a 2K prompt, CPU-only reached 182.64 tok/s prefill and 8.34 tok/s decode. This is enough for an interactive chat. The MoE design helps because the model activates only 6B parameters for each token. 2. The VRAM tiers converge at long context At a 2K prompt, 96GB was 2.796x faster than 24GB. At a 245K prompt, the advantage decreased to 1.451x. Decode at 245,760 prompt tokens: 24GB: 14.89 tok/s 32GB: 15.41 tok/s 48GB: 16.96 tok/s 96GB: 21.61 tok/s https://preview.redd.it/ced41pzdkrmh1.png?width=940&format=png&auto=webp&s=de3dc2112846a5f16663e0235a41bfa0b193b515 Every configuration loses speed at long context. The fastest configuration loses most of its lead. Only 12 of the 48 layers keep a growing attention cache. The other 36 use Gated DeltaNet. This keeps context memory relatively low, but it does not make long-context decode free. 3. PLE on CUDA was 55.6x slower in this build The GGUF contains a 27.2 GiB per-layer token embedding table. I tested two placements: System RAM: 1,967.9 tok/s prefill and 108.5 tok/s decode GPU VRAM: 575.7 tok/s prefill and 1.95 tok/s decode The CUDA placement was 55.6x slower on decode. https://preview.redd.it/laiv0s5hkrmh1.png?width=1596&format=png&auto=webp&s=e8506136d68aefc96cd98ecc2345cd03d19c9781 I verified that the tensor moved, but I did not isolate the cause of the slowdown. I repeated the test in A-B-B-A order. The order effect was 0.56%. The memory data also confirmed the placement. GPU use increased by approximately 27.5 GiB when the table moved to CUDA. The CPU placement needs approximately 9.2 ms per decode token. The CUDA placement needs approximately 513 ms per decode token. That delay is too large to explain with arithmetic alone. It looks more like synchronization or a per-token transfer, but I have not proved the cause. My conclusion is limited to this build: per_layer_token_embd=CUDA0 reduced decode from 108.5 to 1.95 tok/s in llama.cpp b10666. 4. RAM-resident loading gave 1.87x more prefill I compared mmap with RAM-resident loading at the same 48GB tensor placement. At the 2K prompt: RAM resident: 746.7 tok/s prefill mmap mean: 400.4 tok/s prefill Difference: 1.87x The decode ratio was 0.998, so decode was effectively unchanged. The option is: --load-mode none This means that llama.cpp does not use file mapping. It does not mean that the model is not loaded. https://preview.redd.it/rsqb29nlqrmh1.png?width=1538&format=png&auto=webp&s=63125393483aa292b088a718c1c944d086f648ef The mode needs enough free system RAM. 5. KV layout changed concurrency I tested unified and non-unified KV layouts from 1 to 16 concurrent requests. Both started at 59.0 tok/s with one request. At concurrency 16: Unified KV: 68.8 tok/s Non-unified KV: 92.0 tok/s Non-unified KV gave more total output at high concurrency. Each individual request still became slower. Concurrency increased total server capacity. It did not make one request faster. https://preview.redd.it/w8lxh69tkrmh1.png?width=1583&format=png&auto=webp&s=6101f49343c005c148f4842e7f94e002c0353b2f Non-unified KV also divides the available context between slots, so it is not always the correct setting. Results I left out for now The report also contains: Microbatch results https://preview.redd.it/03ztb8toqrmh1.png?width=940&format=png&auto=webp&s=11757b6864725fe34351912beeb121db8464be8d Preserved-reasoning results Long-context retrieval at full range worked. https://preview.redd.it/szw4lwmpqrmh1.png?width=940&format=png&auto=webp&s=66f5a3d2d5195ff440524bafa87a6fc3705676af Q4_K_XL comparison The final numbers are present in my repo and video, Resources GitHub with the report, scripts, configurations, results and graphs: https://github.com/lukaLLM/Qwen3.8-Flash-Next-VRAM-Benchmark Full video: https://youtu.be/RBlRTUwJMI4 PS: AI was abused while making edits My main question is about the PLE CUDA result. Did anybody reproduce the same slowdown with per_layer_token_embd=CUDA0 on another GPU or a newer llama.cpp build? My other question is about keeping earlier thinking in the conversation. I ran the same five-turn coding conversation with earlier reasoning either kept or removed from later prompts. Keeping it reduced prompt tokens recomputed from 18,403 to 267 because the history stayed append-only. However, the turn-5 prompt grew from 18,387 to 63,223 tokens, and decode ended at 48.9 instead of 65.5 tok/s. This was only one run per arm with temperature 1.0. The arms also started at different speeds, 110.2 and 96.0 tok/s, so I do not trust the exact 69x and 25% figures yet. https://preview.redd.it/phvv2snqqrmh1.png?width=940&format=png&auto=webp&s=55c5967698ed1fb0540ff0fc9c48228e51cd72a4 Did anybody reproduce this tradeoff? Does keeping the earlier reasoning normally save this much prompt recomputation, and how much should the longer prompt reduce decode speed? Or any other finding to increase the efficiency of this model. submitted by /u/FantasticNature7590 [link] [comments]

Source: r/LocalLLaMA | 2026-08-31

Related