Model Releases
EXPERIMENT: Qwen3.8-2.4T-A95B running locally on an RTX 5090 + RTX 5060 Ti at ~0.80 tok/s
I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397
I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397 GiB. The model uses 512 routed experts, with 10 active per token. My hardware: CPU: AMD Ryzen 9 9950X3D 16 cores / 32 threads GPU 0: NVIDIA RTX 5090 32 GB VRAM GPU 1: NVIDIA RTX 5060Ti 16 GB VRAM RAM: 128 GB DDR5 5600 CL36 Swap: 350 GB OS: Arch Linux llama.cpp: Unsloth build 10360 The interesting part is that I also managed to get the model’s native MTP speculative decoding working. After testing several configurations, the best settings I found were: --tensor-split 4,1 --n-cpu-moe 91 MTP n_max = 3 MTP p_min = 0.5 MTP routed experts from block 92 forced to CPU The block 92 expert tensors had to stay on CPU because otherwise MTP would run out of VRAM. Performance For a controlled 32-token generation test: Prompt tokens: 16 Generated tokens: 32 Prompt speed: 0.8299 tok/s Generation speed: 0.8033 tok/s Decode time: 39.84 s Total wall time: 59.14 s MTP draft tokens: 21 MTP accepted: 19 MTP acceptance: 90.48% Without MTP, the same test gave: Generation speed: 0.7750 tok/s Decode time: 41.29 s Total wall time: 60.81 s So my final MTP configuration gives roughly: +3.64% generation throughput -2.74% total wall time Not a huge speedup, but considering this is a 2.45T parameter model running locally on consumer hardware, I’ll take it. I also tested different MTP settings: n_max=2 -> too conservative n_max=3 -> best n_max=4 -> much slower p_min=0.0 -> too many bad speculative tokens p_min=0.5 -> best result p_min=0.7 -> too aggressive Interestingly, p_min=0.7 reached 100% draft acceptance, but it generated too few useful speculative tokens and ended up slower than p_min=0.5. Final llama-server command MODEL="/home/marcin/Pobrane/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf" CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 LLAMA_ATTN_ROT_DISABLE=1 /home/marcin/Pobrane/unsloth-qwen38-runtime/llama.cpp/build/bin/llama-server --model "$MODEL" --alias Qwen3.8-2.4T-A95B --device CUDA0,CUDA1 --main-gpu 0 --split-mode layer --tensor-split 4,1 --n-gpu-layers all --n-cpu-moe 91 -ot 'blk92ffn_(down|gate|up)_expsweight=CPU' --fit on --fit-target 2048,1536 --no-op-offload --ctx-size 512 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --batch-size 128 --ubatch-size 32 --threads 16 --threads-batch 16 --parallel 1 --spec-type draft-mtp --spec-draft-device CUDA0 --spec-draft-ngl all --spec-draft-cpu-moe --spec-draft-n-max 3 --spec-draft-p-min 0.5 --temp 0 --top-k 0 --top-p 0.95 --min-p 0.0 --cache-ram 0 --ctx-checkpoints 0 --load-mode mmap --no-warmup --no-ui --host 127.0.0.1 --port 8200 --metrics With MTP enabled, VRAM usage after generation is roughly: RTX 5090: 29.6 GB used / ~2.6 GB free RTX 5060 Ti: 12.3 GB used / ~3.6 GB free One important info: 0.80 tok/s is from a controlled 32-token test, not a large multi-prompt benchmark, so I wouldn’t present it as a universal average for every workload. Longer prompts, context size, output patterns and expert routing can change the result. Still, seeing a 2.4T MoE model actually generating locally on a 5090 + 5060 Ti at around 0.8 tok/s is pretty wild. submitted by /u/mossy_troll_84 [link] [comments]
Related
- Nifer is insane. 700t/s with Qwen 3.6 35B (no thinking). Purpose build for RTX5090. Full 250k context too.
- Running Qwen 3.6 35B A3B-Q8_0 gguf on a cheap radeon 7600 at 18 token/s * update increased to 21 t/s
- Running Qwen 3.5 35B A3B-Q8_0 gguf on a cheap radeon 7600 at 18 token/s
Source: r/LocalLLaMA | 2026-08-13