Qwen 3.8 27b in 24gb of VRAM

Thought i would just add my own flags here for llama.cpp (literally pulled and rebuilt latest today). Running on a 4090 FE and 48gb of DDR4 ram on WSL2. Basically you have 3 knobs you can tune for maximum context. I prefer to keep my kv cache at q8_0 maximum (though i am interested in comparing q6_0 eventually in beellama). So you can control the batch/ubatch, whether mtp is on or off, and the quantization of the drafter model's kv cache. At the end of the day I found that you can either have medium context at q8 with faster TPS or larger context at q8 with slower TPS. BTW I dont have the ability to use iGPU since im on an older AMD cpu, so you might even get better results. I'm stuck with about 300-400mb used in VRAM because of my monitor. Heres the full run command: ./build/bin/llama-server --model "/home/models/Qwen3.8-27B-Q4_K_M.gguf" #unsloth --host 0.0.0.0 --port 8081 -b 256 -ub 256 -fitt 0 --cache-type-k q8_0 --cache-type-v q8_0 --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 --cache-ram 32768 --ctx-checkpoints 32 --threads 12 --threads-batch 12 -ngl 999 --no-mmap --mlock --reasoning-preserve --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 -np 1 --spec-type draft-mtp --spec-draft-n-max 2 #remove this for no MTP and larger ctx -lv 4 Prompt processing speed was consistent even at such a low batch size surprisingly. Without MTP: 194,048 context tokens ~1650 prompt processing tps ~40 tps gen With MTP: 131,584 context tokens ~1650 prompt processing tps ~ 65 tps gen (i sometimes saw 80tps when writing pure python code) So its a trade off. You can get 47% more q8 context at 38% less tps, or vice versa. I think if you manage the 131k context with smart sub-agent approaches you can definitely work with it. Will have to mess with it more. Also i found basically no difference in setting the draft kv quant to q4_0 vs q8_0, but i didn't test it too much. Hovered around 70% acceptance either way, so i deferred to using the one that has less VRAM requirements. Anyone else able to beat this on 24gb systems? Also if anyone has used beellama and thinks its worth moving over, let me know your experience! submitted by /u/sisyphus-cycle [link] [comments]

Source: r/LocalLLaMA | 2026-08-17

Related