Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)

I wanted to share my successful setup for running a Qwen 3.8 27B model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed. 🧠 Key Components Model: Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller from jrell on Hugging Face. It's a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget. Chat Template: I used the Jinja template from peculiar-ragdoll's Qwen-Sharp-Chat-Templates. It helps use fewer thinking tokens without noticeably affecting quality, which is great for speed. Inference Engine: This is crucial. I used beellama.cpp (GitHub link) because it supports the kvarn KV cache types, which are key to this optimization. 🖥️ Optimized llama-server Command (Windows) Here's the polished command I'm running. The magic is in the kvarn cache settings and the tail precision. %LLAMA_DIR%/llama-server.exe ^ -m %MODEL_PATH% ^ -a %MODEL_NAME% ^ --port 11434 ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.0 ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ --parallel 1 ^ --n-gpu-layers 99 ^ --batch-size 1024 ^ --ubatch-size 256 ^ --flash-attn on ^ --spec-type draft-mtp ^ --spec-draft-n-max 2 ^ --cache-type-k kvarn5 ^ <-- Key: Higher precision for K cache --cache-type-v kvarn4 ^ <-- Key: Balanced precision for V cache --kv-tail-tokens 1024 ^ <-- Keeps recent tokens at full precision --ctx-size 100000 ^ --fit-ctx 100000 ^ --jinja ^ --chat-template-kwargs "{preserve_thinking": true, reasoning_effort":medium"}" ^ --chat-template-file %MODEL_JINJA% ^ --no-mmproj-offload ^ --threads 7 ^ --threads-batch 8 ^ --metrics ^ --verbosity 3 ^ --perf 📊 Results & Optimization Notes Metric Result Note Generation Speed 47-50 tokens/second Excellent for a 27B dense model. Context Window 100,000 tokens Successfully pushed from 88k by optimizing the cache. VRAM Usage ~15.93 GB (70 MB free) Perfectly tuned to the limit for maximum context. KV Cache Type kvarn5 (K) / kvarn4 (V) Uses the kvarn types from beellama. The asymmetric mix balances memory and quality. Precision Tail --kv-tail-tokens 1024 This is key. It keeps recent tokens at higher precision, preserving output quality. What I Optimized: KV Cache Quantization: Moving from kvarn5/kvarn5 to kvarn5/kvarn4 saved ~6% VRAM, allowing the context size to increase from 88k to 100k with minimal quality loss. Batch Sizes: Set --batch-size 1024 and --ubatch-size 256 to balance prompt processing speed and VRAM usage. Threading: Adjusted --threads 7 and --threads-batch 8 for my Ryzen 7 CPU. Speculative Decoding: Using --spec-type draft-mtp with 2 draft tokens (the model supports this) gave a huge speed boost. The near-lossless kvarn quantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible. Hopefully, this helps anyone trying to squeeze maximum performance out of a 16GB card! Cheers. submitted by /u/qaf23 [link] [comments]

Related

• After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
• Optimizing Qwen3.6 / Qwen3.8-27B on 16GB VRAM: Complete Benchmark Results and Setup Guide (~30-50tps at 32k to 72k context)
• I might have found the perfect config parameters for qwen 3.8 27b
• AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B

Source: r/LocalLLaMA | 2026-08-29