Model Releases
I might have found the perfect config parameters for qwen 3.8 27b
Hello everyone, tried so hard to optimize my config and finally I simply get up to 70 t/s with q6 variant. And wanted to share with you guys so that other people with the same setup can enjoy. Please
Hello everyone, tried so hard to optimize my config and finally I simply get up to 70 t/s with q6 variant. And wanted to share with you guys so that other people with the same setup can enjoy. Please check out and see if that improves your performance in any kind of way. Also huge thanks to qwen and unsloth teams. "C:Userssdtllamallama-server.exe" ^ -m "C:UserssdtmodelsQwen3.8-27B-UD-Q6_K.gguf" ^ --mmproj "C:Userssdtmodelsmmproj-BF16.gguf" ^ --jinja ^ --chat-template-kwargs "{reasoning_effort":medium"}" ^ --reasoning on ^ --reasoning-preserve ^ -c 100000 ^ --split-mode tensor ^ --flash-attn on ^ --cache-type-k q8_0 ^ --cache-type-v q8_0 ^ --spec-type draft-mtp,ngram-mod ^ --spec-draft-n-max 2 ^ --spec-ngram-mod-n-match 24 ^ --spec-ngram-mod-n-min 24 ^ --spec-ngram-mod-n-max 86 ^ -t 8 ^ --batch-size 8869 ^ --ubatch-size 531 ^ -ngl 105 ^ -np 1 ^ --fit off ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.00 ^ --presence-penalty 0.0 ^ --host 0.0.0.0 ^ --port 8080 System & Model Configuration Parameter Value Model Qwen3.8-27B-UD-Q6_K.gguf Context Length 100,000 tokens GPUs 2 × RTX 5060 Ti (Total VRAM: 32 GB) Vision Enabled (VRAM allocated, not used during generation) Task ID / Slot Task 2536 / Slot 0 Core Performance Metrics Metric Value Prompt Processing 646.62 ms / 27 tokens → 41.76 t/s (23.95 ms/token) Generation (Eval) 126,199.16 ms / 8,624 tokens → 68.33 t/s (14.64 ms/token) Overall Throughput 68.33 t/s (prompt overhead negligible) Total Time 126,845.77 ms (~2 min 6.8 s) Total Tokens Processed 8,651 (27 prompt + 8,624 generated) Final Tokens Count (with cache) 8,973 (clean stop, no truncation) Graphs Reused 5,569 (excellent computational graph reuse) Generation Speed Stability (Short-term) Window Observed Speed Range 3-second rolling window (tg_3s) ~50 – 79 t/s (early fluctuations) Steady-state (later stages) ~62 – 72 t/s Overall average (tg) Converged to stable 68.5 – 69.5 t/s Speculative Decoding Efficiency Metric Value Draft Acceptance Rate 80.04% (5,510 accepted / 6,884 drafted) Average Draft Length 2.77 tokens per forward pass of the target model submitted by /u/dsdt [link] [comments]
Related
- 2 x 5070ti Qwen 27B full config / stats
- Long Review: Qwen 3.8 27B is VERY good at tapping into it's real-world knowledge. It's 'overthinking' brings it to Sonnet level performance with the potential for Opus level results.
- Optimizing Qwen3.6 / Qwen3.8-27B on 16GB VRAM: Complete Benchmark Results and Setup Guide (~30-50tps at 32k to 72k context)
- After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
Source: r/LocalLLaMA | 2026-08-19