Model Releases
First local-LLM tuning attempt: Qwen3.8-27B true Q4_K_M at 13.2 tok/s near 50-61K context on RTX 5080 16GB
This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were
This was my first serious attempt at tuning a local LLM. I started because Qwen3.8-27B IQ3 was fast on my RTX 5080 but the coding quality disappointed me, and the Q4 profiles I tried in LM Studio were much slower than reports here. Hardware: RTX 5080 16 GB i5-14600K 64 GB DDR5-5600 (4 DIMMs) Windows Final model/runtime: Unsloth Qwen3.8-27B UD-Q4_K_M, unmodified (16.46 GB) official llama.cpp b10760 CUDA 13.3 build 65,536 context, one slot Q4_0 K/V cache, Flash Attention medium thinking, text only Pi as the coding agent Results: 49,738 input tokens: 13.247 / 13.260 / 13.261 tok/s across three runs 61,238 input tokens: 13.055 tok/s 4/4 retrieval in every run Pi read a broken implementation plus a separate test, edited only the implementation, ran PowerShell, and got PASS The useful change was selective FFN placement. I kept attention/KV and most tensors on the GPU, but moved the 16 largest FFN tensor groups (about 2.764 GiB) to CPU. Whole-layer offload in LM Studio gave me only 6.633 tok/s around 50K. MTP was surprisingly worse on this machine at deep context. MTP1 reached 8.654 tok/s and MTP3 7.810 tok/s, while disabling MTP reached 13.256 tok/s. My guess is that the CPU-side draft competed for RAM bandwidth with the spilled FFNs. I originally chased the recent ~75 tok/s 5080 post, but the linked 13.5 GB custom quant uses IQ3_S for its FFN tensors. That is a valid speed tradeoff, but I specifically wanted true Q4 weights and a deep-context measurement. I published the exact Windows launcher, tensor override, Pi config, benchmark harness, raw results, model SHA, failed profiles, and methodology here: https://github.com/johnconnor2020/qwen38-27b-rtx5080-16gb Caveats: the recall prompt is synthetic, the Pi task is a practical smoke test rather than LiveCodeBench/SWE-bench, and runs 2/3 reused prompt cache for ingestion (decode speed stayed the same). This is also likely sensitive to RAM bandwidth and llama.cpp version. I would be very interested in comparable true-Q4 50K+ results from other 16 GB cards, or suggestions for a better coding-quality benchmark that is practical to run locally. submitted by /u/nofuture09 [link] [comments]
Related
- club-5060ti refresh: tested RTX 5060 Ti presets, a proper high-context harness, and Qwen3.8 27B
- Optimizing Qwen3.6 / Qwen3.8-27B on 16GB VRAM: Complete Benchmark Results and Setup Guide (~30-50tps at 32k to 72k context)
- Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected
Source: r/LocalLLaMA | 2026-09-02