Model Releases
Finally got qwen 3.8 q4 k_M 27b runnin slick 40tkps load at 240k inf 4qnl for stable agentic coding on a 7900xtx. I have been finally able to give hermes a task and come back to results. Built a panel to manage my inference servers from hermes.
Screenshot of heavy load. I did offload vision to a spare 2060 instead of local ram. The control panel was generated with qwen3.8-27b davidau q4k_m max mtp. Made the panel from scratch over a week, ba
Screenshot of heavy load. I did offload vision to a spare 2060 instead of local ram. The control panel was generated with qwen3.8-27b davidau q4k_m max mtp. Made the panel from scratch over a week, basically just building for speed. Results int he picture are from a live deep coding job. old pc intel 9700, 32gb ddr2400, 7900xtx, 2060, 850 watt psu, ubuntu, llama.cpp, hermes on a rasp pi. Speed is more or less the same with a 2060 running the vision mmproj instead of local cpu offload so I set up a 2nd instance to run from that card. Prefill 900ish hard decode 40ish /home/admin/llama/b11011-vulkan/llama-b11011/llama-server -m /home/admin/models/Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-Q4_K_M.gguf --device Vulkan0,Vulkan1 --split-mode layer --tensor-split 1,0 --mmproj /home/admin/models/mmproj-F16.gguf --image-min-tokens 1024 --mmproj-device Vulkan1 -c 240128 -np 1 --n-gpu-layers 99 --flash-attn on --cache-type-k iq4_nl --cache-type-v iq4_nl --cache-ram 5631 --cache-reuse 256 --jinja --chat-template-file /dev/shm/llama_qwen38/qwen3.8-safe-v2.jinja --chat-template-kwargs {"reasoning_effort":"xhigh","preserve_thinking":true,"max_tool_response_chars":9000} --reasoning-preserve --reasoning on --reasoning-format deepseek --reasoning-budget -1 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0 --temp .77 --top-p 0.95 --top-k 20 --min-p 0.0 --metrics --no-warmup --host 0.0.0.0 --port 8081 --slot-save-path /dev/shm/llama_qwen38/slots --log-file /dev/shm/llama_qwen38/telemetry/engine_debug.log --batch-size 2048 --ubatch-size 512 --threads 6 --threads-batch 6 --n-predict 32000 --cont-batching /home/admin/llama/b11011-vulkan/llama-b11011/llama-server -m /home/admin/models/Qwen3.8-27B-TurboFCFusion-735-882-Here-Uncen-NEO-CODER-MAX-MTP-Q4_K_M.gguf --device Vulkan0,Vulkan1 --split-mode layer --tensor-split 1,0 --mmproj /home/admin/models/mmproj-F16.gguf --image-min-tokens 1024 --mmproj-device Vulkan1 -c 240128 -np 1 --n-gpu-layers 99 --flash-attn on --cache-type-k iq4_nl --cache-type-v iq4_nl --cache-ram 5631 --cache-reuse 256 --jinja --chat-template-file /dev/shm/llama_qwen38/qwen3.8-safe-v2.jinja --chat-template-kwargs {"reasoning_effort":"xhigh","preserve_thinking":true,"max_tool_response_chars":9000} --reasoning-preserve --reasoning on --reasoning-format deepseek --reasoning-budget -1 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0 --temp .77 --top-p 0.95 --top-k 20 --min-p 0.0 --metrics --no-warmup --host 0.0.0.0 --port 8081 --slot-save-path /dev/shm/llama_qwen38/slots --log-file /dev/shm/llama_qwen38/telemetry/engine_debug.log --batch-size 2048 --ubatch-size 512 --threads 6 --threads-batch 6 --n-predict 32000 --cont-batching submitted by /u/W61k3r [link] [comments]
Related
- Local agentic coding Benchmark : Qwen 3.8 27B (in many weights quants / cache quants / engine / reasoning effort) vs others.
- I made a custom llama.cpp build optimized for 7900xtx (one or two). for qwen 3.8 next and 27B. includes optimizations for PciE x4 and tensor parallel. read inside! (no AI slop)
- After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
Source: r/LocalLLaMA | 2026-09-19