After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)

Following up on my previous post about my budget server setup (Intel N100 + RTX 5060 Ti 16GB), a few of you asked for a deeper dive into my actual inference config and real-world agentic performance. Like many of you, I was refreshing the page waiting to download Qwen 3.8 27B the second it dropped. After spending the entire weekend stress-testing it with agentic coding workflows, I managed to run a complete, large-scale project almost entirely autonomously (over 1M total tokens processed, only 3 prompts total). Here is a quick breakdown of the core setup before we dive into the config and workflow details. Quick Specs & Params Model: Qwen3.8-27B-UD-Q3_K_XL.gguf Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM) Context Window: 73,728 (73k context) running comfortably in 16GB VRAM! KV Cache Quant: q4_1 for main context, q5_1 for MTP draft context Speculative Decoding: Native MTP enabled (spec-type = draft-mtp, n-max = 2) Sampling: temp = 0.4, top_p = 0.90, top_k = 15, min_p = 0.02 The Experiment: Building a full API in 3 Prompts Instead of running synthetic benchmarks, I put this setup through a real-world software engineering pipeline: building an unofficial REST API and MCP Server for a legacy vBulletin forum. Prompt 1 (Site Architecture & Analysis): Asked the model to map out the target site. It generated a flawless ~1,500-lines Markdown spec covering structural analysis, scrapable HTML nodes, expected JSON payloads, stack selection, pagination logic, session auth, and search endpoints—far more thorough than I would have written manually. Prompt 2 (Development Architecture): Using the spec as the single source of truth, it designed a modular NestJS API implementation plan broken into 9 execution phases: Phase 1: Project Scaffolding Phase 2: Domain Models Phase 3: Scraping Core (HTTP + Rate Limiting + Retries) Phase 4: HTML Parsers (cheerio) Phase 5: Cache Layer Phase 6: Application Services + REST API Phase 7: Authentication (Cookie Sessions) Phase 8: MCP Server (Primary Deliverable) Phase 9: Hardening, Docs, & Delivery Prompt 3 (Autonomous Agentic Execution): The real test. I instructed OpenCode (using Qwen 3.8 27B) to act strictly as an orchestrator, spawning sub-agents for each task phase. It ran autonomously for ~2 hours. When context limits were approached, OpenCode summarized its state and kept building. It wrote unit tests, enforced linting, and delivered fully functional code—only needing one minor automated fix when fed a edge-case raw HTML payload. The llama.cpp Configuration File Here is my exact --models-preset router configuration file. Note how fit = off is used on the 27B profile alongside ctx-size = 73728 (73k) and q4_1 KV cache quantization to maximize VRAM allocation while preserving native MTP performance. ini ============================================================================== LLAMA.CPP — INFERENCE CONFIGURATION (router mode / --models-preset) ============================================================================== Hardware Target: GPU: 16 GB VRAM (RTX 5060 Ti) CPU: Intel N100, 4C/4T (Debian Headless) ------------------------------------------------------------------------------ GLOBAL / BASELINE ------------------------------------------------------------------------------ [*] --- CPU THREADING ----------------------------------------------------------- Reserve 1 core for OS/services during decode. Use all 4 threads during prompt prefill bursts. threads = 3 threads-batch = 4 --- SERVER / CONCURRENCY --------------------------------------------------- Single slot, disabled continuous batching for maximum single-user throughput. parallel = 1 cont-batching = 0 --- GPU / VRAM FIT --------------------------------------------------------- flash-attn = on fit = on Safety headroom for VRAM physical limit (MiB). Set low (128) because system is headless (100% VRAM available for inference). NOTE: If using MTP draft KV caches, watch out for double VRAM allocation. Bump to 128-256 if you encounter OOMs. fit-target = 128 --- CONTEXT & CACHING ------------------------------------------------------ ctx-size = 65536 context-shift = 1 Disable context checkpoints (avoids reprocessing issues in hybrid architectures) ctx-checkpoints = 0 RAM Prompt Cache (2 GiB) cache-ram = 2048 --- GLOBAL KV CACHE -------------------------------------------------------- cache-type-k = q5_1 cache-type-v = q5_1 --- PREFILL / BATCHING ----------------------------------------------------- batch-size = 2048 ubatch-size = 1024 --- DEFAULT SAMPLING (Coding / Precision) ---------------------------------- temp = 0.2 top-p = 0.95 top-k = 20 min-p = 0.0 repeat-penalty = 1.0 presence-penalty = 0.1 frequency-penalty = 0.0 ------------------------------------------------------------------------------ QWEN 3.8 27B — REASONING & HEAVY CODING PROFILE ------------------------------------------------------------------------------ [qwen3.8-27b] model = /opt/llama-infrastructure/models/Qwen3.8-27B-UD-Q3_K_XL.gguf fit = off ctx-size = 73728 context-shift = 1 Native Model MTP (Speculative Decoding) spec-type = draft-mtp spec-draft-n-max = 2 spec-draft-p-min = 0.85 KV Quantization (q4_1 allows us to fit 73k context in 16GB VRAM) cache-type-k = q4_1 cache-type-v = q4_1 cache-type-k-draft = q5_1 cache-type-v-draft = q5_1 Thinking / Reasoning Budget Params chat-template-kwargs = {"preserve_thinking": true, "reasoning_effort":"medium"} reasoning-budget = 5000 Reduced batch sizes to prevent VRAM spikes during massive prefills batch-size = 1024 ubatch-size = 512 Official / Recommended Quant Sampler Tuning temp = 0.4 top-p = 0.90 top-k = 15 min-p = 0.02 submitted by /u/chiribe [link] [comments]

Source: r/LocalLLaMA | 2026-08-17

Related