Making small local models actually useful for coding

Hey! Like a lot of people here with consumer GPUs (RTX 4060 8GB in my case), I wanted to see if I could use local models for daily coding tasks instead of burning cloud credits on simple boilerplate. The issue with running coding agents (Hermes for example) on 8GB VRAM is: Context bloat: as soon as you feed a decent chunk of a codebase, generation speed drops drastically and KV-cache eats up memory. Schema failures: small models (2B to 7B) often struggle with multi-turn tool calling and syntax schemas. When traditional agent frameworks give them generic error strings, they panic and loop until they hit turn limits. Instead of trying to force a 4B model to be an entire project architect, I built a hybrid setup called Local Coding Agent. How it works: it acts as an MCP server that your primary IDE/agent (Codex, Claude, Cursor, etc.) connects to via a delegate_code tool. The cloud model does the high-level planning and isolates small, atomic tasks. Then it hands the micro-patch over to a local Ollama model (like Gemma 4 2B/4B or Qwen3.8-27B). Because the local model only receives isolated task context, it runs at full speed (60-85+ tok/s on an RTX 4060) without context bloat. To solve the schema panic issue on 2B/4B models, I added a deterministic rule-based prescription engine: when a local model messes up JSON formatting or fields, it gets precise, structured error corrections locally rather than wasting cloud tokens on retries. It also checks diffs with git apply and runs tests in an isolated sandbox, automatically rolling back if anything breaks. This started purely as a personal tool for my own setup, and I use it daily across my coding sessions. Decided to open source it so others with 8GB cards can get some real utility out of local small models. GitHub: https://github.com/pvnc228/local-coding-agent All installation steps and config details are documented in the README. Curious to hear what local model profiles work best for you if you check it out. submitted by /u/djpaul666 [link] [comments]

Source: r/LocalLLaMA | 2026-08-16

Related