Running local LLM's as agents in Claude Code

I hit my token limit three times a day on my max subscription - got sick of that and designed this MCP setup to shift some of the coding load to my local Qwen3.8-27B model. I've been iterating on it now for a bit, and thought I'd share it now. This allows you to use Anthropics' models (Opus, etc) for the main model and shift the well-specified work down to local models all within the same session and context. Similar to how you can use sub-agents now in a project. https://github.com/ccebelenski/localagents No guarantees or support here - you'll need quite a bit of context size and caching memory to make this work well, and I've hardly tested the vllm path at all (llama.cpp seems solid). --metrics and --slots are just about required as parameters (might work without?). Have Claude help you set up the MCP itself. Then it's just "Use local agent for..." as a prompt. My launch for qwen38-27B looks like this for reference: llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL --host 0.0.0.0 --port 8080 --jinja --batch-size 8192 -fa on --main-gpu 1 --split-mode none --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --no-context-shift --presence-penalty 0.0 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 24 --spec-ngram-mod-n-max 86 --reasoning on --reasoning-preserve --cache-type-k q8_0 --cache-type-v q8_0 --fit on --cache-ram 16384 --parallel 2 --metrics --slots --chat-template-kwargs "{reasoning_effort":medium"}" submitted by /u/CodeSlave9000 [link] [comments]

Source: r/LocalLLaMA | 2026-08-27

Related