I ran the 35B agentic comparison someone asked for (stock vs Ornith vs KAT-Coder, 120 runs)

Someone in the comments of my 27B post-train bakeoff asked for the 35B version, so I ran it. Same setup as last time: fresh Coder workspaces on my k8s cluster, each driving my own agent (Hermes) headlessly, models on llama.cpp via llama-swap on one 5090, every call traced through an OTel shim into SigNoz, full transcript per run. 4 models, 6 self-grading tasks, 5 reps, 120 runs, MTP on every arm, identical sampling, hypotheses pre-registered. KAT-Coder-V2.5-Dev matched the best stock pass rate (29/30, tied with Qwen3.5-35B) at half the input tokens of either stock and the cleanest tool behavior I've measured (zero malformed tool-call leaks in 30 runs; stock Qwen3.6 leaked 195 on one task). All six analysts (three model families) independently called its efficiency discipline rather than corner-cutting: baseline tests before edits, one targeted patch per bug, deliverables at the right path every rep. Ornith went 25/30, losing to its own base. Its failures were mechanics, not knowledge: format leaks killing runs at 23 seconds, whole-file rewrites corrupting unrelated files, and one research run that invented a llama.cpp release tag while its own reasoning said "I mentioned v4659 in my draft which is fabricated," then shipped the tag anyway. The grader passed it. Stock 3.6 is the strongest raw analyst and the biggest token waster; stock 3.5 is the quiet reliable one. Full writeup with methodology, tables, and all six cited per-task analyses: https://kmarble.dev/posts/35b-coder-bakeoff/. Transcripts were AI-analyst-read with my spot-verification of every consequential claim. submitted by /u/IvGranite [link] [comments]

Source: r/LocalLLaMA | 2026-07-27