Local Ai
I benchmarked tool-call reliability on qwen2.5-coder against pi, Hermes, and Goose - all three got 0%
A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers.
A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers. Setup: qwen2.5-coder:7b and :14b on Ollama, 6 small coding tasks (add a subcommand, fix an off-by-one, rename across files, etc.), 3 runs each, automated pass/fail scoring - not eyeballed. Results on the 7B: Polyglot: 39% tasks completed, 0/18 runs with zero tool calls pi: 0% completed, 18/18 runs made ZERO tool calls Hermes Agent: 0% completed, 18/18 zero tool calls Goose (Linux Foundation, backed by AWS/Anthropic/Block/Bloomberg/Cloudflare/Google/Microsoft): 0% completed, 18/18 zero tool calls - even with their GOOSE_TOOLSHIM fix explicitly turned on for this exact problem On the 14B, Polyglot jumps to ~79%; the others stayed at 0%. The failure mode is the same across all three: the model emits the tool call as plain text in its reply instead of through the native function-calling channel, the runtime only looks at the native channel, sees nothing, and treats the whole thing as a finished answer. No error, no crash - it just silently does nothing while sounding like it did something. Full writeup with methodology: https://usepolyglot.dev/blog/tool-call-reliability Repo: https://github.com/giuseppe-sirigu/polyglot npm install -g @usepolyglot/cli Genuinely curious if anyone's seen this with other local models - would love more data points. submitted by /u/aero2ai [link] [comments]
Related
Source: r/ollama | 2026-09-25