Technical Feedback: GLM-5.1 Retirement — Significant Regression in Agentic Workloads

Hi Ollama Team, I’m writing regarding the announced retirement of GLM-5.1 from Ollama Cloud on 25 September 2026. I wanted to provide some fairly detailed technical feedback because GLM-5.1 is not simply a model we happen to prefer. We have been benchmarking models for use within an agentic software-development and orchestration environment, and GLM-5.1 has demonstrated materially different runtime behaviour from GLM-5.3. In our testing, GLM-5.3 cannot currently be treated as a drop-in replacement for GLM-5.1. ENVIRONMENT / USE CASE Our workload is primarily agentic rather than conventional single-turn chat. The model is used as part of an orchestration system involving: - autonomous software-development tasks - structured task briefs - repository inspection - reasoning across existing project state - tool invocation - terminal operations - file creation/modification - validation and test execution - iterative correction - completion reporting - multi-step agent workflows - OpenClaw-style agent orchestration - model routing across multiple workers/models The key metric for us is therefore not simply subjective answer quality. We care about whether a model can reliably progress through: reasoning → planning → tool execution → observation → correction → validation → final task completion within the available execution envelope. BENCHMARK RESULT One of our controlled benchmark runs, internally identified as T02-R2, produced the following result: GLM-5.1 Median result: 26 / 31 GLM-5.3 Median result: 0 / 31 This is an extremely significant regression for our workload. Importantly, the GLM-5.3 result was not caused primarily by poor generated code, misunderstanding of the task, or an inability to reason about the problem. The dominant failure mode observed was execution-time exhaustion during the model's thinking/reasoning phase. In other words, GLM-5.3 frequently consumed so much of the available execution window before progressing sufficiently into actionable tool use that the agent failed to complete the benchmark objective. This resulted in a paradoxical behaviour: GLM-5.3 may possess greater theoretical reasoning capacity and a substantially larger context window, but under an agent runtime with finite execution budgets it can perform substantially worse because the reasoning phase prevents the model from reaching the execution/completion phase. CONTEXT WINDOW DIFFERENCE For reference, the models we evaluated expose approximately: GLM-5.1: 202,752 token context GLM-5.3: 1,048,576 token context We recognise that the approximately 1M-token context capability of GLM-5.3 is a major architectural improvement. However, for our agent workloads, context capacity is not currently the limiting factor. Execution efficiency is. A model with a 1M-token context window is of limited operational benefit if an autonomous worker frequently exhausts its execution budget before completing the tool-driven task. AGENTIC PERFORMANCE CHARACTERISTIC GLM-5.1 currently exhibits a behaviour which is particularly valuable for autonomous agents: Sufficient reasoning to understand the objective. Relatively rapid transition from reasoning to execution. Effective use of tools. Continued progression through the task. Validation of the resulting state. Production of a final completion response within the runtime envelope. GLM-5.3 appears considerably more reasoning-heavy. For interactive use this may be beneficial. For autonomous execution, however, excessive pre-execution reasoning becomes a systems-level problem. The issue is therefore not simply: "GLM-5.1 gives us better answers." It is closer to: "GLM-5.1 currently has a substantially better reasoning-to-action ratio for bounded autonomous agent execution." That distinction is important to us. MODEL EVALUATION PHILOSOPHY We are deliberately benchmarking models based on end-to-end task completion rather than evaluating them purely through conversational prompts. For an agent system, we consider successful completion to include the entire chain: prompt interpretation → planning → correct tool selection → tool execution → state observation → error recovery → continued execution → verification → terminal task state → final report A model that produces excellent reasoning but does not reach the terminal state is therefore scored as an unsuccessful agent run. This methodology is why the difference between GLM-5.1 and GLM-5.3 became so visible. OPERATIONAL CONSEQUENCE We are working towards a model-routing architecture where different models can be selected according to workload characteristics. GLM-5.1 was emerging as a particularly useful model for execution-oriented workers. GLM-5.3 may ultimately be better suited to: - very large context analysis - architectural reasoning - research - planning - complex synthesis while GLM-5.1 currently appears better suited to certain: - bounded coding tasks - autonomous execution loops - tool-heavy workflows - repository operations - iterative agent tasks Retiring GLM-5.1 therefore removes a behaviourally distinct model from the available model pool rather than simply replacing an older model with a universally superior version. REQUEST / SUGGESTIONS If technically and commercially feasible, we would strongly value one of the following: Continued availability of GLM-5.1 as a legacy cloud model. An extended deprecation period while GLM-5.3 agentic execution behaviour is improved. A GLM-5.3 "fast" / "agent" / reduced-reasoning profile designed specifically for tool-driven autonomous execution. Runtime parameters allowing developers to control the model's reasoning budget separately from: - output token budget - context window - tool execution budget - overall request timeout. A supported mechanism to cap internal reasoning duration/tokens before forcing transition towards action/tool invocation. Guidance from Ollama regarding the recommended GLM-5.3 inference configuration for latency-sensitive autonomous-agent workloads. For agent runtimes in particular, explicit control over the reasoning budget would be extremely valuable. An ideal control surface might conceptually expose parameters such as: reasoning_effort max_reasoning_tokens reasoning_timeout max_total_execution_time tool_bias / action_bias or an equivalent abstraction. That would allow the orchestration layer to deliberately choose between: deep analytical reasoning and fast execution-oriented reasoning depending upon the worker role. WHY WE ARE RAISING THIS We are not arguing that GLM-5.3 is an inferior model overall. It clearly has capabilities that GLM-5.1 does not, particularly its substantially expanded context window. Our concern is narrower and measurable: For the bounded autonomous-agent workload we have tested, GLM-5.1 currently completes tasks substantially more reliably than GLM-5.3. Our T02-R2 result — median 26/31 versus 0/31 — was sufficiently pronounced that we do not consider GLM-5.3 a safe transparent replacement for GLM-5.1 in our orchestration stack at this point. We would be very interested to know whether Ollama is seeing similar behaviour from other agent developers, and whether agent-runtime optimisation is part of the planned work around GLM-5.3. We would also be happy to provide additional information about the benchmark methodology, execution envelope and observed failure modes if that would be useful to your engineering team. Kind regards, uTuned! submitted by /u/uTuned [link] [comments]

Source: r/ollama | 2026-09-17

Related