MiniCPM5-2B vs. Spark-X2.5-4B / -64% thinking, x1.5 speed while keeping the accuracy of xhigh

We gave two small LLMs a real customer-service task. One got 100/100. The other stopped with the job half-done. I wanted to test something more meaningful than benchmark scores: can a small model actually complete a multi-step agent task without handing the work back to the human? The task: A customer says: “My order is MB20260908017. The left earbud has no sound. I'd like a replacement. Could someone pick this one up tomorrow afternoon?” The agent has to: Look up the order Check the replacement policy Check nearby inventory Create the replacement Schedule the pickup Reply with the actual reference numbers The catch: the closest warehouse is Shanghai, but it has 0 stock. Suzhou is 95 km away with 12 units. Also, the system date is Sept 9, so “tomorrow afternoon” means Sept 10, not some vague future date. The backend actually validates everything and writes to a database, so the model can't just say it completed the task. MiniCPM5-2B It got the entire chain right: get_order → policy → inventory → replacement → pickup It correctly chose Suzhou (WH-SZ) and scheduled pickup for 2026-09-10, 14:00–18:00. The database confirmed both records existed. Score: 100/100 Time: 8.35s 527 completion tokens Spark-X2.5-4B It also correctly checked the order, policy and inventory, and correctly chose Suzhou. But then it stopped. Instead of scheduling the pickup, it asked the customer: “Could you let me know what time range you’d prefer?” The problem: “tomorrow afternoon” was already enough. The pickup function has an optional time range and defaults to 14:00–18:00. So the replacement was created, but the pickup wasn't. Score: 84/100 Time: 12.60s 1,460 completion tokens Why I found this interesting Both models understood the important business constraints. Neither picked the out-of-stock Shanghai warehouse. The difference was in the last mile of agency: Model A: “I have enough information → finish the workflow.” Model B: “I have enough information → ask the human to make another decision.” And that's exactly the kind of thing that doesn't show up clearly in a normal benchmark. The full setup also has two additional test cases: one where Shanghai inventory changes and one where the order is outside the replacement window, so the correct behavior is to stop rather than fabricate a replacement. Curious how others would evaluate this kind of end-to-end agent benchmark. Is task completion / database state more useful than another benchmark score? submitted by /u/Equivalent-Grass-527 [link] [comments]

Source: r/LocalLLaMA | 2026-09-16

Related