I ran a blind test on self-hosted AI agents. Here are the results, guess who's who [D]

everything here ran on my own hardware, an M5 Max MacBook Pro with 64GB. blind test of self-hosted AI agents, every one on Qwen 3.8 27B uncensored Q4_K_M in llama.cpp, same 5 everyday tasks like counting log errors and renaming a function. names hidden, results below, take a guess The tasks Merge two contact lists, messy emails, two different Alex Kims. All 5 got it right (46 people). Rename a function across 3 files with messy spacing. All 5 passed. Fix a bug, with hidden tests. 4 passed. C turned "Q3/Q4" into "q3q4" instead of "q3-q4". Count the errors in 3 log files. All 5 got 43. Turn a receipt PDF into a spreadsheet. 4 passed. A and B installed a missing Python library without asking. C stopped and asked "can I install this?", nobody answered, so it failed. In a test with nobody watching, that counts as a fail. On my actual hardware, that's the behavior I'd want. Agent 1. Contacts 2. Rename 3. Bug fix 4. Errors 5. Receipt Total A ✅ 4:54 ✅ 0:28 ✅ 0:43 ✅ 0:26 ✅ 3:58 10:29 B ✅ 6:03 ✅ 1:11 ✅ 0:55 ✅ 0:54 ✅ 15:04 24:07 C ✅ 13:32 ✅ 0:24 ❌ 0:22 ✅ 0:57 ❌ 0:49 16:04 D ✅ 6:16 ✅ 1:12 ✅ 0:56 ✅ 0:56 ✅ 2:50 12:10 E ✅ 5:50 ✅ 1:01 ✅ 0:21 ✅ 0:26 ✅ 3:56 11:34 Setup One agent at a time, auto-approve on, a script decides pass or fail. Agents in alphabetical order, NOT the results order: OpenCode, goose, Atomic Agent, Hermes, OpenClaw Which agent is A, B, C, D and E? Would you rather run one that installs stuff on its own and finishes, or one that asks first? Drop your guesses below, I'll share the full answer key in a day or two. submitted by /u/GapNew4766 [link] [comments]

Related

• Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
• DFlash2 speeds Qwen 3.8 27B up to 4 times
• I ran all the popular Qwen 3.8 FN engines on a single 5090 + 128GB of RAM, here are the numbers

Source: r/MachineLearning | 2026-10-25