Research
Can LLMs improve their own harness? Self-improving harnesses are a hot topic. And one interesting question is whether LLMs can generate and …
Can LLMs improve their own harness? Self-improving harnesses are a hot topic. And one interesting question is whether LLMs can generate and improve them on their own. They find that two full rewrites
Can LLMs improve their own harness? Self-improving harnesses are a hot topic. And one interesting question is whether LLMs can generate and improve them on their own. They find that two full rewrites of the harness raise held-out scores by only 2.80 points, and the 95% interval crosses zero. Each frontier model writes a complete harness from a fixed coding bootstrap, plays five development games through it, and then rewrites its whole codebase twice. Every version is scored on 20 games that never influence the rewrites. Across 27 lineages, first harnesses range from zero to near-ceiling. The revisions add very little. Gains on the development games overstate the held-out gains, and revisions that cut inference cost often lower the score. Human-steered systems still score higher on the same games, so better designs exist. The models just do not find them reliably on their own yet. https://arxiv.org/abs/2610.30499 Chat with Paper: https://academy.dair.ai/papers/can-llms-improve-their-own-harness-a-controlled-study-on-arc-agi-3-2610.30499
Related
- Pay close attention to custom harnesses. This is a super interesting paper showing the impact of training inside the harness. Qwen3-14B goes…
- The Top AI Papers of the Week (May 4 - 10) - Conductor - HeavySkill - Horizon Generalization - 1,000 Synthetic Computers - Self-Improving Pr…
- Banger paper from CMU on harness learning. (bookmark it) Also, pay attention to this important new AI engineering skill of improving agents …
- Recommended work from Meta on harnesses for proactive agents. The ProPerAgent harness raises accuracy from at most 15% with standard memory …
Source: DAIR.AI (X) | 2026-10-24