New benchmark on LMs fixing bugs before users run into them

Hi! This is a new benchmark that I created together with other researchers at Meta, Stanford, Harvard, UW. Most benchmarks these days seem to test models to just fix a bug that I as a user already encountered. But shouldn't we expect models by now to also find bugs before anyone runs into them? So in SWE-sweep we just hand an agent a big codebase and ask it to find & fix as many bugs as it can. We then give a score based on a hidden set of bugs that we know about in the repos. All the bugs are real-world bugs. We do a lot of filtering to make sure the bugs are actually discoverable & fixable from reading the repo alone. https://preview.redd.it/irffy7x5v2th1.png?width=1080&format=png&auto=webp&s=847a08f1a255fe39addd004cbb2a65561b1cf2a6 We're still expanding the leaderboard list with more local models (unfortunately it's always a big harder with funding/infra etc), but right now it seems like it's quite hard to beat Luna xhigh in terms of cost efficiency. Also the scores are way lower than I would've expected. Some tasks are legitimately superhuman in practice (like fixing up all of numpy), but there's also lots of small repos, where I would've expected a lot more from current models. Everything is open source (MIT license) on github and you can find paper etc. on the website. Happy to answer questions here, also super curious what open weights models you'd recommend running next (we're working on an update next week). submitted by /u/klieret [link] [comments]

Related

• Ling-3.0-flash is another potential model to test before qwen3.8 27b
• What’s the community’s favorite benchmark to validate performance?
• A collection of small domain-specific benchmarks for local models (30+ and growing)
• Putting the question before the context took my local Qwen from 89% to 100% on a decision benchmark, and from ~400 ms to ~80 ms

Source: r/LocalLLaMA | 2026-10-02