DeltaML-Bench: Evaluating Machine Learning Agents on Real-World Research Repositories

arXiv:2608.19653v1 Announce Type: cross Abstract: Autonomous agents for machine learning experimentation must navigate heterogeneous repositories, repair training pipelines, and evaluate candidate improvements under realistic compute constraints. Existing benchmarks only partially capture these conditions. We introduce DeltaML-Bench, a benchmark comprising 48 tasks sourced from research papers that require agents to improve published baselines within imperfect, open-source repositories. We evaluate GPT-5 and Claude Sonnet 4 with a standard Modular agent and a search-based ARG scaffolding. In the 4 x 6h allocation, ARG raises GPT-5's per-run success rate from 9.4% to 33.9%; in the 2 x 12h allocation, GPT-5 ARG reaches 49.0%. Modular configurations exhibit specification gaming rates as high as 47.9%, while no gaming is observed in the evaluated ARG configurations. These results indicate that scaffolding design and integrity checks are important considerations when deploying agents for autonomous ML experimentation.

Related

• From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level
• OmniCode: A Benchmark for Evaluating Software Engineering Agents
• LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis
• DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis

Source: arXiv cs.AI | 2026-08-21