AIRA_2: Overcoming Bottlenecks in AI Research Agents

arXiv:2603.26499v2 Announce Type: replace Abstract: Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA_2, which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA^{agger}_{2} achieves a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, outperforming the strongest baseline, which achieves 72.7%. On AIRS-Bench, AIRA_2 exceeds human state-of-the-art on 6 out of 20 diverse research tasks. Ablations confirm that each architectural component is necessary, that performance follows a predictable scaling law that transfers across LLM backbones, and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

Related

• Lifetime-Aware Design for Item-Level Intelligence at the Extreme Edge
• Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior
• Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
• TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training
• Tree Training: Accelerating Agentic LLMs Training via Shared Prefix Reuse

Source: arXiv cs.AI | 2026-04-14