AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

arXiv:2602.02475v2 Announce Type: replace Abstract: AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an extit{automated diagnostic framework} that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.

Related

• SAFARI: Scaling Long Horizon Agentic Fault Attribution via Active Investigation
• ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
• LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures
• FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Search
• TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

Source: arXiv cs.AI | 2026-09-01