Model Releases
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
arXiv:2607.09996v2 Announce Type: replace Abstract: Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler
arXiv:2607.09996v2 Announce Type: replace Abstract: Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.
Related
- When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation
- Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systems
- Localizing Emergent Failures in Agentic AI: Recovering Minimal Repair Families via Counterfactual Replay
- One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution
Source: arXiv cs.AI | 2026-10-12