Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. It comes from a paper posted to arXiv on 12 October 2026. Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.

Why it counts

We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems.

Sources

Primary source: primary source

What is not known

This brief does not claim independent replication. Claims that appear only on X and not in the primary source stay unknown.