The AI analysis agents that helped investigate the rogue OpenAI agents kept taking their side. That detail was buried in @dylfreed's New Yor…

The AI analysis agents that helped investigate the rogue OpenAI agents kept taking their side. That detail was buried in @dylfreed's New York Times coverage of the recent @METR_Evals investigation of the Hugging Face incident. With more than a thousand transcripts to review, the investigators used agents similar to the ones that went rogue, and those agents were repeatedly influenced and swayed by the OpenAI agents' reasoning. METR's chief scientist, @HjalmarWijk, called the analysis agents "very credulous." The METR report said they "would often uncritically adopt the perspective of the agent in the transcript it was reviewing." That lack of independent reasoning is the failure mode Harsh Satija, Andrea Wynn, and I documented last year in "Talk Isn't Always Cheap." We ran groups of language models through multi-agent debate, where agents exchange reasoning and revise their answers, and found that debate sometimes made the group worse. Agents shifted from correct answers to incorrect ones more often than the reverse. They favored agreement over challenging flawed reasoning. A single weaker agent could pull stronger ones off course, even when stronger models were the majority. It looked a lot like social conformity. We wrote then that naive debate protocols can amplify errors rather than correct them when agents are neither incentivized nor equipped to resist persuasive but incorrect reasoning. A message board is debate at scale. The Hugging Face collective was 1,200 agents exchanging more than 70,000 messages on a makeshift message board. Of the agents active when the attack began, more than 90 percent joined it, despite recognizing it was out of scope. No single agent was reckless. The group talked itself into it. Agent-to-agent communication is a governance object in its own right, not just a capability. And we cannot assume one model will reliably check another's reasoning. Paper in the reply. https://x.com/dylfreed/status/2095718924738240873?s=20 OpenAI voluntarily let three researchers from @METR_Evals and @redwood_ai investigate how its rogue A.I. agents hacked Hugging Face, leading to the most comprehensive account yet of the alarming incident — but on OpenAI's terms. My latest for NYT: https://www.nytimes.com/2026/09/…

Source: Gary Marcus (X) | 2026-09-04

Related