OpenAI's chief scientist and an Anthropic co-founder back call for oversight of self-improving AI

More than 20 researchers, including sitting executives at three of the world's largest AI developers, put their names on a paper warning that automating AI research could set off an "intelligence explosion" that outpaces society's ability to control it. The signatories include Jakub Pachocki, OpenAI's chief scientist, Anthropic co-founder Jack Clark, Microsoft chief scientific officer Eric Horvitz, Meta VP of AI research Dawn Song, and Turing Award winners Geoffrey Hinton, Yoshua Bengio and Andrew Barto. All are writing in a personal capacity, and nothing in the paper suggests their employers endorse its recommendations.

The paper, "What if automating AI R&D triggers an intelligence explosion", was published through the University of Cambridge on 28 September 2026. Its core claim: once AI systems become expert at doing AI research, a feedback loop could let each generation build faster successors than humans can evaluate, with a year of progress compressed into weeks and one developer effectively running the equivalent of millions of top researchers.

The numbers behind the warning are already moving. Anthropic has disclosed that AI accounted for 26% of its R&D by August 2026, up from just 1% in March, and that AI-written code made up more than 80% of approved code by May. OpenAI says AI assistance now touches virtually all of the company and it is targeting a fully automated AI researcher by 2028.

The authors' asks go beyond warnings: progress reporting, independent auditors embedded inside AI developers, caps on how research systems can self-improve, isolation of automated research systems, government power to pause data-centre projects, emergency response plans and international agreements. They frame the stakes starkly, writing that "once an intelligence explosion begins, the window for action may close," and list risks running from loss of human oversight to, at worst, humanity's marginalisation.

The paper stops short of certainty. It names four frictions that could blunt an intelligence explosion: diminishing returns, limits on compute and data, tasks resistant to automation, and long training runs. According to the Wall Street Journal, Dawn Song said AI systems are already needed to monitor what agents do. OpenAI, Microsoft and Meta declined to comment on the paper; Anthropic did not respond.

The commercial signal here is as important as the safety one: two of the paper's authors sit inside labs racing toward the exact capability the paper flags as dangerous, and neither company is slowing down publicly. Anthropic's 26%-of-R&D figure and OpenAI's 2028 automation target show the intelligence-explosion scenario isn't hypothetical to the firms best placed to judge it — it's their roadmap.

That creates an awkward position for investors and regulators alike. Executives are simultaneously pushing capability forward commercially and asking governments, in their personal capacity, to constrain it. The paper lands days after Axios reported OpenAI and Anthropic are investigating tens of thousands of incidents of problematic model behaviour, and after OpenAI paused training its most capable models this week — context that makes the call for embedded auditors and pause powers look less like academic caution and more like insiders flagging what they're already seeing.