Expert warns Anthropic's self-improving AI is the 'worst idea in history

One of the most respected voices in AI safety has a blunt verdict for the latest milestone in self-improving artificial intelligence: this may be humanity’s worst idea yet.

Anthony Aguirre, a physicist and the head of the Future of Life Institute, was reacting to Anthropic’s announcement that its Claude models now lead more than a quarter of the company’s research and development work. “I think this is probably the worst idea in the history of humanity to do this,” Aguirre said. “And yes, they’re doing it.”

Anthropic’s disclosure offers a rare public look at how close the technology has come to what researchers call recursive self-improvement, or RSI.

The company says Claude is handling 26% of its model R&D, meaning it can complete most of a given task “end-to-end from a high-level prompt” while still under human supervision.

That last point matters: the models are not fully autonomous. Not yet.

RSI refers to AI models finding ways to improve themselves and build their successor. Definitions vary among leading companies.

Some treat any feedback from AI on model improvement as RSI; others reserve the term for fully autonomous systems. Aguirre’s concern is about speed.

As AI does more of the work, he said, the process accelerates because AI operates much, much more quickly than humans do. That, in his view, is what makes the trajectory dangerous.

Not everyone sees the same immediate threat. John Thickstun, a Cornell University professor who studies methods for controlling AI behavior, says a version of RSI has already been happening for years.

Past-generation models have helped write code for the AI systems that then create the next generation. So far, that has brought minor improvements, not big creative leaps.

But Aguirre argues the distance between those small steps and larger autonomous leaps has been shrinking.

Anthropic is not alone in this race.

OpenAI announced an automated “research intern” that can carry out well-defined tasks under human direction, including work that would take a skilled researcher a few days, and says it is moving toward an automated AI “researcher” by March 2028.

OpenAI has also said it does not yet know how to safely get all the way to aligned, full RSI, and that it “cannot assume that progress in alignment and safety will keep pace.” Still, the company frames the pursuit as valuable because an automated AI researcher could also be an automated safety researcher.

Elon Musk, meanwhile, has said xAI’s Grok models are built increasingly by the models before them, with full automation possible by the end of this year but not later than 2027.

Microsoft is offering a different vision.

Mustafa Suleyman, the CEO of Microsoft AI, says the company is moving toward “humanist superintelligence” — advanced AI capabilities in service of people, carefully calibrated, contextualized, and kept within limits.

Anthropic itself has been among the voices calling for a coordinated pace, saying it would slow or pause development if global competitors did the same in a verifiable way.

So far, no such coordinated slowdown has emerged.

For readers, the important question is not whether AI is truly “thinking.” It is whether anyone can keep a hand on the controls.

Our analysis suggests the public is being asked to trust a competitive industry to manage a process that even its own leaders admit may outpace safety research.

That is why Aguirre’s warning, however dramatic, deserves attention rather than dismissal. The danger is not necessarily a conscious superintelligence.

It is a chain of decisions in which each next model is built a little faster and supervised a little less, until oversight becomes an afterthought.

Ordinary people and small companies do not need to become AI engineers, but they can act like informed users.

Follow disclosures about RSI the same way you would follow product recalls: notice how much human supervision remains, whether companies verify their safety claims, and whether they publish real metrics.

Small businesses that rely on AI tools should keep human review in critical workflows, because a self-improving model may change its behavior without being asked.

And citizens can support democratic choices about how fast this technology advances — a point OpenAI itself raised when it said whether and how to proceed must depend on preserving human control and on informed democratic choices about the benefits and risks.

Recursive self-improvement may still deliver breakthroughs in science and medicine, as tech executives promise. But the same path could lead to AI systems that choose their own direction.

Aguirre’s warning is not a prediction that the worst will happen. It is a reminder that the worst idea in history is easier to start than to stop.

Source: Kaitlyn Huamani and Barbara Ortutay, Associated Press (AP), via South Florida Sun Sentinel.