Microsoft's Suleyman wants 'humanist superintelligence' to avoid runaway AI

As artificial intelligence labs move closer to systems that can build their own successors, Microsoft's AI chief is pushing a different path: not full-speed autonomy, but a "humanist superintelligence" designed to serve people within clear limits.

The phrase offers a possible answer to the industry's deepest worry — that recursive self-improvement could escape human control.

Recursive self-improvement, or RSI, describes AI that finds ways to improve itself and create the next version of itself. Some leading developers say artificial intelligence is approaching that point.

Anthropic revealed this week that Claude, its model, now carries out 26% of the company's model research and development.

Anthropic says Claude can finish most tasks "end-to-end from a high-level prompt," but still works under human supervision. Autonomous RSI, with no human in the loop, remains in the future.

That future is what worries critics.

Anthony Aguirre, president and CEO of the Future of Life Institute, warns that the process could speed up beyond human ability to react. "As AI is doing more of it, it gets faster, because AI operates just much, much more quickly than the humans do," he says.

He added that the current push is "probably the worst idea in the history of humanity."

Others see the threat differently.

John Thickstun, assistant professor of computer science at Cornell University, says a softer version of RSI has been happening for years: developers use past-generation models to write code for the systems that produce the next generation.

That method has produced useful but modest gains, not sudden leaps. Yet Aguirre believes companies are now far closer to big, fast jumps.

Some labs are openly chasing those jumps. OpenAI has introduced an automated "research intern" for well-defined tasks and set a target of an automated AI "researcher" by March 2028.

It also said it does not yet know how to "safely get all the way to aligned, full RSI" and "cannot assume that progress in alignment and safety will keep pace." Elon Musk has said that for xAI's Grok models, "humans are gradually getting less and less in the loop" and "every successive model is built by the one before it." He added that full automation could arrive by the end of this year, but not later than 2027.

Microsoft is using different language. Mustafa Suleyman, CEO of Microsoft AI, has said the company wants "humanist superintelligence" — advanced AI in service of people and humanity.

In a 2025 essay, he described it not as "an unbounded and unlimited entity with high degrees of autonomy," but as AI that is "carefully calibrated, contextualized, within limits."

What does that mean in practice? Our analysis suggests it is more of a philosophical stance than a proven safety system.

Suleyman's wording gives a clear target: superintelligence should remain tied to human purposes.

But it is not yet clear from public statements how any lab, including Microsoft, would keep a future system "within limits" once it operates faster than humans can monitor.

The gap between promise and mechanics is where the real risk sits.

The industry's division over this gap is already visible. Anthropic says it would slow or pause development if global competitors did the same in a verifiable way.

OpenAI argues the next step must depend on preserving human control and informed democratic choices.

Suleyman's humanist framing can be seen as another way to draw that line — by insisting that even a superintelligent system should be an assistant, not an independent ruler.

For ordinary people and small companies, the practical move is to look at how AI tools are governed, not just what they can do.

Before relying on a model for important work, ask whether a human can review and correct its output, whether the system's decisions can be traced, and what happens after an update changes behavior.

Keep a manual fallback for tasks where mistakes are costly. Treat any lab's safety statement as a starting point, not a guarantee.

The direction of the next few years is not sealed. Suleyman's "humanist superintelligence" may become a meaningful counterweight to runaway RSI, or it may remain a slogan.

What will decide that outcome is whether the companies promising limits can actually prove them.

Source: Reporting by Kaitlyn Huamani and Barbara Ortutay, The Associated Press. This article also draws from public statements by Anthropic, OpenAI, xAI and Microsoft.