Trump and Xi alone may have the chance to stop AI from spiralling out of control

After the Cuban Missile Crisis brought the world to the edge of nuclear war, global powers embarked on a concerted effort to pull it back from the brink.

The non-proliferation treaty of 1968, which limited the spread of nuclear weapons, has been a monumental success. Only a handful of countries today have access to 80-year-old technology, and those that do have not used it.

In the decades since, no technology has proved so dangerous as nuclear weapons as to require similar international co-ordination.

But those who fret about artificial superintelligence now believe a similar global effort is needed to prevent an AI-led disaster.

Anthropic, the world’s most valuable AI company, called for a mechanism to slow down or pause the development of advanced AI as it warned that the technology could get out of control more quickly than many think.

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the head of its research division said. They added that it would “likely be a good thing” if development could be delayed.

Anthropic – recently valued at $US965 billion ($1.37 trillion) – said it had raised the alarm because it believed AI was improving much faster than our ability to understand and control the systems.

Within the company itself, bots are not just writing code, they are also ordering around other bots and carrying out their own research.

Before long, they say, AI could be building itself, a process called recursive self-improvement. This could start a feedback loop in which progress goes parabolic.

Sceptics say this is mere marketing. Anthropic said last week it had filed for an initial public offering expected to value it at more than $US1 trillion.

What could be more valuable than a technology so powerful that world leaders need to rein it in? AI that builds itself has been a premise of the pitch decks the company uses to raise money for years.

David Sacks, Donald Trump’s former AI tsar and a frequent critic of Anthropic, suggested the warning was an attempt to secure a public bailout, saying it was a sign “you might be trying to get your frontier AI lab nationalised”.

But concerns about powerful AI are becoming increasingly prominent. Anthropic has kept its most powerful AI system, Mythos, out of public hands because of its ability to find security flaws in critically important computer systems.

Andrew Bailey, governor of the Bank of England, has raised the alarm about AI cra