Anthropic urges global freeze on AI as it warns of losing control

The world’s most valuable artificial intelligence company has called for a global freeze on the development of the most powerful AI systems, warning that humanity is approaching a point where it could lose control of the technology it is racing to build.

In a blog post published on Thursday (Friday AEST), Anthropic, the San Francisco company behind the Claude chatbot, said it would be willing to suspend work on more capable systems if it could be confident that its rivals would do the same.

“If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing,” executives wrote.

Anthropic has every commercial reason to keep building. It recently overtook ChatGPT maker OpenAI to become the world’s most valuable AI firm, at a valuation of about $US965 billion ($1.3 trillion). It has also withheld its most powerful system, known as Mythos, from public release over fears it could be used to carry out devastating cyberattacks, and it said this week it would extend access to Mythos to Australian organisations for the first time, as part of a staggered rollout meant to fix security flaws before the product reaches hackers.

The Albanese government welcomed the intervention. Andrew Charlton, the assistant minister for science and technology, who is leading Labor’s AI strategy, said the pace of progress “must not outstrip understanding or undermine safety across society, and no company should develop AI that is unsafe”.

“The government welcomes calls for a more considered approach to frontier AI development,” Charlton said. “Labor believes technology should work for people and their benefit.”

Anthropic’s argument turns on a concept it calls “recursive self-improvement”: the point at which AI systems become capable of designing and building their own successors with little human input. Anthropic head of research Marina Favaro and its president Jack Clark said the industry was edging towards that threshold, and that crossing it could trigger an explosion in capabilities that outpaces society’s ability to manage the consequences.

To make the case, the company published internal data on how quickly Claude is improving, and it said more than 80 per cent of new code had been written by Claude rather than by humans. In one instance, the company said Claude had resolved more than 800 software bugs in April that it estimated would have taken a human engineer four years to fix.

Favaro and Clark called the situation an “arms control problem”. They said