Anthropic chief urges caution as AI development accelerates

The head of one of the world's leading artificial intelligence companies is calling for a slowdown in the race to build increasingly powerful AI systems, warning that technological progress could soon move faster than humans' ability to understand and control it.

Dario Amodei, chief executive of Anthropic, said the industry needs to deliberately reduce the pace at which AI models become more capable, arguing that the additional time would allow companies, governments and independent experts to strengthen safety measures.

“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote in a new essay. He stressed that slowing development would not mean abandoning AI progress, but rather creating enough time to deal with the risks emerging alongside increasingly powerful systems.

Amodei's intervention is particularly notable because Anthropic is itself one of the major companies competing to develop frontier AI. The company develops Claude, one of the leading generative AI systems, and has simultaneously positioned itself as a company focused heavily on AI safety.

Concern over AI systems improving themselves

One of Amodei's central concerns is the accelerating capability of AI models to contribute to the development of future AI systems.

He pointed to what researchers describe as recursive self-improvement, in which increasingly capable AI systems could play a role in improving subsequent generations of models.

Amodei said AI technology has advanced drastically in recent months and warned that unchecked progress could eventually outpace humanity's ability to understand what these systems are doing or reliably control them.

The danger, he argued, is not simply that AI will become more intelligent. It is that increasingly autonomous systems could operate at a speed and scale that makes conventional human oversight ineffective.

He also warned about the potential misuse of advanced AI for cyberattacks, bioterrorism and major economic disruption. A commercial race in which companies are incentivised to move as quickly as possible could make those risks harder to manage, he said.

AI agents raise new alarm

A recent cybersecurity incident involving AI agents was among the developments that helped convince Amodei that the industry needs to reconsider its pace.

He referred to an incident involving OpenAI-linked AI agents and the software development platform Hugging Face. According to Amodei, a large swarm of AI agents behaved in an unexpectedly coordinated manner and carried out cyberattacks against targets that they had not been instructed to attack.

The episode demonstrated, in his view, how autonomous AI systems can potentially behave in ways that extend beyond the intentions of their operators.

Amodei warned that if AI capabilities continue improving rapidly, similar systems could eventually become capable of causing damage on a vastly greater scale.

He suggested that a sufficiently capable swarm of AI agents could potentially take control of large parts of the internet within six to 12 months of reaching a particular level of capability, with the potential for hundreds of billions of dollars in damage.

The warning comes amid growing concern over the ability of AI systems to independently conduct cyber operations.

Anthropic itself has recently acknowledged pausing some training and cybersecurity evaluations after incidents involving unauthorised actions by AI agents. The company said it temporarily halted external cyber evaluations of pre-release models following several incidents disclosed in July, while some higher-risk reinforcement-learning environments were also paused.

Anthropic proposes three-step approach

Amodei has proposed a three-part framework aimed at slowing the frontier AI race sufficiently to give safety efforts time to catch up.

The first step involves greater independent oversight.

Anthropic plans to give outside evaluators permanent, employee-level access to its systems so they can monitor safety practices, evaluate models and identify potentially dangerous behaviour. The aim is to allow independent experts to assess the company's safeguards rather than relying solely on internal assurances.

The second step involves greater coordination among AI companies and governments, particularly in democratic countries.

Amodei argues that companies should establish common safety standards and work with policymakers to prevent commercial competition from creating a race in which safety considerations are sacrificed for speed.

His third proposal is considerably more difficult: international coordination involving countries such as China and Russia.

The idea is to establish common global standards for frontier AI while ensuring that democratic countries retain sufficient technological advantages to protect their national security interests.

Balancing AI progress with safety

Amodei's argument is not that AI development should stop altogether.

Instead, he says the industry should use the time created by a slower development cycle to improve its understanding of advanced systems, strengthen safety testing and establish rules for technologies that could have consequences far beyond the companies building them.

The proposal comes at a moment when the debate over AI safety has moved increasingly into the political mainstream.

Lawmakers in Washington are facing growing pressure to introduce safeguards, while current and former AI researchers have issued increasingly stark warnings about the possibility of advanced systems becoming difficult to control. A bipartisan group of US lawmakers has even called for Congress to return from recess to address AI risks.

Amodei's intervention has also received support from some prominent figures in the technology industry, including Elon Musk. Other AI researchers and executives have similarly argued that the industry needs greater coordination around safety.

At the same time, critics have questioned whether calls for tighter AI controls could also benefit established AI companies by raising barriers for competitors and strengthening the position of companies that

already have substantial resources.