Anthropic warns AI could soon start improving itself. Critics aren’t convinced

Anthropic, the creator of the Claude chatbot, has issued a warning that AI systems could soon reach a stage called recursive self-improvement. This stage would allow AI to autonomously design and build more advanced versions of itself with limited human intervention, increasing the risk of humans losing control over the technology. The company argues in a June 4 blog post titled *When AI Builds Itself* that society should prepare for the possibility of slowing or temporarily pausing AI development. Anthropic highlights its own operations as evidence of accelerating change, noting that Claude now writes over 80% of the code merged into its systems, up from low single digits before the 2025 release of Claude Code. The company’s engineers also ship eight times more code per quarter than they did a few years ago, signaling a shrinking human role in AI advancement. Anthropic proposes establishing a global coordination mechanism to enforce such a pause, drawing a loose comparison to arms-control agreements on intermediate-range nuclear missiles. However, critics like Noah Giansiracusa, an associate professor of mathematics at Bentley University, dismiss the idea as unrealistic. Giansiracusa argues that no treaty or competition pressure would compel major players, including China or figures like Elon Musk, to slow down development. The proposal follows a pattern that has raised skepticism among researchers. Two months prior, Anthropic unveiled Mythos, an AI model that sparked concerns among experts. Anthropic did not respond to questions about how a potential pause would function or address criticism that it may be overstating its systems’ capabilities. Giansiracusa further argues that a coordinated slowdown is impossible, stating there is ‘zero chance’ of industry-wide cooperation. The debate underscores the challenges of AI governance, where rapid technological advancement outpaces societal and regulatory frameworks.