Anthropic urges a way to pause AI development as risks grow with the tech advances
Anthropic, the company behind the Claude chatbot, proposed a coordinated global pause in advanced AI development to mitigate risks of losing human control over rapidly advancing systems. The company warned that AI could soon achieve recursive self-improvement, potentially accelerating beyond human oversight, while also citing recent research on AI-driven cyber threats like self-spreading 'worms' as urgent concerns.
Anthropic, developer of the Claude AI chatbot, called for a mechanism to temporarily pause advanced AI development to address growing risks of human control loss. In a blog post on Thursday, the company highlighted rapid advancements in AI capabilities, including autonomous coding and potential recursive self-improvement—where an AI could design its own successors. While such progress could benefit fields like science and healthcare, Anthropic cautioned it might also increase risks of misalignment with human values. The proposal suggests a coordinated slowdown among top AI labs to enable societal structures and alignment research to keep pace with technological growth. Anthropic’s internal research institute plans to collaborate on building systems to verify compliance, ensuring no single actor exploits a pause to secretly advance. The company emphasized the need for global coordination to prevent less cautious competitors from gaining an advantage during a slowdown. Anthropic’s call follows recent research from the University of Toronto, which demonstrated how AI tools could create adaptive 'worms' capable of spreading across computing networks. Lead researcher Nicolas Papernot warned that security threats extend beyond the most powerful AI models, underscoring the urgency of addressing vulnerabilities. The company’s co-founder Jack Clark and research institute head Marina Favaro argued that a pause would allow time to develop safeguards aligning AI with human intentions. The proposal comes as Anthropic prepares for a potential trillion-dollar valuation through an upcoming IPO, competing with OpenAI, the creator of ChatGPT. Despite its own rapid growth, Anthropic stressed the need for collective action to manage AI risks, framing the pause as a precautionary measure rather than a rejection of progress. The company did not specify timelines or enforcement mechanisms but emphasized the necessity of global cooperation to prevent fragmented or ineffective responses.
This content was automatically generated and/or translated by AI. It may contain inaccuracies. Please refer to the original sources for verification.