Anthropic calls for plan to pause AI development if society struggles to manage the technology
Anthropic, an AI company, proposed a coordinated global plan to pause or slow AI development if risks from self-improving systems outpace societal controls, emphasizing the need for multi-lab agreements and oversight. The company warned unilateral pauses could fail if other labs continue advancing, and plans to convene discussions with policymakers, researchers, and other AI firms to address risks like recursive self-improvement.
Anthropic, a leading AI developer, has called for a verifiable mechanism to slow or temporarily pause advanced AI development if systems begin improving themselves faster than society can manage the risks. The company warned that full recursive self-improvement could increase the danger of humans losing control over AI systems, making security, monitoring, and behavioral shaping more critical. As of May, over 80% of the code merged into Anthropic’s codebase was generated by its own AI system, Claude, highlighting rapid self-improvement. Anthropic argued that a global pause option would allow societal structures and alignment research to keep pace with technological advancements, but cautioned that poorly coordinated slowdowns could backfire if less cautious competitors continue advancing. A meaningful pause would require agreement among multiple well-resourced AI labs, clear trigger conditions, and oversight mechanisms, the company stated. While a single company could unilaterally pause development, it would have limited impact, primarily shifting leadership rather than fostering broader global coordination. Anthropic’s research arm, the Anthropic Institute, plans to study and develop systems to support such a slowdown. In the coming months, the company will host discussions with policymakers, researchers, civil society groups, and other AI firms to address key risks, including recursive self-improvement and coordination mechanisms. Last month, Anthropic concluded a funding round valuing the company at $965 billion and confidentially filed for a U.S. initial public offering. The proposal reflects growing concerns about AI’s rapid advancement and the need for structured global governance.
This content was automatically generated and/or translated by AI. It may contain inaccuracies. Please refer to the original sources for verification.