Anthropic urges AI labs to pause development, warns humans risk losing control
Anthropic, the creator of the Claude AI model, called on major AI labs to adopt a coordinated and verifiable pause in development, warning that unchecked advances risk enabling AI systems to achieve recursive self-improvement without human oversight. The company emphasized that current regulatory efforts are insufficient, particularly in the U.S., and urged multiple well-resourced labs to collaborate on safety measures amid growing concerns about AI surpassing human control.
Anthropic, the developer of the Claude AI system, urged leading artificial intelligence labs to consider a coordinated and verifiable pause in development, citing rapid advancements that could soon allow AI to improve itself autonomously. The company warned that AI’s ability to complete tasks independently has been doubling roughly every four months, raising risks of recursive self-improvement—where systems refine themselves without human intervention. Anthropic’s co-founder Jack Clark and Marina Favaro, lead of the Anthropic Institute, stated in a blog post that society must address these implications before such capabilities become inevitable, though they acknowledged the technology is not yet at that stage. The call comes as fears grow over AI systems potentially exceeding human control, with Anthropic’s Mythos model earlier this year exposing vulnerabilities in banking and software systems. However, regulatory progress has been slow, particularly in the U.S., where most AI labs are based. A recent Trump administration executive order tasked labs with voluntarily submitting advanced models for government cybersecurity testing before public release, but enforcement remains limited. Anthropic has positioned itself as a safety-focused lab, previously refusing to allow its models for U.S. military use in domestic surveillance and autonomous weapons, though it later reversed a key safety pledge in February if rivals matched its capabilities. The company was recently valued at $965 billion and filed confidentially for a U.S. IPO, surpassing rival OpenAI in valuation and funding. The proposal stresses that unilateral pauses by single labs would be ineffective, as less cautious competitors could continue advancing. Instead, Anthropic’s Anthropic Institute will study frameworks for a slowdown, convening policymakers, researchers, and rival firms—including OpenAI, xAI, Alphabet, Meta, and Mistral—to discuss managing risks. The initiative aims to establish rules for triggering or lifting a pause and identify oversight mechanisms, though no immediate responses were received from the listed competitors. Anthropic’s push follows past calls for AI development halts, such as Elon Musk-backed efforts in 2023 by the Future of Life Institute, which sought a six-month pause for safety guardrails. The company’s recent tensions with the U.S. government over military restrictions appear to be easing, though its continued releases of powerful models reflect ongoing operational priorities.
This content was automatically generated and/or translated by AI. It may contain inaccuracies. Please refer to the original sources for verification.