Artificial intelligence firm Anthropic proposed a worldwide halt on the advancement of highly powerful AI systems due to concerns over potential loss of human control. The San Francisco-based company, known for its Claude family of AI models, emphasized in a report the need for a global slowdown in cutting-edge AI development to allow societal structures and alignment research to keep pace with technological advancements.
Anthropic suggested that for a pause to be effective, major AI companies in various countries, particularly in the US and China, would need to collectively agree to halt progress simultaneously under verifiable regulations. Without a coordinated global approach, the company warned that companies and governments would face challenges balancing safety measures with competitive and geopolitical pressures.
Although encountering resistance from industry peers and White House officials who question the severity of the risks raised by Anthropic, the company’s Mythos model has been acknowledged for its cybersecurity capabilities and restricted distribution to select organizations.
While the proposal may encounter opposition in Washington and Silicon Valley, where concerns about ceding technological dominance to China prevail, President Donald Trump mentioned exploring collaboration with China on AI safety during a recent visit to Beijing. Trump also signed an executive order allowing a preliminary review of the most potent US AI models before public release.
Anthropic likened the issue to nuclear arms control treaties but highlighted the difficulty of monitoring AI training compared to physical assets like missile silos, emphasizing the necessity of a collaborative framework. The company intends to convene government officials, scientists, advocacy groups, and rival AI companies to devise a feasible system for coordinated AI development.
In light of internal data indicating AI’s rapid self-improvement capabilities and the potential for recursive self-improvement, Anthropic cautioned about the diminishing human role in the AI development process. While acknowledging that complete autonomous improvement is not imminent, the company stressed the need for preparedness as AI continues to accelerate its own advancement.
