Anthropic Warns AI Agents Could Near Self-Improvement Without Human Input

Anthropic said AI development is accelerating toward a point where software “agents” could build, train, and improve themselves with little or no human involvement, and argued that slowing frontier AI development would be ideal to address the technology’s implications. According to Cointelegraph, Anthropic Institute lead Marina Favaro and Anthropic co-founder Jack Clark wrote in a Thursday blog post that agents can already run code, delegate hours of work to other agents, and may be nearing the ability to take over more of the development cycle. They said Anthropic is delegating a growing share of AI development to AI systems, speeding up its work, and that with enough compute the trend could point to an AI system capable of autonomously designing and developing its own successor. Favaro and Clark added that AI model improvement has been roughly doubling every four months rather than every seven months, and said Anthropic’s Claude model authors around 80% of the code merged into the company’s codebase.

Favaro and Clark said recursive self-improvement is “not inevitable,” but could arrive sooner than many institutions are prepared for, particularly if human- and AI-authored code quality reaches parity. They argued that humans could shift from writing code to reviewing it, but warned that if people cannot review code as quickly as systems like Claude can generate it, human review could become the bottleneck to AI development. They also pointed to broader industry efforts, noting that in December OpenAI said it is researching how to safely develop and deploy increasingly capable AI, including systems capable of recursive self-improvement, with goals such as avoiding catastrophic behavior and keeping systems controllable, auditable, and aligned with human values. Favaro and Clark also referenced an open letter released Thursday by tech leaders, including some from Anthropic and OpenAI, urging lawmakers to strengthen guardrails amid concerns AI could help overcome “knowledge barriers” that have historically limited the creation of biological weapons.

While calling a slowdown or temporary pause an option to let societal structures and alignment research keep pace, Favaro and Clark said a unilateral slowdown could backfire if it allows less cautious actors to catch up, potentially leaving everyone less safe. They said that without a global coordination mechanism, companies and governments may face difficult safety decisions under competitive and geopolitical pressures. The blog post followed Anthropic’s April decision to rule out releasing its AI model, Claude Mythos, to the public for now due to concerns about threats to global cybersecurity, after the model was able to easily create software exploits. Separately, the report noted that AI agents are gaining popularity among crypto users, with some executives suggesting agents that settle transactions could boost adoption and volumes; Circle CEO Jeremy Allaire predicted in January that billions of AI agents would operate on users’ behalf within five years. It also cited crypto investment firm Keyrock, which reported last month that AI agents settling payments moved from concept to reality in the past 12 months, with $73 million settled across 176 million transactions.