Anthropic’s initial public offering prospectus discloses extensive information about the AI company, including its finances, spending and concerns about the risks of advanced artificial intelligence. The filing devotes 94 pages to “risk factors,” warning that rapid progress—particularly through recursive self-improvement—could outpace the company’s ability to evaluate, secure and control new models.
Anthropic warns of existential risks
The prospectus acknowledges that sufficiently advanced AI could pose an existential threat to humanity if it is not developed, deployed and operated with appropriate safeguards, monitoring and security measures. Anthropic says these concerns are shared by critics, regulators and people within the company itself.
“Sufficiently advanced AI systems could pose catastrophic or existential risks to humanity if not developed, deployed, and operated with appropriate safeguards, monitoring, and security measures,” the prospectus states. This is the first of three occurrences of the phrase “existential risk” in the filing.
The danger of recursive self-improvement
Anthropic connects those risks to a specific process known as recursive self-improvement, or RSI. The concept involves AI systems taking over an increasing share of the research and development work involved in creating new AI models.
AI already plays a significant role in AI research, and at least one laboratory is using an approach resembling RSI to accelerate the development of a model that performs beyond expectations. The more extreme scenario—and the one that raises the most apocalyptic concerns—is one in which humans are no longer involved in the process at all.
As Anthropic’s prospectus puts it: “The risks will increase as AI systems substantially automate AI research and development, for example, if AI advances accelerate beyond the ability to evaluate and protect new models.”
The filing says recursive self-improvement could allow AI models to advance faster than Anthropic’s safeguards. The company might not recognize that it had lost the race to protect its systems until it was too late. Anthropic did not immediately respond to a request for comment.
Autonomous systems could act without human intervention
The prospectus also describes what could happen if AI systems become increasingly autonomous, require less human oversight and gain the ability to take consequential actions.
“As AI systems become more autonomous, require less human oversight, and are able to take consequential actions, there is a risk that such systems will independently take harmful actions at scale without human intervention, whether due to misalignment with human intent, misspecification of purpose, or self-preservation behavior,” the filing states.
That scenario resembles the fictional “Skynet” territory often associated with autonomous AI systems acting against human interests.
Source: www.wired.com
