Earlier in the month, Anthropic revealed that it had identified five instances in which individuals utilized its AI models for projects that could potentially assist in the development of biological weapons. These activities included helping form a proposal related to gain-of-function research on a virus transmitted by mosquitoes at a military facility, as well as devising experiments aimed at enhancing the infectivity of bird flu in mammals. Although Anthropic was unable to assess the intent behind these actions, the company decided to block the accounts linked to them.
This uncertainty raises critical concerns. The very tools that researchers might use to create vaccines can just as easily be manipulated to produce harmful pathogens. Similarly, while it’s possible to formulate a treatment targeting cancer cells without affecting healthy ones, there is potential to adjust those tools to create drugs that harm healthy cells instead. Users don’t necessarily need to explicitly instruct Claude to “create a bioweapon”; they can simply pose as scientists addressing vital biological challenges.
Anthropic has been particularly vocal about the implications of AI in bioweapon development, while OpenAI has also introduced certain safeguards. In contrast, other AI firms, especially those offering open-weight models—whose code can be downloaded and altered freely—generally release their tools with minimal protections and oversight regarding usage.
A significant portion of the industry discourse centers on whether to "pace the frontier," meaning to slow the pace of model development to allow safety measures and countermeasures to catch up. This approach is valid as long as it leads to improved biodefense systems. However, merely delaying model development does not address the ease with which synthetic DNA can be obtained or what state governments are doing to enhance their bioweapons capacities.
For many years, creating biological weapons was uncommon due to the extensive expertise required across diverse scientific fields. Mastery in immunology, virology, culturing pathogens, and stabilizing dangerous mutations necessitates a combination of skills that are not easily found together.
AI is narrowing this knowledge gap. Models designed from biological literature can bridge concepts across various subdisciplines, enhancing the capabilities of individuals with expertise in specific areas by equipping them with knowledge from other fields.
Both a Harvard report and a RAND study released this year independently assessed this landscape and reached similar conclusions: AI is expanding the pool of individuals capable of orchestrating large-scale biological attacks and enhancing the efficacy of existing state bioweapons programs as well. While neither asserted that we have crossed a critical threshold of danger, the consensus leans toward the identification of risks that warrant immediate attention.
The State Department's declassified intelligence reports have consistently indicated that Russia and North Korea possess offensive biological weapons capabilities, raising alarms about China and Iran as well. The Washington Post highlighted a significant expansion at Sergiev Posad-6, a former Soviet bioweapons facility, showcasing new high-containment structures. It is essential to note that much of the intelligence activity predates the advent of AI, but its implications for expanding state programs and novel designs are undeniable.
Terrorist organizations and other non-state actors have also sought these weapons through the establishment of independent capabilities or by targeting existing facilities. This involves not just amateur attempts but rather serious stakeholders equipped with resources and empowered through AI-driven biological engineering tools.
Moreover, there is a second obstacle that is rapidly diminishing.
A recent paper published in Nature Communications reported an experiment in which researchers procured genetic segments of the 1918 influenza virus, with a staggering 36 out of 38 DNA synthesis companies fulfilling their orders. This provided a proficient individual the means to reconstruct the virus. While regulations from the U.S. Select Agent Program — overseen by the CDC and the USDA — prohibit complete genetic sequences capable of replicating perilous pathogens, shorter, easily assembled segments still lack regulation.
Yet, the broader issue lies in the realization that even a perfectly functioning screening system that restricts access to harmful DNA fragments would still fall short. The DNA synthesis market operates globally; sequences restricted in one country can be obtained from another. If someone is determined enough, access to the necessary materials will eventually be secured.
The pace of threats is accelerating faster than our defense mechanisms can adapt. An AI-generated genetic blueprint could transform into an actual pathogen within weeks. Admittedly, the transition from conception to realization of a viable pathogen remains complex, and the failure rates are significant. Nonetheless, as more individuals pursue this pathway, the likelihood of success for some increases.
The field of biomedicine is currently experiencing one of its most fruitful eras in decades, largely fueled by these very AI tools. Imposing extensive restrictions could hinder advancements in understanding and treating diseases that could otherwise be addressed more swiftly. Crafting barriers to prevent the misuse of tools for bioweapons development, without stifling therapeutic progress, poses a significant challenge.
In light of this, a layered defense strategy is essential, beginning with both access control and model design. It's reasonable to allow verified researchers access to state-of-the-art biological models for legitimate purposes, provided there is adequate oversight in place. Furthermore, developers should test their models for potential dangers before public dissemination.
Governments should consider requiring DNA synthesis companies to authenticate their clients, screen for fragmented orders, and participate in independent audits to ensure system efficacy. Although there is a general agreement on the need for these measures, their effectiveness remains uncertain.
Encouraging open-weight AI developers to embrace similar testing standards and safeguards as leading labs (like Anthropic and OpenAI) is important, though complete compliance is not guaranteed. It's also crucial to remember that once such models are released, they cannot be retrieved.
Much of the ongoing discourse around AI regulation hinges on the issues of speed, autonomy, and control over future developments. A temporary pause in progress might offer a valuable opportunity to address all three of these concerns.
However, this time to regroup is only beneficial if it contributes to bolstering defenses. Relying on every lab, nation, and open-source developer to maintain biosecurity standards is impractical; failure to do so could lead to the increased likelihood of developing and potentially using biological weapons. Building robust defenses against such threats takes considerable time, and unfortunately, we may not have the luxury of years to prepare.
Ashish K. Jha, a former White House COVID-19 response coordinator, now serves as a senior fellow at the Belfer Center at Harvard Kennedy School and is also the co-founder and CEO of BioRadar, which focuses on systems for detecting biological threats.
