In recent weeks, amid extensive discussions surrounding the potential dangers of superintelligent AI leading to human extinction, another troubling story has emerged that received far less attention. A CNN report highlighted a situation where the US military's dependence on unreliable chatbots nearly sparked a conflict with China.
The resignation of engineer Jacob Coxon from Anthropic generated significant headlines, as he criticized the AI race between OpenAI and Anthropic, stating that they are “racing straight towards self-improving superintelligence and gambling with our lives.” His portrayal of a potential "terminator" scenario, where machines surpass human intelligence and view humanity as a threat, has captivated public and media attention, inciting a wave of fear-driven reporting and urgent calls for regulation.
In response, California Governor Gavin Newsom mandated the development of an “AI kill-switch,” while New York Governor Kathy Hochul is similarly investigating the feasibility of such measures. Additionally, Senator Bernie Sanders and Congressman Greg Casar have introduced measures aimed at banning artificial superintelligence altogether.
Amid this frenzy over hypothetical dangers posed by superintelligent machines, a more immediate concern unfolded. This recent incident nearly brought the US to a critical tipping point, not due to a rogue AI, but because of reliance on flawed systems that are marketed as near-superintelligent.
On September 18, CNN reported an alarming near-miss where an intelligence review suggested that a Chinese ship was carrying nuclear weapon components. The military, believing this information to be accurate, readied the necessary forces for an interception. Such an engagement could have easily escalated into a broader conflict, potentially instigating World War III.
It was only moments before these military actions were set into motion that officials conducted a deeper examination of the intelligence report, revealing that it had been generated by a chatbot and was based on erroneous assumptions about the ship's cargo.
Unlike the sensational narratives surrounding fictional AI threats, this incident did not prompt a widespread governmental response or call for regulation emphasizing the dangers of unreliable automated systems. Notably, the press largely refrained from warning about the close brush with war arising from a military service's reliance on flawed chatbot technology.
Current large language models (LLMs) that power these chatbots are often dubbed “stochastic parrots,” essentially designed to mimic patterns in their training data. Media portrayals that classify them as “powerful” or “rogue” contribute to misguided perceptions of their reliability, particularly within critical applications like military operations.
Even CNN, amidst its critical reporting, referred to the military’s chatbot as presenting “profound risks” due to its status as “powerful, new and relatively poorly understood technology.” However, the reality is that today’s LLMs are neither poorly understood nor particularly effective for such high-stakes contexts. They are systems that produce plausible-sounding text based on vast, variably-sourced datasets that are further refined for specific users.
While it remains unclear which specific chatbot was involved in the military’s close call, it’s likely that the underlying LLM was modified to generate reports resembling intelligence summaries. The potential drawbacks of such models have been widely recognized, highlighted in previous discussions like the 2021 paper titled "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"
The suggestion that the behavior of these systems is somehow enigmatic stems from intentional obfuscation from their promoters and a misunderstanding among users regarding their outputs. Unlike the fantastical assertions from executives at OpenAI and Anthropic, the real risks associated with these systems arise from the unfounded belief in their superintelligence rather than any actual capabilities they possess.
The drawbacks of placing trust in flawed automated software are evident, as seen in cases of hospitals incorrectly categorizing patients or governments mistakenly conducting airstrikes on civilian sites due to misleading intelligence assessments.
Amidst these discussions, the marketing efforts of companies promoting AI steer public concern towards imaginary threats rather than addressing the proven issues tied to their products.
It is imperative that regulations surrounding these so-called “AI” systems be established, not due to any inherent magical qualities, but because they are prone to error and should not be applied in life-and-death scenarios without rigorous scrutiny. Any automation implemented in critical sectors such as military or healthcare must be carefully assessed, accompanied by clear accountability measures.
As former FTC chair Lina Khan has underscored, laws are already in place that govern the AI sector. Federal agencies and regulators can utilize existing legislation to safeguard the public against the AI industry’s irresponsible promotion of unreliable products veiled as advanced intelligence.
It is crucial that journalists and lawmakers challenge these corporate narratives, ensuring accountability rather than simply echoing corporate messages.
