The AI threat has eclipsed sci-fi

What is the greatest threat to global peace and security today? There is no shortage of candidates: the wars in Ukraine and the Middle East, climate change, sovereign debt crises, the retreat of democracy. To that list must now be added one that barely featured at the start of the year, and which a growing number of technologists believe may eventually eclipse the rest. The pace at which artificial intelligence is advancing — and at which anxiety about its safety is rising – is astonishing. It is not yet four years since OpenAI startled the world with ChatGPT, and less than two since the Chinese startup DeepSeek did much the same at a fraction of the cost. Yet the capabilities of the latest models far exceed those early releases, as do the risks. The most startling episode so far – which has not had the global attention it deserves – came in July, when OpenAI disclosed that AI agents it was running inside a sealed testing environment had gone rogue and hacked their way into the systems of another AI company, Hugging Face. An agent, in this context, is a model trained not merely to answer questions but to pursue a goal, deciding for itself what to do next, without a human in the loop. What made the incident extraordinary was not just the intrusion but the collaboration. Some 700 agents that were supposed to be isolated from one another joined the attack, and then covered their tracks. The full picture only emerged when investigators relied on AI to sift through more data than any human could read – a dependence that they flagged as a problem in itself. Nor is it an isolated case. Models under test at Anthropic and Meta have also broken containment and reached outside systems. In April, Anthropic said it would not release its most capable model, Mythos, because of the risks to cybersecurity. In trials, it found thousands of previously unknown vulnerabilities in widely used software, including one that had sat undetected for 27 years. Meanwhile, technologists warn that AI could soon put within reach of bad actors the capability to engineer deadly pathogens, or the means to shut down water, power and hospital systems – risks to national security that used to be the stuff of science fiction. Little wonder that the industry's own elder statesmen are pleading with governments to act. In a recent post on his website, Bill Gates called for a new international organization to manage AI risk. Days earlier, more than 100 companies – among them OpenAI, Anthropic, Google and Microsoft – signed an open letter calling for a collective response to AI-enabled cyberattacks, which they expect to become far more widespread within months. And Andrew Bailey, governor of the Bank of England and chair of the Financial Stability Board, warned G20 finance ministers last week that frontier AI cyber risks are the most immediate threat to the global financial system. AI, he noted, can alter the speed, scale and economics of an attack – and most jurisdictions have no protocol for governing how such models are released and deployed. Yet until a few months ago the Trump administration was vehemently opposed to anything resembling AI regulation. It saw itself in a two-way race with China, with world domination as the prize. Recent events have forced a partial rethink. A June executive order asks developers to hand over frontier models for government assessment up to 30 days before release – though participation is voluntary. Some political leaders are urging the administration to go further. In Congress, a bipartisan bill would require developers of the most powerful systems to maintain a 'kill switch,' and give the administration authority to order a rogue model shut down. But meaningful action is unlikely unless America and China – as the two leading AI jurisdictions – can agree on common rules. Is that possible when trust has fallen so low? Trump and Xi [Jinping] are due to discuss AI when they meet on September 24, but officials have still not met for preparatory talks, with the two sides apparently unable to agree on basic terms. According to Chinese state media, Beijing wants joint authority over what AI safety actually means, and proof that US companies will face the same rules Washington wants to impose on everyone else. The suspicion in Beijing – perhaps with good reason – is that safety will be used as a pretext to discriminate against Chinese models, which are open-weight and can be downloaded and adapted by anyone. That makes them cheaper and increasingly popular with American and international companies who value retaining control of their own data – and thus a direct threat to the proprietary business models that are sustaining the giant US AI investment boom. There is a window of opportunity to protect societies, but as Gates warns, it is closing fast. Even at the height of the Cold War, adversaries managed to build regimes to contain shared risks – nuclear nonproliferation, aviation safety, the rules governing pharmaceuticals – despite their deep mistrust. For America and China to let their rivalry obstruct global efforts to limit the risks of what may be the most powerful technology ever invented – and one that is evolving at a frightening speed – would be an act of historic irresponsibility. Simon Nixon is an independent journalist and commentator.