AI’s Tightrope Walk, When Capability Starts to Look Like a Weapon

Further to my blog at the weekend on the compound systemic risk from an AI runaway event, something further to consider following initial comments I have received, concerns the threshold for offensive use classes of AI falling under Article 36, Additional Protocol-1 and Weapons Review of Lethal Autonomous Weapons System . Noting also that whilst the US has signed, it has not ratified ‘Additional Protocol 1’ and states it is not formally bound by Article 36 as a treaty obligation. 

I suspect in some quarters the view is being taken that AI is navigating an increasingly fine line between advanced software and a capability that may require an Article 36, Additional Protocol-1 of the Geneva Convention and or equivalent  ‘Weapons Review of Lethal Autonomous Weapons System(s)’.

Why do I think this? This is consistent with US Cyber Command’s published AI roadmap, which aims to scale cyber operations and improve adversary disruption and with its commander’s statement that AI is being applied to exploitation and manoeuvre in cyberspace. AI has almost certainly already been operationalised within the US cyber mission (Israel, Russia and China undoubtably as well), including activities that support offensive effects, even though the specific capabilities, targets and degree of autonomy remain undisclosed.

The legal threshold is not crossed simply because a model is powerful, unpredictable or capable of misuse. It is crossed when an AI enabled system is developed, acquired or adopted as a weapon, means or method of warfare, with Cyber a very real battlefield in itself. The decisive issue is therefore not what the model is called but what it is authorised to do. That distinction is becoming harder to maintain, and the US most recent restriction of Anthropic’ s Fable 5 AI model sends some strong reinforcing signals to this end.

A model that identifies vulnerabilities, selects targets, generates exploits, controls autonomous platforms or materially influences lethal decisions may begin as general purpose technology but become part of a military capability through integration, permissions and intended use.

The same tension applies in cyber operations. An AI agent that autonomously compromises systems or disrupts critical infrastructure may produce effects comparable to a conventional weapon, even though its mechanism is software. AI developers and governments are therefore balancing on a tightrope. Keep systems too constrained and their strategic value diminishes. Give them greater autonomy, tool access and decision authority and they move closer to the territory of weapons level scrutiny.

The deeper issue is that AI changes continuously. A one time legal review may no longer be enough. The future may require persistent review of models trustworthiness across criteria like permissions, data, behaviour and operational effects, amongst others.

Of concern is how AI is also illustrating the outmoded nature of instruments such as Article 36. It was drafted for relatively stable weapons whose characteristics, effects and intended uses could be assessed before deployment. AI does not fit that model comfortably, if at all. Its behaviour changes through retraining, fine tuning, software updates, new data, tool access, operational context and interaction with adversaries or other autonomous systems. The provision also assumes that the object of review can be clearly defined as a weapon, means or method of warfare, whereas AI sits across intelligence, targeting, cyber operations, decision support and autonomous execution without fitting neatly into any single category. A one off pre-deployment review may therefore approve a capability that later behaves materially differently from the system originally assessed. Article 36 remains legally valuable but it is poorly adapted to technologies that are dynamic, general purpose, continuously evolving and capable of producing emergent or cascading effects like AI or AI augmented.

AI requires broader, continuous governance review throughout its lifecycle, without that evolution, the first true test of AI weapons governance may come only after an irreversible event has already occurred.

The Pall Mall Process provides a valuable governance focused foundation by demonstrating how states, industry and civil society can establish shared expectations around powerful dual use capabilities before binding international law is in place. Its principles of accountability, precision, oversight and transparency, applied to the development, purchase and use of commercial cyber intrusion capabilities, could be extended to advanced AI systems capable of enabling cyber operations or systemic disruption.

The next logical step then is to build on these governance foundations to create an international body as a response to moving beyond voluntary responsible behaviour commitments into continuous monitoring, mandatory incident escalation, coordinated containment and international recovery capability when AI-enabled threats cross borders or sectors.

After all, the commercial AI developers themselves acknowledge the growing difficulty of containing advanced models even within controlled sandbox environments. That should be treated as a warning, not a footnote, once increasingly autonomous systems are connected to live tools, networks and infrastructure, containment failure ceases to be a theoretical possibility and becomes a matter of when not if.