AI doom scenarios often assume that sufficiently advanced intelligence will eventually understand, predict and manipulate the world better than humans can. The kicker here is that civilisation is not a bounded game. It is a complex adaptive system in which participants respond to one another, interventions change behaviour and genuinely novel events continually alter the rules.
AI may become extraordinarily capable within such systems but the model remains a compressed representation of reality rather than reality itself. Goodhart’s Law reminds us that once a measure becomes a target, behaviour adapts around it. The Lucas critique makes a related point: that historical relationships cannot necessarily be assumed to persist once actors change their behaviour in response to intervention. Applied to AI both suggest that an intelligent system acting upon the world may progressively invalidate the very assumptions on which its predictions were built.
This challenges the more extreme image of superintelligence as near omniscience. Chess can be mastered because its rules, objectives and state space are bounded. Civilisation is not chess. Real world systems contain hidden variables, nonlinear feedback loops, irrational actors, regime changes and events that have never previously occurred.
The greater risk may therefore not be that AI perfectly masters complexity but that it becomes powerful enough to materially alter complex systems without fully understanding the consequences. That risk becomes more profound when AI itself constitutes a growing proportion of the actors inside those systems. Financial markets populated by autonomous traders, machine customers negotiating supply chains, defensive cyber agents fighting offensive agents and procurement systems negotiating continuously with other machines create something qualitatively different. AI is no longer merely modelling the complex dynamic environment. AI becomes part of the complex dynamic environment.
At that point, systemic behaviour may emerge that no individual model intended. Each agent can behave rationally and correctly against its own objective while the collective outcome becomes unstable. Models may be wrong but decisive, locally optimal but systemically damaging, or self-invalidating because their actions change the environment from which their assumptions were derived.
The resulting danger looks rather different from the familiar image of one omnipotent machine taking control. It looks more like millions or billions of autonomous systems interacting at machine speed, creating feedback loops and outcomes that neither humans nor individual AI’s can reliably predict.
Imagine thousands of AI machine customers autonomously buying cloud capacity, energy, insurance or logistics. One detects a possible GPU shortage and brings purchases forward. Supplier agents raise prices; other procurement AIs interpret the increase as confirmation of scarcity and accelerate buying; financing and insurance agents tighten terms; logistics systems reprioritise supply. Within minutes, a weak signal becomes a real shortage.
Nothing has been hacked and no AI has simply gone rogue. Every agent may be behaving rationally against its own objective, yet collectively they create a machine-speed bank run; prediction > automated action > market signal > more automated action > self-fulfilling crisis. The danger is not AI taking over the world; it is millions of legitimate autonomous decisions combining into systemic chaos faster than humans can intervene.
Paradoxically, then, the strongest challenge to the classic AI doom scenario may expose a more credible source of risk. AI may never perfectly master the complexity of the world. It may instead become deeply embedded within that complexity and simply cause chaos that follows all the rules.
The central question is therefore not simply whether AI becomes smarter than humanity but whether we can build sufficient trust, governance, observability and intervention into machine-speed systems before their collective behaviour begins producing outcomes nobody, human or machine, fully intended.
The cyber response to such emergent AI behaviour needs to move beyond securing individual models and focus on the system of interacting agents. This means establishing strong machine identity, verifiable authority, least privilege access, transaction provenance, behavioural monitoring and continuous assurance across agent-to-agent interactions. Security controls must be capable of detecting abnormal feedback loops, coordinated or cascading behaviour and locally rational actions that create systemic risk, with automated circuit breakers to slow, isolate or terminate activity before instability propagates. In effect, the objective is to create a trust and control layer for autonomous systems, one that can continuously answer the key challenges when things go sideways. which agent acted, under whose authority, using what data, with what outcome and can the action be constrained or reversed?
For now there is perhaps too much trust being thrown around. I see organisations trusting the vendor demo, accepting the assurance slide, they admire the efficiency gains and assume someone, somewhere has thought through what happens when thousands of autonomous systems begin reacting to one another at machine speed. That is not innovation; it is wilful blindness with an API (application programming interface). If organisations cannot answer the aforementioned control layer challenges, they are not governing AI, they are simply accelerating into uncertainty. Machine-speed chaos will not politely wait for a committee meeting, a quarterly risk review or the vendor’s next patch cycle. It will produce machine-speed consequences and for some organisations those consequences will be machine-speed business terminal events.
Leave a Reply