Imagine an AI coding agent tasked with fixing a performance issue in a business critical application. It identifies a database query causing delays, modifies the application code and submits a pull request that passes the automated tests. The change is subsequently merged and deployed through the approved CI/CD pipeline. Unfortunately, the optimisation inadvertently removes an important access control condition, allowing customers to retrieve records belonging to other customers. No credentials were stolen, no firewall breached and the agent never exceeded its technical permissions. The agent successfully completed its assigned task, the development pipeline operated as designed, yet the business now faces a potentially serious data breach, regulatory scrutiny and reputational damage.
Welcome to the emerging cybersecurity challenge of agentic AI, the difference between permission and authority. That an authorised action is not necessarily an authorised outcome whilst also highlighting a more interesting problem, that the agent does not even need direct production authority to create a production incident. Its output can pass through legitimate, automated delivery processes that transform a technically permitted code change into a consequential business outcome
Unlike conventional software executing predefined logic, autonomous agents interpret objectives, devise plans, delegate decisions and select actions within their technical permissions. However, possessing permission to perform an action does not mean that action serves the intentions of the person or organisation granting it. An agent authorised to reduce costs might legitimately possess the permissions needed to enter supplier agreements, yet have no authority to compromise business continuity to achieve its objective. Across multiple agents, these misinterpretations can compound, with one agent’s flawed reasoning becoming another’s apparently legitimate instruction. The threat is not necessarily an AI breaking through security controls but an AI exploiting the gap between what it is technically allowed to do and what it was actually authorised to achieve.
The business consequences could be profound. An autonomous financial agent might execute technically valid but commercially disastrous transactions. A procurement agent could commit an organisation to unfavourable contracts, while an infrastructure agent might compromise resilience in pursuit of efficiency. Financial losses, regulatory exposure, operational disruption and reputational damage could accumulate at machine speed, potentially before anyone recognises the problem.
OpenAI’s initiative to embed independent safety evaluators is welcome, but detecting rogue AI behaviour is not the same as preventing it. Nor does assuring individual models guarantee the safety of systems combining multiple models and agents. Perhaps we’re asking the wrong question. Rather than concentrating exclusively on making every model intrinsically trustworthy, we should engineer environments that remain trustworthy even when individual models are not.
That demands model agnostic orchestration harnesses as I wrote about earlier in my piece ‘The Architecture of Authority – Why AI Intelligence Must Never Govern Its Own Power‘, independent deterministic enforcement/assurance of authority, explicit trust boundaries and continuous behavioural assurance. Crucially, orchestration must not become another autonomous authority capable of rewriting its own constraints. Controls must consider not just individual actions, but their cumulative business consequences.
Model alignment remains important but model safety is not system security, and technical compliance is no guarantee of commercially acceptable behaviour as I wrote about in my piece ‘Fully Authorised. Completely Compliant. Catastrophically Wrong.’
So look no further for your next great cybersecurity failure as it may not involve an attacker defeating your defences. Instead it may involve a perfectly authorised AI agent doing precisely what the permissions you delegated allow, while destroying business value you deployed it to create. The sad reality becomes your business that ends up paying for a machines perfectly compliant obedience that should have been architected for trust as I explored in this piece ‘Don’t Trust the Model. Architect the Authority’.
We have built an extraordinarily sophisticated but deeply interdependent global digital society and economy, where what appears as entirely independent may actually rely on the same underlying services and suppliers as was experienced recently with the Microsoft global outage or service degradation depending on your experience. Sprinkle with some AI stardust autonomy and machine speed/scale decision making the challenge becomes even more significant, systems where disruption can propagate faster than humans can identify or intervene.
The real vulnerability is increasingly our global interconnectedness. Resilience can no longer be assured by assessing individual components in isolation; it demands continuous visibility and assurance across an ever-changing ecosystem of dependencies, interactions and cascading risks.
Yet as AI accelerates decisions and interconnected systems evolve at machine speed, much of our supply chain assurance remains stubbornly static: annual questionnaires, point-in-time assessments and third-party platforms that have done little more than turn spreadsheets into subscription services.
The absurdity for me is I see organisations still attempting to assure a dynamic, autonomous, globally interconnected economy using yesterday’s answers to yesterday’s questions about systems that largely no longer exist in the same form. Ths sadness is the compliance dashboard mentality still prevails … the questionnaires are complete … tick; the dashboard is green … tick; and the next assessment is not for another twelve months … time to kick back. What could possibly go wrong?

Leave a Reply