It’s Not the Model, Stupid … Standards Must Govern Authority, Not Intelligence

This week has barely opened and OpenAI’s gambit to stay in the headlines is rolling in the form of a call for the United States to lead an international coalition on AI standards.

The instinct is understandable, but there is a danger that we repeat a familiar mistake, creating principles after capability has already escaped the boundaries those principles were supposed to define.

We already have political declarations through Bletchley, Seoul; international principles/codes in the form of the G7 Hiroshima Process, OECD principles; frameworks like NIST AI Risk Management Framework, a genuine standard like ISO/IEC 42001 (Information technology AI Management system) and the Council of Europe’s AI convention which is a treaty. Calling all of them simply standards would understate the important differences in their legal and operational force. My point is do we really need more, or do we need to look through unbiased eyes and cognition that is tuned to the exponential machine world of today and not the old linear world of yesterday. Ater all our linear world already suffers from regulatory lag with innovation, now we are in a machine speed world of exponential leaps expressed in hockey stick curve fashion, starting from scratch is simply not an option.

My sightline on the problem is as I state above not for the absence of standards et al and their counterweights. It is that we still confuse governance with control.

Most currently ask what an AI system can do, how capable is it, can it deceive, replicate, conduct cyber operations, circumvent safeguards or act autonomously?  All valid questions but increasingly they may not be the most important one.

The more consequential question is Who authorised it to act? As I wrote about in my piece titled ‘The AI Risk Nobody Is Measuring – Authority‘.

A highly capable model confined to a sandbox may present less immediate danger than a weaker autonomous agent with credentials, application programming interfaces (APIs), payment authority, cloud privileges and permission to make decisions at machine speed. The issue therefore is that standards focused principally on controlling model development are addressing only part of the ecosystem.

There is also a deeper architectural point. If we design the system correctly, the model itself should become almost irrelevant to transactional trust. Not something I guess the Frontier labs want to hear, as they continue to battle to stake a claim on the public psyche. OpenAI’s current proposals are still largely model centric, based on common capability evaluations, risk management, incident reporting, external assessment and internationally compatible standards for advanced AI systems. It is also supporting work such as Appia to create reusable conformity evidence across models, infrastructure and applications. Don’t get me wrong, that is useful but it has an obvious structural weakness the model is the least stable part of the system. Models change, they are fine tuned, distilled, forked, locally hosted, combined with tools, wrapped in agents and, in the open-model world, potentially stripped of safeguards altogether, only a ‘load manifest’ away which collapses Open AI’s standards  fantasy. Even OpenAI’s own governance material recognises that the absence of observed failures does not establish reliability across all settings. So I would separate AI safety standards from transactional trust standards.

(Look for a future piece I am drafting on the infrastructure revolution that will democratise models for everyone no kill switch included).

My view is that Trust should not depend on whether the underlying model is OpenAI, Anthropic, an open-weight model, a private enterprise model or an uncensored local model. The transaction should be trusted because the surrounding architecture proves identity, validates delegated authority, constrains permissions, enforces policy, records provenance and produces verifiable evidence. In other words, we should not need to trust the model in order to trust the transaction.

However there is an important limit to that proposition. A trustworthy transaction is not necessarily a trustworthy outcome. Capability creates potential, authority creates consequence.

An architecture may prove perfectly that an agent was authorised to make a payment, deploy some code or change a customer record. It does not, by itself, prove that the information informing that decision was correct, that the authority should have been granted in the first place, or that the resulting action will not create harmful consequences elsewhere.

That means AI assurance needs to distinguish at least three things:

1. Transactional trust – was the action properly authenticated and authorised?
2. Epistemic trust – were the data, evidence and reasoning upon which the action relied sufficiently trustworthy?
3. Consequential trust –  did the action remain acceptable once its effects propagated into the wider system?

This matters because autonomous systems will increasingly operate as networks rather than isolated agents. Individually authorised actions may combine into systemic outcomes that no single policy decision anticipated. A thousand compliant transactions can still produce a non-compliant consequence.

So back to the clarion call for AI standards. I believe these cannot stop at controlling access and authority. They must also govern the delegation chain, the integrity of the evidence informing decisions, the provenance of data, the interaction between agents, aggregate behaviour, human accountability and the ability to interrupt cascading effects.

There is a further danger though, the control architecture itself becomes part of the attack surface. What I mean is if the identity provider, policy engine, delegated authority mechanism, monitoring system or evidence trail can be compromised, an attacker no longer needs to defeat the AI model. They simply convince the control plane that an illegitimate action is legitimate.

The architecture therefore needs its own roots of trust, independent verification and separation of duties. This changes the standards problem again because trying to govern every model, every variant and every future capability may become increasingly impractical. Governing the architecture through which models are permitted to act may be far more durable but only if that architecture governs authority, evidence and consequence not authority alone.

So my warning to the standards champions is that any meaningful global framework therefore needs to move beyond principles into enforceable machine speed assurance, common measures, independent testing, intervention thresholds and incident reporting, but above all, it must govern authority to operate.

That does not mean giving every AI agent a permanent identity. Persistent machine identities may themselves become a source of risk by allowing privilege, reputation and authority to accumulate. As I discussed in a piece that asked ‘Who Gave the Machine Permission?‘.

The recent Mills review, title ‘AI and the future of retail financial services‘, on the whole I felt is on point. There is one nuance I would change in Mills’ language, he says agents need a ‘trusted identity’. I would be cautious about interpreting that as every agent having a permanent identity. What I did find interesting is Mills is really trying to govern three different things at once, notably the model, the agent and the financial action where in a commercial hosted model, regulators may potentially influence all three. However to the point made earlier for an uncensored local model, they will have almost no leverage over the first. Again we see the architecture has to move away from trust in the model towards what it is permitted to do. That is a much stronger model (excuse the pun!). So I would characterise the Mills Review as a surprisingly practical regulatory blueprint, provided it is treated as an architecture programme rather than a conventional policy programme, time will tell.

In what I have witnessed in the wild across some diverse organisations what has directed my thinking is that agents should instead increasingly operate through ephemeral, purpose bound identities attributable to an accountable human, organisation or system, issued for a defined task, constrained by explicit permissions and duration, and revoked or allowed to expire when that task is complete. This requirement is not persistent identity, it is persistent accountability.

This is where the role of the CISO changes as I hypothesised at the weekend when suggesting ‘The New AI Era CISO Mandate, From Security to Authority’. In which I suggest the CISOs role evolves into the ‘Architect of Authority‘, not merely defending infrastructure, but determining what humans and machines are permitted to do within it and increasingly whether the evidence, delegation and consequences surrounding those actions remain trustworthy, which is where AI governance collides with accountability.

After any serious AI driven failure, the uncomfortable question will not simply be whether the model was intelligent, aligned or predictable, it will be:

• Who gave it the authority?
• Who approved the access?
• On what evidence?
• Who determined that evidence could be trusted?
• Who considered the consequences beyond the immediate transaction?
• Who allowed the machine to cross from recommendation into action?

Standards that cannot answer those questions risk becoming little more than post event paperwork.

We may eventually discover that the central governance challenge of the AI era was not how to make every model trustworthy. It was how to build systems in which trust did not depend on the model, authority did not escape accountability and compliant actions could not quietly accumulate into unacceptable outcomes.

So be warned, because when the inquiry comes, ‘the AI did it‘ will not be an answer. The much more visceral question you will need to be able to answer is ‘What did you authorise it to do, what did you allow it to believe and what happened when those actions combined?’