Beyond Code Assurance – Why AI Regulation Must Assure Decisions as Well as Software

For decades, software assurance has been built on a relatively stable premise. If engineers correctly implement a well defined specification and that implementation can be tested, reviewed and verified, confidence follows. The question stakeholders and particularly regulators have traditionally sought to answer is straightforward ‘Did the engineers implement the specification correctly?’ That model served us well because conventional software executes explicit instructions written by humans (error rates accepted as inherent as I wrote about in 2017 – Digital Dementia born of Artificial Intelligence). Every decision ultimately traces back to code someone intentionally created. Artificial intelligence (AI) changes that assumption.

Modern AI systems do not simply execute human written logic. They make decisions based on billions of learned weights acquired during training. Although the underlying software architecture may be thoroughly understood, the behaviour that emerges from those learned parameters cannot be fully inferred by reviewing the source code alone.

The regulatory question therefore changes fundamentally. Instead of asking whether the software was implemented correctly, regulators must increasingly ask ‘Does this trained model behave acceptably under the conditions in which authority has been delegated to it?’ This is not a subtle distinction. It is an entirely different assurance problem.

Consider healthcare. AI is increasingly supporting and in some cases directly influencing, diagnosis, treatment prioritisation, clinical workflow and resource allocation. The software itself may contain few defects, yet the model could still exhibit unexpected behaviour, produce unsafe recommendations, develop biases or fail under conditions that were not anticipated during development. Reading the source code will not reveal these behaviours because they are not encoded as explicit instructions. They emerge from the interaction of billions of learned weights within the model. Consequently, assurance must evolve from software assurance to decision assurance.

The challenge is therefore no longer simply assuring that engineers built the software correctly, as we know and referenced above there are statistical error rates for example in 1,000 lines of code expect between 15 – 50 bugs that could constitute vulnerabilities (quoting from the book  ‘Code Complete: A Practical Handbook of Software Construction, Second Edition 2nd Edition’). It is assuring that the intelligence created by that buggy software continues to behave acceptably wherever society delegates authority to it.

What exists today is a collection of governance, risk management and testing frameworks. They move us in the right direction but they generally stop short of providing the kind of continuous behavioural assurance that safety critical AI is ultimately requiring. The most interesting development, in my view, is NIST’s emphasis on TEVV (Test, Evaluate, Verify and Validate). Other candidates you may often see sited as a show of assurance. yes I use the term ‘show’ as that is the truth behind what these actually offer in practice. Include, ISO/IEC 42001, which certifies the management system, but does not help to define if a particular model is safe or continues behaving acceptably. The often sited European Union (EU) AI Act, which is primarily a regulatory compliance regime rather than an operational assurance methodology. Then there are the sector regulation attempts (FDA, MHRA, aviation, etc.) which are evolving in a fragmented way because adaptive AI behaves differently from traditional medical devices or software.

Traditional assurance looks something like this:

Requirements > Design > Code Review > Testing > Deployment

Emerging AI assurance looks more like:

Governance > Training > Model Evaluation > Red Teaming > Validation > Deployment > Continuous Monitoring

However for high consequence sectors like healthcare, aviation etc, I believe the lifecycle ultimately needs to become:

Governance > Training > Model Evaluation > Validation > Authorisation > Continuous Assurance > Re-authorise  > Continuous Monitoring

When I reference re-authorisation I mean Zero Trust class re-authorisation, that becomes conditional rather than permanent. The whole process is analogous to aviation where aircraft are not certified once and forgotten. They undergo continuous inspection, maintenance, incident reporting and periodic recertification because the object of assurance is not simply how the aircraft was built but whether it remains airworthy.

My point is that currently regulatory confidence can no longer depend solely upon secure development lifecycles, code reviews, functional testing and the period pen-test. These remain necessary but they are no longer sufficient. They must be complemented by evidence that demonstrates how the trained model behaves in practice, including continuous evaluation, independent behavioural testing, adversarial red teaming, interpretability research where appropriate, operational monitoring and ongoing revalidation as models are updated or retrained.  This represents a profound change in regulatory philosophy. The technology is evolving at machine speed while our regulatory machinery still operates at committee speed. Traditional software regulation certifies that software was built correctly. AI regulation must increasingly certify that delegated decision making remains trustworthy throughout its operational life.

The implications extend well beyond healthcare. Critical national infrastructure, financial services, aviation, autonomous transport and defence are all beginning to delegate decisions that were previously reserved for humans. In these sectors, assurance can no longer end when the software is released. It must continue for as long as the AI exercises authority.

The future challenge is therefore not simply regulating AI technology. It is establishing independent mechanisms capable of continuously assuring the behaviour of systems entrusted with decisions that affect lives, livelihoods and national resilience. Can we continuously produce independent evidence that any AI remains trustworthy to exercise delegated authority today?

It took decades for software engineering to develop anything approaching mature assurance and even now we patch, update and apologise with admirable regularity (every Tuesday has become a popular cadence), AI offers no such luxury. Closed weighted models evolve in months, capabilities in weeks and risks sometimes overnight, open weighted models can take on a life of their own. Attempting to regulate AI using the tempo of 20th century software assurance is rather like bringing a sundial to a Formula 1 pit lane. By the time we’ve agreed the framework, the technology will already be assuring us that everything is perfectly under control.