The AI Nuremberg Hypothesis – What Did You Do After You Knew?

Following on from my blog yesterday the comments on the Frontier AI Labs fear mongering by the US administration, particularly President Trump crystalised a thought that had been gnawing at me, there is something increasingly peculiar about this aspect of the AI debate. Some of the people building the most powerful technology humanity has created are simultaneously telling us that it might destroy us, yet they carry on building it.

This triggered one of those cognitive hookups, in which a memory came forward of my late brothers passion for World War 2 history and a set of the Nuremberg War Trial leather bound volumes ‘Trial of the Major War Criminals before the International Military Tribunal’, 42 in all, which were bequeathed to my sons. It got me considering an ‘AI Nuremberg Hypothesis’ emergent from a bisociative connection between the Nuremberg principles of individual responsibility and the increasingly explicit warnings coming from frontier AI labs.

The core theme being if you knowingly create, finance, authorise or fail to restrain something you believe could cause catastrophic harm to humanity, does warning everyone beforehand absolve you of responsibility? Or does the warning establish that you knew? I hinted at this in my last piece ‘The AI Risk Nobody Is Measuring – Authority‘.

Let me make on thing very clear at the start of this thought exercise, this is emphatically NOT equating today’s AI founders etc with the perpetrators of the Holocaust. That would trivialise an historical atrocity of almost unimaginable proportions. The parallel is instead with one of the enduring principles emerging from Nuremberg, that responsibility does not necessarily disappear into an organisation, hierarchy, state or machinery through which harm occurs.

The Nuremberg Principles subsequently formulated by the UN International Law Commission established individual responsibility under international law and, importantly, that acting under orders (that’s you developers, researchers and associated personnel) did not relieve an individual of responsibility where moral choice was possible.

If these narratives form the AI Frontier labs are proven true, even in part, AI may eventually confront us with a 21st Century variation of that problem.

Imagine the inquiry following some future AI catastrophe. Before it sit founders who created the systems, executives who commercialised them, investors who financed their expansion, governments that encouraged an AI arms race and institutions that watched the warnings accumulate. The documentary evidence could make extraordinary reading.

The researchers and developers warned that increasingly capable AI might become uncontrollable. They discussed catastrophic and existential risk. They called for regulation. Yet capability increased, billions of dollars poured in and deployment accelerated in release cycles that denied cross boarder scrutiny yet carried that risk. The inevitable question would be devastatingly simple, ‘If you genuinely believed this could happen, why did you continue?’  A defence of ‘I told you so‘ becomes more admission of knowledge and foreseeability if not of tacit guilt rather than any defence.

That becomes particularly interesting as frontier AI companies contemplate public markets with their heralded $trillion valuations. Risk disclosures may inform investors about uncertainty, but can a prospectus really transfer civilisation scale risk from those creating it to shareholders and ultimately society? Surely humanity cannot become the involuntary counterparty to an initial public offering (IPO) risk statement.

Investors deserve similar scrutiny, although with a much higher threshold. Simply owning shares cannot reasonably create responsibility for everything a company subsequently does. However what happens when sophisticated investors knowingly finance further capability after being specifically warned that catastrophic outcomes are plausible or safeguards inadequate? At some point the question changes from ‘what did you invest in?’ to ‘what did you know, what did you enable and what choices remained available?‘ Or more likely greed drives the decision and they all pile in regardless in the FOMO (fear of missing out).

Governments face an even harder question. Article 36 of Additional Protocol 1 to the Geneva Conventions (as I have referenced in earlier pieces) already requires states studying, developing, acquiring or adopting a new weapon, means or method of warfare to determine whether its employment would be prohibited by international law.

Back to the response from President Trump to the latest warnings from the AI laboratories makes this question more acute. Faced with industry leaders themselves arguing that development may need to slow, Trump has largely dismissed the catastrophic scenarios as exaggerated and returned instead to the geopolitical imperative of beating China; in effect, the race must continue because America must win it. That may be understandable as national strategy, but it is much harder to reconcile with precaution when those closest to the technology are simultaneously warning of potentially catastrophic consequences. More importantly for the AI Nuremberg Hypothesis, the warning has now travelled beyond the laboratory. Governments cannot subsequently argue that they were unaware of the claimed risks; they have been placed explicitly before political leaders and a conscious choice is being made about how to respond. History has repeatedly demonstrated the danger of allowing strategic necessity, economic advantage or obedience to institutional objectives to overwhelm individual judgement. If the laboratories are saying slow down and governments knowingly choose race faster, responsibility can no longer reside conveniently with Silicon Valley alone. The knowledge and therefore the accountability question, has crossed the threshold into the state.

It is worth noting that general purpose AI is not automatically a weapon and Article 36 should not be stretched beyond its legal purpose. However when frontier models migrate into offensive cyber operations, autonomous systems, targeting, intelligence and military decision making, the distinction becomes considerably harder to maintain as I have covered in earlier missives.

If a state knowingly turns commercial AI into a means or method of warfare, at what point does an Article 36 review become unavoidable?

Responsibility also does not simply stop at national borders. The UN General Assembly is already addressing safe, secure and trustworthy AI. The UN’s emerging international AI governance machinery exists precisely because these technologies transcend individual jurisdictions. The International Committee of the Red Cross has long promoted legal reviews of new weapons. The Convention on Certain Conventional Weapons has wrestled with lethal autonomous weapons. The G7 Hiroshima Process, OECD principles and now the EU AI Act all recognise, in different ways, that advanced AI requires governance, accountability and risk management. However, none currently creates a global criminal offence of recklessly developing frontier AI, nor should we pretend otherwise, but collectively they destroy another potential future defence ‘Nobody realised governance was necessary’ when we plainly do.

That last point creates perhaps the most uncomfortable element of the hypothesis. The founders build it, Corporations commercialise it, Capital scales it, Governments encourage, acquire and potentially weaponise it, International institutions debate how to control it and the the global population inherit the consequences, never authorising the experiment.

This is where I come back to the hypothesis at the top of this missive, the Nuremberg analogy matters, not because today’s AI entrepreneurs are equivalent to yesterday’s war criminals, but because Nuremberg demonstrated something much more enduring, extraordinary systems do not make individual responsibility disappear.

International law has repeatedly evolved after humanity discovered that existing rules were inadequate for previously unimaginable forms of harm. Nuremberg itself was part of that evolution, followed by the Genocide Convention, Geneva Conventions, Additional Protocols, international criminal tribunals and ultimately the International Criminal Court.

AI presents us with a rare opportunity to reverse that sequence. Instead of constructing accountability after catastrophe, perhaps we should determine it beforehand because if those developing AI genuinely believe even a fraction of the existential risks they increasingly describe, the defining question for founders, boards, investors, governments and international institutions is no longer simply whether they warned us, it is what they did after they knew.

Nuremberg asked who was responsible for what had happened. My AI Nuremberg Hypothesis asks something considerably more useful, ‘who will be responsible if we knowingly allow it to happen again?’