The Pygmalion Fallacy

Listening to the narratives from OpenAI CEO Sam Altman following the recent headlining incident (or canny marketing ploy!) I get a sense that like Pygmalion, the mythical sculptor who fell in love with the statue he had carved, today’s AI pioneers risk becoming captivated by their own creation. It begs the question why do we keep mistaking computation for consciousness?

The debate surrounding artificial intelligence (AI) has become dominated by such questions of capability. How powerful can models become? How quickly can they reason? Which benchmark have they surpassed this week? These are important questions but they increasingly distract us from the one that matters most … what does it actually mean to be intelligent?

The question itself is not new. Alan Turing first asked whether machines could think, we have been captivated by the question. Yet perhaps we have spent too little time asking whether we have first defined what thinking actually is. Then more than four decades ago, philosopher John Searle’s Chinese Room thought experiment challenged the assumption that manipulating symbols could ever amount to genuine understanding. Today, large language models (LLM) have become extraordinarily sophisticated symbol processors, yet the underlying philosophical question remains unresolved.

We have become remarkably good at debating the ethics of AI while avoiding the more uncomfortable question that precedes it; what do we actually mean by intelligence? before deciding what AI should do, surely we must first decide what it means to think, to understand and ultimately to be human.

As I touched on in an earlier piece from a different perspective, ‘Stop Calling It a Hallucination … the Myth that is AI’ The danger is compounded by the language we increasingly deploy. AI systems are said to think, understand, reason, learn, hallucinate and even want. These are not neutral technical descriptions. They are words borrowed from biology, neurology and psychology that subtly encourage us to attribute human qualities to statistical computation.

Yet the disanalogies between brains and computers are incomparably greater than the analogies. Human consciousness is not merely hosted by biology; it is constituted by it. Intelligence emerges from being an embodied organism, living through time and space, continuously shaped by memory, emotion, relationships, fear, desire, pain and mortality. An LLM processes mathematical relationships between symbols. However remarkable that achievement may be, it bears little resemblance to lived human consciousness.

There is no evidence that AI possesses subjective awareness beyond its own self-reports. To believe an LLM, or any size of model for that matter, is conscious because it claims to be is rather like believing a parrot understands the meaning of the words it repeats. Fluency is not consciousness. Problem solving, computation and information processing are not synonymous with intelligence. Computers do not think. Humans think, sometimes with the assistance of computers.

This distinction matters because words shape policy, investment and ultimately the delegation of authority. If we casually redefine intelligence to include statistical prediction, we risk granting machines legitimacy they have not earned on plagiarised human insight often without license from the creators. The future challenge is not regulating AI itself but deciding which forms of judgement, responsibility and moral agency should remain irreducibly human.

Perhaps Artificial Intelligence will come to be recognised as one of history’s most consequential misnomers. Not because the technology failed but because the name persuaded us to confuse imitation with understanding.

Therein lies a warning to humanity. If we continue down that path, history may record that whilst performing the most extraordinary act of self-deception ever witnessed; we spent decades insisting our machines were becoming human, only to discover we had quietly started behaving like the machines instead. We stopped valuing wisdom because prediction was faster, surrendered judgement because optimisation was cheaper and in the process outsourced responsibility because an algorithm looked more objective on a PowerPoint slide but more to the point we allowed our objectivity to atrophy. Pygmalion’s tragedy was falling in love with his own creation. Ours may be far worse, convincing ourselves that the statue was alive, then asking it what kind of civilisation we ought to become!