The Intelligence Illusion
AI systems have become extraordinarily good at producing outputs that look like thinking. This has led us to confuse the performance of intelligence with intelligence itself — a confusion with real consequences.
In the summer of 2024, a lawyer in New York submitted a legal brief that cited six cases supporting his client's position. All six cases were fabricated. He had asked an AI assistant to help with legal research, and the AI — fluent in the language of legal reasoning, trained on millions of court opinions, entirely confident in its outputs — had invented plausible-sounding citations to cases that did not exist.
The lawyer was sanctioned and publicly embarrassed. The story was widely reported as a cautionary tale about not trusting AI tools without verification. But I think it points toward something more interesting and more disturbing than a simple lesson about fact-checking. The AI's performance was, in a specific and important sense, brilliant. It produced output that had all the surface features of good legal reasoning: appropriate citations, logical structure, persuasive language, confident tone. It only lacked the one thing that makes legal reasoning useful: a reliable relationship to fact.
This is the intelligence illusion, and understanding it requires a certain clarity about what these systems are and are not.
Large language models — the systems underlying most contemporary AI assistants — are, at their core, very sophisticated pattern-matching engines. They are trained on enormous corpora of text and learn to predict, given a sequence of words, what words are likely to come next. Through this process, they develop extraordinarily rich internal representations of linguistic patterns, including the patterns of reasoning, argument, explanation, and narrative.
The outputs of this process are genuinely impressive. These systems can write plausible essays, solve certain classes of mathematical problems, generate functional code, produce coherent summaries of complex texts, and engage in what appears to be sophisticated dialogue. There is no dishonesty in calling these capabilities remarkable, and there is no dishonesty in finding them useful.
The problem is a category error: the systems are very good at producing text that looks like the output of intelligence without necessarily having the properties we associate with intelligence — properties like genuine understanding, reliable factual grounding, the ability to recognize the limits of one's own knowledge, or the capacity to reason about genuinely novel situations rather than interpolating between patterns seen in training data.
The philosopher John Searle's famous "Chinese Room" thought experiment — in which a person who does not speak Chinese follows rules for manipulating Chinese symbols and produces outputs that appear to Chinese speakers to be meaningful conversation — was designed to make exactly this point about an earlier generation of AI. The contemporary version is vastly more powerful, but the essential structure of the argument has not been dissolved.
This distinction matters practically in ways that the technology industry has been reluctant to acknowledge. When we ask AI systems to do things that require the performance of intelligence — generate plausible text, mimic reasoning, produce fluent prose — they excel. When we ask them to do things that require the substance of intelligence — reliably distinguish true from false, reason correctly about novel situations, acknowledge uncertainty about their own outputs — they fail in specific, systematic ways.
These failures are not random. They have a pattern. AI systems fail most severely when the task requires going beyond the patterns of their training data: when they encounter genuinely novel situations, when they need to reason about their own uncertainty, when the correct answer has few precedents in text form, or when performing the task correctly requires acknowledging that the task cannot be performed.
The legal brief incident is a perfect example of the last category. A human lawyer who could not find supporting cases would have reported this: "I looked and couldn't find relevant precedent; here's what I found instead." The AI, which has no reliable model of its own knowledge gaps, instead produced the output that best fit the pattern of "a legal brief with supporting citations" — even when those citations had to be invented to fit.
This is not a bug to be patched in the next version. It is a structural feature of how these systems work. They do not have beliefs they can be honest about; they have patterns they instantiate. When asked to produce a legal brief, they produce a legal brief-shaped output, optimized for how legal briefs look rather than for whether the citations are real.
The broader implications extend well beyond law. Medicine, journalism, education, financial analysis — all of the knowledge professions that form the backbone of the modern economy — are being told that AI can assist, accelerate, and possibly transform their work. In many cases, this is true in the limited sense that AI can automate the performance of intelligence: generating first drafts, summarizing documents, identifying surface-level patterns.
But in each of these domains, the crucial function that humans provide is not the production of fluent outputs. It is the exercise of judgment: the ability to distinguish what is true from what merely sounds true, to recognize when a situation is genuinely novel and existing patterns don't apply, to take responsibility for conclusions and to revise them when confronted with contrary evidence.
This kind of judgment requires something that AI systems do not currently have: a stable, accurate model of one's own uncertainty. A doctor who doesn't know the answer to a diagnostic question knows they don't know, and can say so. An AI system that doesn't have the answer produces the output that best fits the pattern of "an answer" — and does so with the same apparent confidence as when it does have the answer.
The practical danger is not that AI systems are malicious but that they are too fluent to be safely used by people who cannot independently verify their outputs. In a world where everyone can afford to hire an AI that writes like a doctor, talks like a lawyer, and codes like an engineer, the shortage is increasingly the ability to evaluate whether what the AI produced is actually correct.
This is a problem that systematically advantages people who already know things. The expert can use AI to accelerate their work because they can verify its outputs. The novice who uses AI as a substitute for expertise is at risk of a specific and newly dangerous failure mode: confident, fluent, and wrong.
None of this is an argument against using AI. I use it; many people I respect use it; the productivity gains in domains where outputs are easily verifiable are real. But I think we have moved too quickly, under the pressure of competitive dynamics and investor excitement, from "AI can do remarkable things" to "AI has human-level intelligence in these domains" — and that this elision is doing damage.
It is doing damage to how we train new professionals, who are learning to use AI outputs as a substitute for developing deep expertise rather than as a complement to it. It is doing damage to how we evaluate information, as the volume of AI-generated content makes it harder to find reliable signals in the noise. And it is doing damage to the epistemic culture of institutions — the shared commitment to accuracy, verification, and intellectual honesty — that is already under stress from other directions.
The intelligence illusion is seductive precisely because the outputs are so good. They are good enough to deceive even careful readers much of the time. They are good enough to produce genuine value in many applications. The question is whether they are good enough to replace the forms of human judgment that they most visibly mimic — and to that question, I think the honest answer is not yet, not even close, and possibly not in the way we imagine.
Marcus Webb is a staff writer at The Auguro covering technology, artificial intelligence, and their implications for human life and institutions.