He Built the Warning. Nobody Listened.

Joseph Weizenbaum did not become a dissident by choice. He became one because he watched his own creation get used as evidence for a conclusion he had explicitly argued against — and then spent the next thirty years unable to stop it.

Most people who know the name ELIZA know the broad outline: MIT, 1966, a chatbot that mimicked a therapist, users who got attached. What the textbooks tend to skip is what happened to Weizenbaum after that. What it cost him. And why the people who should have taken his warning seriously decided, collectively, that he had simply lost his nerve.

The moment that changed Weizenbaum's understanding of what he had built did not happen in a lab. It happened in his office, with a person he knew well.

His secretary was not a naive user. She was a highly educated professional who had watched Weizenbaum assemble ELIZA line by line. She understood what the program was: a script that matched patterns in text and rearranged them into questions. There was no mystery about the mechanism. She had seen behind the curtain. And then one afternoon she asked Weizenbaum to leave the room so she could speak with it privately.

Weizenbaum wrote about this moment with something close to genuine disturbance — not amusement, not scientific curiosity, but a discomfort he could not fully resolve. The program had not deceived her. She had, in some sense, chosen the illusion with full information. That distinction mattered to him enormously. It meant the effect was not a bug in human cognition that better education could fix. It was something more fundamental about how people construct the experience of being understood.

He named the phenomenon the ELIZA Effect: the tendency to attribute understanding to a system that is merely performing the surface behaviours associated with understanding. He intended it as a technical term with moral weight. A warning label on the tin.

The warning did not land as a warning.

By the late 1960s, other researchers were proposing that ELIZA-style programs be deployed as low-cost psychiatric therapy — a way to extend mental health services to patients who couldn't afford or access a human clinician. Weizenbaum was appalled. In his view, this was not a question of technical readiness. It was a category error. Therapy, he argued, required a human being capable of genuine moral responsibility for another person. A program that reflected words back was not a therapist with limited capability. It was something categorically different, wearing a therapist's costume.

His colleagues largely didn't agree. Some thought he was being precious. Some thought he had simply grown uncomfortable with the implications of his own work. The informal verdict inside the field was that Weizenbaum had gone soft — that the man who had built the most famous demonstration of human-computer interaction had developed a philosophical objection to human-computer interaction. It was easier to read him that way than to take the argument seriously.

He published Computer Power and Human Reason in 1976, a full-length moral broadside against the direction the field was taking. It was reviewed respectfully and largely set aside. He spent the final decades of his career at MIT as a kind of internal exile — present, vocal, cited occasionally, never quite central. He died in 2008. The programs he had warned about were by then embedded in customer service systems on every continent.

While Weizenbaum was losing his argument with the field, a different dynamic was quietly shaping what the field said in public.

DARPA — the Defense Advanced Research Projects Agency — had become the primary funder of American AI research by the mid-1960s, allocating millions of dollars annually to university laboratories on the stated basis that strong AI was not merely possible but imminent. The British Science Research Council was funding parallel efforts. Both were government bodies with limited technical expertise and a strong preference for confident predictions over measured ones.

Researchers who wanted continued funding had a structural incentive to make bold claims. Not dishonest claims, necessarily — most of the researchers believed what they were saying — but claims calibrated to what funders wanted to hear. The feedback loop was quiet and powerful: laboratories that promised breakthroughs got grants; laboratories that described incremental progress toward an uncertain goal got less. The "promise culture" the field developed wasn't a conspiracy. It was an emergent property of how science gets funded when the science is new enough that no one outside it can evaluate the claims.

The predictions that Minsky and Simon made in the 1960s — that machines would be doing any human job within twenty years, that the problem of AI was nearly solved — were not outlier positions. They were the ambient temperature of the field. Grant applications were written at that temperature. Progress reports were written at that temperature. Nobody sat in a room and decided to oversell. The overselling was structural.

Sir James Lighthill had nothing to defend.

He was an applied mathematician at Cambridge — one of the most eminent scientists in Britain, a Fellow of the Royal Society, a man whose reputation rested entirely on fields other than AI. When the British government asked him in 1972 to review the state of AI research and advise on funding, he approached the question the way an auditor approaches a set of books: with no prior investment in what he would find.

What he found was a gap. Not fraud, not incompetence, but a persistent and systematic mismatch between the domains in which the programs worked and the domains in which they had been claimed to work. The programs succeeded in what he called toy domains — bounded, simplified environments constructed specifically to make success possible. Real-world problems were messier, less forgiving, and combinatorially larger in ways the programs had never been tested against.

His 1973 report was published. The Science Research Council read it. Funding was cut substantially, and in some areas eliminated entirely. The response from inside the field was fierce — several prominent researchers published rebuttals — but the money did not come back. In the United States, DARPA was arriving at similar conclusions through its own channels. The formal American cuts came slightly later, but the pattern was the same.

The researchers who had been quietly raising doubts through the late 1960s — and there were some, working carefully at the margins — found that their caution had been more right than they had been given credit for. They did not receive much credit then, either. The field moved on.

The Optimism Era ended with a funding collapse. But the two things it produced — the ELIZA Effect and the toy domain problem — did not end. They travelled forward.

Every AI cycle since has contained a version of the same dynamic: genuine, impressive results in bounded domains; predictions about the general case that outrun the evidence; a reckoning when the gap becomes visible. The researchers were not uniquely reckless. They were working at the frontier of a new field with incomplete information and structural incentives toward optimism. That is not a historical aberration. It is a description of how new fields behave.

What Weizenbaum understood — and what the field spent decades not wanting to hear — is that the surface behaviour of intelligence and the substance of intelligence are not the same thing, and that humans are constitutionally bad at telling them apart. He wasn't arguing that the programs were worthless. He was arguing that mistaking fluency for comprehension was a specific, repeatable error with real consequences. He was right. He was right in 1966, and the error he described is the most important diagnostic tool available for understanding what is happening with AI today.

This post gives you the human story underneath the era: the man who built the warning, the culture that ignored it, the outsider who finally said what needed saying. What the video does is show you the mechanism — why the ELIZA Effect still operates on people who know exactly what a language model is, and how to use the toy domain problem as a practical test for evaluating AI claims you'll encounter this week.

If you've read this far, you have the context to get considerably more from the video than someone coming in cold.

The programs worked. The predictions got bolder. And then a government report cut the funding and the first AI summer ended. The field had forgotten to check whether the optimism was warranted.

Sound familiar?