When Sellers Warn You Their Product Could End the World

When the people selling a product tell you it might end the world, your instinct is to look for the angle. Talk up how dangerous the thing is, and somehow that sells more of it. AI labs like OpenAI and Anthropic have warned for years that their systems could be existentially dangerous — lately loud enough to suggest labs should coordinate to slow themselves down. The natural response: this must be a plot. The trouble is that the plot theory, tested against evidence, falls apart. Understanding why it falls apart hands you a reusable tool for judging anyone's motives.

The three cynical stories

Hype. If the AI is smart enough to kill everyone, it's smart enough to run your sales department. Doom talk is exotic bragging.

Distraction. Every column about human extinction is a column not written about plagiarism, copyright, or data-center water use. The far-future scare buries near-term harms.

Regulatory capture. The most popular. Regulatory capture — a term from economist George Stigler's 1971 work on regulation — is when an industry gets regulators to write rules that entrench incumbents and lock out rivals. If the product is "too dangerous for amateurs," only a few big, connected firms can be trusted to build it. Safety talk, on this view, is lobbying in a lab coat.

Each is plausible as a story. The question is whether any survives contact with who is warning, and when.

The model: a falsification test for motives

Here is the established idea doing the work, named plainly. The philosopher Karl Popper argued that you don't confirm a claim by collecting examples that fit it — you test it by asking what evidence would prove it false, then looking for that evidence. A motive accusation is a causal claim: financial interest is causing the worry. And causal claims live or die by correlation and confounding — the basic logic of statistics. If X causes Y, then where you remove X, Y should weaken or vanish. If Y shows up just as strongly without X, then X was never the cause; something else (a confounder) explains both.

So the falsification test is one question: if the cynical story were true, what pattern should the evidence show — and does it? A motive only some people have should produce worry only in those people. Map the worry against the incentive.

Independent experts are just as alarmed. In a 2024 survey of 2,778 AI researchers who had published at major conferences — mostly academics with no company stake — a large share put the odds of extremely bad outcomes, including human extinction, at 5% or more. Geoffrey Hinton and Yoshua Bengio, two of the three "godfathers" of deep learning, turned to full-time warning without running big labs. The third, Yann LeCun, runs a major AI operation and dismisses the risk. The worry doesn't track the paycheck — if anything it runs backwards.

The warnings predate the companies. The decisive point. Sam Altman called machine superintelligence "probably the greatest threat to the continued existence of humanity" in February 2015 — before OpenAI existed. Dario Amodei was arguing about these risks before he worked in tech at all. A company cannot be a conspiracy to promote a belief its founders held before the company, the product, or the industry existed. The cause can't come after the effect.

People quit, at great cost, to keep saying it. Daniel Kokotajlo walked away from roughly $2 million in equity rather than sign a non-disparagement clause, so he could keep warning. Others have left labs forfeiting large sums because they thought the work was too dangerous. Leaving the company does not reduce the worry — the exact opposite of what "it's all a sales pitch" predicts.

Put the three together and the correlation the conspiracy requires isn't there. Insiders, outsiders, and ex-insiders worry at roughly the same rate. The belief floats free of the financial incentive. The motive theory fails its own test.

The other view, taken seriously

None of this shows the worry is correct. Smart people have talked themselves into sincere apocalypses before. Once a belief defines a professional community, groupthink and shared assumptions can keep it alive regardless of reality. "They genuinely believe it" and "they're right" are different claims, and only the first is defended here.

A softer version of the capture worry also survives. A founder can sincerely fear catastrophe and notice that safety rules would conveniently kneecap competitors; sincerity and self-interest coexist in most people. What the evidence rules out is the strong claim — that the fear was manufactured to serve the business. It wasn't manufactured; it came first. But watch for safety proposals quietly shaped to favour incumbents, because the soft version predicts exactly that.

There's also a coherence problem for the conspiracy theory. "Please regulate us, slow us down, and tell the public our product might kill them" is close to the worst marketing plan ever devised. A real scheme would be smarter than the thing in front of us.

The transferable move

The useful idea isn't about AI. It's a rule for reading motives: a motive explanation has to match the pattern the motive would produce — so test where the motive is absent.

The mirror case makes it concrete. When internal tobacco-industry documents surfaced in the 1990s, they showed company scientists privately acknowledging cancer risk while the firms publicly denied it — meaning employees-with-the-motive said one thing and independent scientists said another. The worry did correlate with the interest. The cynicism was earned. The AI case is the reverse: people with the supposed motive and people without it say the same thing, and many said it before they had any motive at all.

Carry one question into every "they only say that because…" argument: Does the worry correlate with the interest, or not? If people with no stake believe it just as strongly, and believers held the belief before they had a stake, the motive story is dead — whatever you conclude about the belief itself.

What to watch

The real disagreement isn't about hidden motives; it's about whether the alarm is justified. Two signals separate sincere-and-right from sincere-and-wrong. First, whether independent academic estimates of catastrophic risk keep clustering well above zero as the field matures, or drift down as systems prove more controllable than feared — that tells you about the belief. Second, whether labs ever accept safety rules that genuinely cost them against rivals, rather than rules shaped to favour incumbents — that tells you how much self-interest is mixed in.

Expectation: ~85% that over the next two years, surveys of unaffiliated AI researchers continue to show a median non-trivial probability (above 5%) of catastrophic outcomes — meaning the worry stays decoupled from corporate incentive, whatever its ultimate accuracy.

Distilled from Astral Codex Ten