Your AI Risk Beliefs Are Probably Incoherent — Here's How to Find Out

Most people who hold strong opinions about AI existential risk have never actually mapped where those opinions come from. They know what they believe. They don't know why they believe it, how confident they should be, or which single fact would flip their view. This is the difference between having a position and having a model — and almost everyone in AI risk discourse has the former while claiming the latter.

The Core Insight: Confidence and Justification Are Different Axes

You can be highly confident in a belief you've never examined (inherited from a trusted source) or deeply uncertain about a belief you've researched extensively (because the evidence itself is ambiguous). Most people conflate these. An epistemic audit forces them apart.

The question isn't "do you believe AI poses existential risk?" It's a chain of separable sub-questions:

1. Will sufficiently capable systems get built?
2. Will they pursue goals misaligned with human survival?
3. Will misalignment translate into catastrophic power?
4. Will safeguards fail to catch this in time?

Your overall p(doom) is a downstream product of your credences on each link. If you can't state your confidence on each link separately, your headline number is decoration, not analysis.

The Framework: Audit, Route, Structure

Audit yourself. For each node in the causal chain, don't just record your belief — record your epistemic status: Is this a settled view backed by evidence you've personally examined? A borrowed take from a source you trust? Or genuinely unexamined territory? This third category is usually larger than people expect, and it's usually hiding under the first.

Route your effort. High personal uncertainty + high leverage on your overall view = your next research priority. This is just expected-value reasoning applied to your own learning. Stop reading whatever's trending. Read whatever would actually move the node in your model that's both uncertain and load-bearing. If an entire field is stuck at high uncertainty on some link, that's not a reason to guess — it's a flag marking where the next essay, literature review, or research agenda should go.

Structure discussion. Most AI risk debates fail because two people are arguing about different links in the chain while believing they disagree about the conclusion. One person doubts capability timelines; the other accepts timelines but doubts the "misalignment → catastrophe" jump. Without decomposition, they talk past each other indefinitely. A shared structure turns debate into location-finding: which node do we actually disagree on?

Why This Matters Beyond AI Risk

This is a general antidote to a specific failure mode: mistaking social consensus for personal justification. Any domain with a long causal chain and high stakes — pandemics, climate tipping points, nuclear risk — rewards the same discipline. The audit is a template for epistemic honesty under uncertainty, not just an AI-risk tool.

Do This Now

Take your current p(doom), or whatever complex belief you hold with unexamined confidence, and force it through three questions:

• What's the causal chain underlying this number?
• For each link, is my confidence earned or borrowed?
• Which single link, if resolved, would move my overall estimate the most?

If you can't answer the third question, you don't have a model. You have a vibe wearing a probability's clothes.

Sources & Further Reading

https://www.lesswrong.com/posts/aHJ8MkPTWPDnuFLRL/an-epistemic-audit-for-existential-risks-from-ai