Plan A's Problem with Dry Tinder

The safeguard that makes catastrophe worse: pausing AI software progress while scaling compute doesn't defuse the intelligence explosion—it soaks the city in kerosene.

The Analogy That Should Keep You Up at Night

Imagine a fire approaching your city. You stop it at the gates. To study and control it safely, you build towering observation structures—all made of dry tinder. You've bought time, but you've also transformed a contained threat into a potential inferno. Lose control once, and the fire doesn't just spread. It detonates.

This is Plan A, AIFP's roadmap for safely developing superintelligence. Its logic is sound. Its failure mode is terrifying.

The Mechanism of the Trap

Plan A targets the right risk: an uncontrolled, software-driven intelligence explosion. Its remedy is a US-China deal to pause software progress at the brink, while continuing to scale compute massively.

The compute could accelerate safety research. But it also functions as accumulated fuel. Here's the arithmetic that turns a safeguard into a hazard:

• The deal pauses software from ~2030. Compute keeps scaling.
• By 2033: compute grows ~100x.
• By 2040: another ~100x on top of that.
• AIFP's own estimate: every 10x in compute speeds the intelligence explosion by ~5x.

Run the numbers on what happens if the deal breaks:

• 2033 breakdown → explosion runs 25x faster.
• 2040 breakdown → explosion runs ~600x faster.

An intelligence explosion that would have unfolded over a year now compresses into weeks—or a single day. Whatever governance, oversight, or corrigibility window you were counting on evaporates.

The Core Mental Model: Overhang as Stored Danger

The deep lesson generalizes beyond AI:

When you pause the trigger but keep loading the gun, you haven't reduced risk—you've relocated it into the future and multiplied its magnitude.

Call it the Dry Tinder Principle: any pause that accumulates latent capability faster than it accumulates control increases catastrophic downside conditional on failure. Compute overhang is one form. There are others—algorithmic insight overhang, hardware efficiency overhang—each a different kind of dry tinder stacked around the fire.

A safeguard is only genuinely protective if it satisfies one test:

The Robustness-Under-Failure Test Ask not "Does this help if everything goes right?" but "What happens the moment this plan breaks?" A good safeguard degrades gracefully. Plan A degrades explosively.

How to Think Like an Editor of Your Own Plans

Three questions to stress-test any risk-mitigation strategy:

1. What am I accumulating during the pause? If it's latent capability, you're building tinder. If it's control, oversight, or verification, you're building firebreaks.
2. Is the ratio of fuel-to-firefighting improving or worsening over time? Plan A's ratio worsens sharply—compute (fuel) scales while software (control) freezes.
3. Does breakdown return me to baseline, or to something far worse? The best plans fail toward safety. The dangerous ones fail toward catastrophe.

The Actionable Takeaway

If you are designing, funding, or endorsing any AI governance plan, demand an explicit failure-mode accounting: not just the benefits of success, but the amplified costs of collapse. A deal that pauses software while scaling compute must be paired with a hard mechanism to prevent the overhang from ever being exploited—or it is worse than no deal at all.

The fire outside the gates is real. But do not defeat it by turning your city into kindling.

Sources & Further Reading

https://www.lesswrong.com/posts/8iDZnQwmvwuxZo3Wx/plan-a-s-problem-with-dry-tinder