The Hidden Unity Behind Bayesian Statistics: Why Generating Functions Are Everything

Here's the insight that reframes singular learning theory: free energy, generalization loss, and WAIC aren't separate quantities you memorize — they're all shadows cast by the same object. That object is a cumulant generating function. Once you see this, the notoriously unintuitive WAIC formula stops looking like a magic trick and starts looking inevitable.

The Core Move: Stop Studying Observables. Study Their Generator.

Most treatments of Bayesian asymptotics present free energy, losses, and WAIC as a list of definitions to be individually derived and memorized. This is backwards. It's like studying the positions of planets without learning that gravity produces all of them.

The actual structure:

1. Define the relationship between a true distribution, a statistical model, and a prior. This triple is your entire universe — everything downstream is a consequence of how the model can (or cannot) represent the truth.
2. Define the observables — the raw and normalized versions of the Bayesian quantities you care about.
3. Construct the cumulant generating function of the Bayesian predictive distribution.
4. Differentiate. The basic theory of Bayesian statistics — the exact relationships between free energy, loss, and WAIC — falls out as derivatives and expansions of this single function.

This is the generating-function mindset: instead of computing each moment or observable by brute force, you build one object whose derivatives are the observables. The hard work goes into constructing the generator once; everything else is calculus.

Why This Matters for Your Own Thinking

This isn't just a technical trick for singular learning theory — it's a transferable mental model:

The Generator Heuristic: When you find yourself with a family of related quantities that all feel connected but whose relationships are opaque, ask: is there a single function whose transformations produce all of these as special cases? Moment-generating functions, partition functions in statistical mechanics, characteristic functions in probability — this pattern recurs because it works. Complexity in the outputs often hides simplicity in the generator.

Practical implication: If you're staring at a formula (like WAIC) that seems arbitrary, don't accept it as a black box. Ask what it's a derivative of. The formula usually becomes obvious in hindsight once you find the generating object — and your intuition about why it behaves the way it does under different priors or model singularities becomes far sharper.

The Payoff for Singular Models

In regular (non-singular) statistical models, this generating-function machinery reduces to classical asymptotics you already know (AIC, BIC). The reason singular learning theory needs this heavier framework is that singular models — where the map from parameters to distributions is not one-to-one — break the classical Gaussian approximations. The generating function approach doesn't care: it still cleanly produces the free energy, loss, and WAIC asymptotics, even when the geometry underlying the model is singular and classical tools fail silently.

Takeaway

Before you memorize another formula in Bayesian statistics, ask what generating function it's hiding. The relationships between free energy, generalization error, and WAIC aren't coincidences to be memorized — they're derivatives of one unifying object. Learn to build that object, and the rest becomes calculus, not memorization.

Sources & Further Reading

https://www.lesswrong.com/posts/ZmHsfaG8YtvMaa4Mm/singular-learning-theory-comprehensive-2-1