The Weird Math That Links Zipping Files to Training Language Models
Here's a strange fact from 2002: you can reconstruct the entire family tree of human languages — which ones are related, how closely — using nothing but gzip. The same boring compression tool you'd use to shrink a folder of vacation photos.
The trick works like this. Take a document in some language you don't recognize. Compress it. Note the file size. Now tack a small snippet from a second document onto the end of the first one, and compress that combined file. If the snippet slots in nicely — barely adding to the file size — it means the compressor, having learned the patterns of document one, already "knew" a lot of what it needed to handle document two. That's your signal: these languages share deep structure. If the snippet bloats the file, the two languages are strangers to each other. Do this pairwise across a pile of documents, and the resulting web of "how well do you compress in my shoes" distances is enough to redraw the actual tree of linguistic ancestry — no linguist required, no dictionaries, no grammar rules. Just file size.
What's actually happening under the hood has a name: cross-entropy. And the wild part is that this same quantity, cross-entropy, sits at the mathematical core of how today's language models are trained. Same formula, two completely different-looking jobs. When math does that — hands you an identical expression in two unrelated corners of a problem — it's not a coincidence. It's a hint that something deeper connects them.
To see why, it helps to start somewhere much simpler than gzip or GPT: a robot.
Imagine you're sending movement instructions to a robot on a mission — up, down, left, right — and you know in advance that "up" gets sent half the time, "down" a quarter of the time, and "left" and "right" each an eighth of the time. If you're clever about it, you don't waste the same number of bits on every instruction. You give the frequent one, "up," a short one-bit code. You give "down" two bits. You give the rare ones, "left" and "right," three bits each. Common things get short codes, rare things get long ones — this is just how Morse code and Huffman coding and basically all compression works.
There's a tidy formula lurking behind this: the ideal number of bits for a symbol is the negative log (base 2) of its probability. Rare symbols have small probabilities, and negative-log-of-small-number is big — so rare symbols earn long codes. Frequent symbols get short ones. Shannon called this quantity the "information content" of an event, though honestly it's easier to think of it as "how many times do you have to cut something in half to shrink it down to this probability."
Now here's the twist that actually matters. Suppose mission control changes the plan. The robot should now head right more often — "right" jumps to a coin-flip's likelihood, "left" drops to one-in-four, "up" and "down" fall to one-in-eight each. But nobody updates the code book. You're still using the old one-bit-for-up, two-bit-for-down encoding, even though the world has changed.
How wasteful is that, exactly? You can just do the arithmetic. Weight each old code length by its new frequency, and add it up. You get 2.625 bits per instruction on average — worse than the 1.75 bits you'd get if the code had been built for the new distribution from the start.
That number, 2.625, is cross-entropy. It's the answer to the question: if I built my compression scheme around one distribution of reality, and reality turns out to be a different distribution, how many bits am I now stuck paying? It's the cost of a mismatch between your expectations and the truth — measured in bits.
Squint, and you can see gzip's language tree trick doing exactly this. A compressor trained on document A has, in effect, built a code book optimized for language A's patterns. Feed it a snippet of document B, and the extra bits it needs are precisely a cross-entropy measurement — the tax you pay when your assumptions (Q) meet a reality (P) they weren't built for. Small tax, close relatives. Big tax, distant cousins.
And this is the same quantity — dressed differently — that shows up when a language model is trained to predict the next word. At every step, the model has some internal guess about what's coming next (its Q), and the actual next word arrives from the real distribution of language (its P). Training is, quite literally, the process of shrinking that cross-entropy — of building better and better code books for reality.
Which means next-token prediction was never really the deep story. The deep story is compression. Predicting the next word well and compressing language well are, mathematically, the same act wearing two different hats — and once you notice the formula winking at you from both directions, you can't really unsee it.
Distilled from MinutePhysics (YouTube)