How to Make an AI Model Faster Without Retraining It: Token Merging

When a machine transcribes speech, it does not process the recording all at once. It chops the audio into a long sequence of small time-slices, converts each slice into a list of numbers — a token, the model's internal representation of a moment of sound — and then runs the step that makes modern AI work: every token "looks at" every other token to decide what was said. This looking-at-everything is called attention. It is what lets the model use context — the fact that a muffled syllable is a "t" and not a "d" often only becomes clear from the words around it.

Attention is also the expensive step, and it is expensive in a specific, punishing way. Each token is compared against every other token, so the work grows with the square of the sequence length. Double the tokens and you roughly quadruple the compute. Long audio produces a long token sequence, and a long token sequence makes the model slow and costly to run.

Whisper, OpenAI's widely used speech model, shows both the value and the pain. It transcribes dozens of languages — including rare, low-resource ones with little training data — without separate training for each. That is genuinely useful. But it is heavy. Running it at scale, or on a phone, hurts.

The obvious fixes are bad. Shrink the model and it transcribes worse. Retrain it and you need money and data you may not have — especially for a language spoken by a few million people. The paper takes a third route, and the route is the point.

The core idea: neighbouring tokens are often near-duplicates

In a stretch of audio, many adjacent slices carry almost the same information. A held vowel, a pause, a stretch of steady background noise — the model dutifully makes a separate token for each slice, then burns attention comparing tokens that say nearly the same thing. That effort is redundant.

Token merging removes it. Here is the actual mechanism, because this is the part you want to be able to explain.

Inside the model, each token is already a vector — a point in a high-dimensional space where "similar sound" means "nearby point." Merging works by measuring similarity directly: it takes the tokens' vectors and computes how close each is to its neighbours (a standard closeness measure between vectors). Where two neighbouring tokens are highly similar, it fuses them — literally averaging their vectors into one token — and drops one from the sequence. It also records which tokens were merged. The shortened sequence flows through the expensive attention layers. Then, before the model emits its final transcript, the merged token is "unmerged" — its result is copied back out to each of the original positions it stood in for. So the output stays aligned to the original audio; you only paid attention's quadratic cost on the shorter sequence.

The critical feature: no retraining. You take an already-trained model and insert the merging step at inference time — while it runs, not while it learns. It is like handing an editor a finished document and telling them to collapse repeated paragraphs before proofreading, then restore them afterward: less to check, same final text.

Why this could have failed, and why it didn't

The obvious fear: throw away tokens, throw away information, lose accuracy. For rare languages that fear is sharper, because the model already has so little to work with.

What makes the paper useful rather than a one-off trick is that it tests this systematically — across sixteen languages, three model sizes, and even on models already fine-tuned for specific low-resource languages. (Fine-tuning here uses DoRA, a cheap way to adapt a large model by adjusting a small slice of its numbers rather than all of them — so you can specialise a model without retraining the whole thing.)

Three findings worth carrying:

1. Accuracy barely moves. Across most low-resource languages and model sizes, merging sped the model up with almost no loss in transcription quality. The redundancy really was redundant — the merged tokens weren't carrying distinct information.
2. It stacks with fine-tuning. Merging still worked after a model had been adapted to a language. The two efficiency tricks operate on different things — one removes redundant computation, the other adjusts what the model knows — so they don't interfere.
3. The gains hold across scales. It helped small and large models alike, so you are not forced to trade "fast" against "accurate."

The portable idea

Strip away the speech and a general principle remains:

Much of the cost in a system comes from processing redundancy the system generated itself. You can often cut that cost after the fact, without rebuilding anything, by collapsing near-duplicates and restoring them at the end.

The trick needs three ingredients. First, a cost that grows faster than linearly with the number of items — here, comparisons growing with the square of the sequence length. That superlinear growth is what makes shrinking the input pay off disproportionately: cut the sequence by a third and you cut the work by more than a third. Second, genuine redundancy among the items. Third, a cheap way to detect and merge duplicates without touching the underlying machine.

That pattern is everywhere once you look for the superlinear cost. A database join whose cost explodes with the number of rows can pre-collapse identical rows, run once, and expand the result. An image model can fuse identical patches of sky before its attention layers compare every patch to every other. Any all-pairs computation — anything where cost scales with pairs, not items — is a candidate. The diagnostic question: where is my system paying the full quadratic price to process copies of the same thing?

The deployment lesson is the sharpest. The default assumption is that a cheaper AI model means a dumber one, or an expensive retraining job. This work opens a third door: leave the trained model untouched, and strip out the redundancy it generates while running. Before you pay to shrink or retrain, ask whether the model is simply doing the same work twice — and whether you can make it stop, for free, at inference time.

Distilled from arXiv NLP/LLMs