How to Prove Your New Model Isn't Worse — Without Paying for Labels

Every model in production gets updated. You retrain it, fine-tune it, compress it to run cheaper, or your vendor quietly swaps it for a new version. Each update is a gamble. The new model might be better. It might be worse. And "worse" in a medical triage system or a fraud detector is not an inconvenience — it's harm to real people.

So here is the practical problem. You have Model A running. Someone hands you Model B and says "this is an upgrade." How do you prove B is not worse than A before you let it touch users? The obvious answer — run both on a big labeled test set and compare accuracy — is expensive. A "label" is the ground-truth correct answer for an input, and those come from human experts: a doctor confirming the diagnosis, an analyst confirming the fraud. For a strong guarantee you need many of them, every single time you update. Do this weekly and you go broke.

A protocol called DISCERN gets the same guarantee for a fraction of the labels. It rests on one clean observation.

The insight: the difference lives only where they disagree

Take any input where Model A and Model B produce the same answer. On that input, whatever error one makes, the other makes too. They contribute nothing to the difference in error rates. They cancel exactly.

So the gap in performance between two models is determined entirely by the inputs where they disagree.

Now the consequence that makes this pay. You care about one number: how much worse B is than A — the difference in their error rates. Write that difference as a sum over all inputs. Every agreement contributes zero. So the whole difference is carried by the disagreement set. And you can find that set with no labels at all. Run both models on your ordinary incoming traffic. Wherever the two outputs differ, flag it. Spotting a disagreement needs no expert — you just compare two outputs. The expensive step, labeling, is now only ever needed on disagreements. Everything else is settled for free.

The worst-case bound: why a quiet update needs zero labels

Here is the piece that lets you certify some updates without labeling anything. Suppose A and B disagree on a fraction rho of traffic. On the agreements, the error difference is zero. On the disagreements, you don't yet know who's right — but you know the damage is bounded. In the absolute worst case, B is wrong on every single disagreement while A is right on every one. Even then, B's error rate can exceed A's by at most rho.

That's the lever. If rho is smaller than the harm you're willing to tolerate, you're done — even the worst case is acceptable, and you spent nothing. A near-identical update (say a compression that changes 2% of outputs) clears the bar for free. In their tests, 56% of harmless updates certified this way with zero labels.

Two tiers, spending labels only when forced

DISCERN turns this into a sequential procedure with two gears.

Tier one, zero labels. Watch unlabeled traffic, count the disagreement rate rho. If worst-case harm (bounded by rho) is already inside tolerance, certify B and stop.

Tier two, audited. If rho is large enough that the worst case would be unacceptable, you can no longer wave it through — you need to know how often B actually loses on the disagreements. So you start labeling, but only sampled disagreements, never agreements. Each label tells you, on a case where they differed, which model was actually right (against whatever metric defines "worse" — misdiagnosis, missed fraud). You accumulate evidence until you can either promote B or reject it.

The part that makes it trustworthy: anytime-valid evidence

Standard statistics has a landmine called "peeking." A normal test earns its error guarantee for one pre-committed sample size. If you instead keep checking as data arrives and stop the moment it looks convincing, you've cheated: the more times you look, the more chances a random fluke crosses the line by luck. Your real false-alarm rate balloons past the 5% you thought you had. This is a major reason published findings fail to replicate.

DISCERN uses a confidence sequence — a bound designed to hold at every moment you might stop, all at once, not at a single planned endpoint. The construction pays for this upfront: it widens the interval just enough that the probability of it ever being wrong across the entire unbounded run stays under your budget. So you can peek constantly, stop whenever the evidence is strong, and the guarantee still holds. It even holds if the process choosing which cases to label is adversarial — actively trying to fool you — which matters because real-world sampling is never clean.

And the guarantees compose. You can run an unbounded sequence of updates — promotion after promotion, month after month — all drawn from one fixed error budget. The audit doesn't decay as you keep shipping.

Why the savings are provably large

A naive auditor, blind to the pairing, samples random inputs and labels them — most of which are agreements that carry zero information about the difference. DISCERN labels only disagreements, so every label it buys is informative.

The proven label cost scales as rho²/eps², where eps is the tolerance you set — the harm gap you're trying to rule out. Read that formula as two knobs. The rho² term: rarer disagreements mean fewer informative cases exist and the worst-case harm is already smaller, so you need far less evidence — cost falls fast as models converge. The 1/eps² term: a coarser tolerance (you only care about big regressions) is cheap; demanding to detect tiny ones is expensive, the usual price of statistical precision. Against a naive auditor the improvement is a factor of 1/rho fewer labels: agree 95% of the time and you save roughly twenty-fold. And this is a matched bound — provably near the best any method can do, not a heuristic that happens to work.

The empirical run backs it: 14,000+ replayed audit streams, 785 real update pairs including fine-tunes of language models up to 1.4 billion parameters. Target false-alarm rate 5%; measured miscoverage 0.0002 — far safer than promised. It caught 98.6% of genuine regressions with zero false alarms. And every audit emits a machine-checkable evidence record, so a regulator or downstream team can verify the verdict rather than trust it.

The portable idea

Strip away the machine learning and you keep a thinking tool worth carrying everywhere:

To compare two options, spend your expensive measurement only where they differ. Everywhere they agree, the comparison is already settled — and the total gap can never exceed how often they differ.

That second clause is the part people miss. Agreement isn't just uninformative; it caps the possible difference, which means a small disagreement set lets you bound the answer before measuring anything.

This reaches far past models. Comparing two medical treatments? Patients who'd respond identically to both tell you nothing — the signal is where the treatments diverge. Two accounting systems, two policies, two suppliers, two drafts of a contract? Don't re-examine the identical parts. Route your scarce, costly attention — expert time, lab tests, careful review — to the disagreements, and bound the rest by how small that set is.

Most comparison work wastes effort re-confirming things both options already agree on. The move is to notice that agreement is self-certifying, and disagreement is the only place worth paying to look.

Distilled from arXiv Machine Learning