Who Actually Gets Smarter When AI Helps Them Work?

Give a professional an AI assistant and they produce better work. That part is not in dispute. The harder question is what happens to the human underneath. Do they get better at the job itself, or do they quietly hollow out — producing good output today while losing the ability to produce it tomorrow on their own?

A new randomized controlled trial gives the sharpest answer yet, and the answer is uncomfortable: the people who gained the most durable skill from AI were the ones who needed it least.

The experiment

The setup was clean. Take 133 practicing patent lawyers across eleven law firms. Patent law is a good test case because the work is high-skill, judgment-heavy, and easy to score by other experts. Randomly split the lawyers into two groups. One group gets a custom AI drafting assistant for three months. The other does not. Everyone's work is graded blind — by expert patent attorneys who don't know who used AI.

"Randomized controlled trial" just means the group assignment was random, so any difference in outcomes can be credited to the AI rather than to which lawyers happened to be better already. Randomizing is what turns "AI users did well" into "AI caused them to do well."

Two things got measured, and the distinction between them is the whole point.

Performance with AI. How good is the work when the lawyer has the tool in hand?

Judgment without AI. After three months of AI-assisted practice, take the tool away and test the underlying skill. Here they had lawyers "redline" a patent application — mark up an existing document for errors and weaknesses. This is core expert work. No AI allowed. This measures what the human actually absorbed.

The results, in two halves

With AI, everyone improved. Work quality rose by about 0.34 standard deviations at ten days and 0.38 at ninety. (A standard deviation is a unit of spread; 0.3–0.4 SD is a solid, real improvement — enough to move someone from the middle of the pack toward the upper third.) Junior lawyers gained the most. This matches every other study of AI in white-collar work: the tool lifts output, and lifts the weakest workers most. So far, the optimistic story holds.

Then the tool came off — and the story flipped. On the no-AI judgment test, the AI-trained group still beat the control group by 0.32 SD. But that advantage came entirely from the senior lawyers, who gained 0.45 SD. The junior lawyers, who had shone brightest while using AI, showed no average gain in underlying skill at all.

Worse, their scores didn't just fail to rise — they split apart. Fewer mediocre scores, but more genuinely poor ones and more genuinely good ones. The AI didn't lift the juniors uniformly. It sorted them.

The mechanism: AI amplifies judgment it can't supply

Here is why this happens, and it is the portable idea worth carrying out of the paper.

An AI assistant produces plausible-looking output. To turn that output into real skill, you have to do something with it: notice where it's wrong, understand why the good version is good, reject the confident-but-flawed suggestion. All of that requires a mental model of what correct looks like — the exact thing expertise is.

A senior lawyer already has that model. So when the AI hands them a draft, they're running a continuous internal test: right, wrong, right, subtly wrong. Each judgment reinforces and sharpens the model. AI becomes a sparring partner.

A junior lawyer doesn't have the model yet. The AI's output looks correct because they can't see what's missing. So they absorb it wholesale — sometimes it's good and they look brilliant, sometimes it's flawed and they don't catch it. That's the bifurcation. Without a foundation to judge against, the AI isn't a teacher. It's a coin flip they can't referee.

The rule: AI multiplies the judgment you bring to it. It doesn't manufacture judgment you lack. If your internal model is strong, AI compounds it. If it's weak, AI can't tell you that — it just hands you output you're not equipped to evaluate, and your results scatter.

Where else this bites

This is not about patents. It's about any skill you're tempted to hand to a capable assistant before you've built the judgment to check it.

A student who uses AI to write essays before learning to structure an argument produces good essays and no writing ability. A junior programmer who lets AI generate code before understanding the logic ships working features and never learns to debug. An analyst who accepts AI's summary of the data before knowing what a suspicious number looks like — same trap. In each case the output is fine and the person is not developing. The competence lives in the tool, and it leaves when the tool does.

The same logic runs in reverse and it's good news for experts. If you already know a domain cold, AI is the best amplifier you've ever had, precisely because you can catch its mistakes and keep only its gold.

The one caveat that matters

A trial has to hold everything else fixed to isolate the AI's effect. That's its strength and its blind spot. In the real world, nothing stays fixed. Firms will notice which tasks juniors should still do by hand, and reassign work accordingly. People adapt their habits when the stakes are theirs. So the long-run picture may be brighter than three frozen months suggest — the allocation of humans to tasks evolves, and more people end up more productive, not fewer.

But the core lesson stands, and it's one to act on now: build the foundation before you outsource the practice. Use AI to sharpen judgment you already have, or to check work you could have done yourself. Be wary of using it to skip the phase where the judgment is actually formed. The tool rewards the expert and quietly abandons the novice — while making both of them look, for a while, exactly the same.

Distilled from Marginal Revolution