Qurated: AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
AgentLens: Why Your Coding Agent's "Pass Rate" Is Lying to You
The single most important insight: A binary pass/fail score tells you whether an agent succeeded — but users don't experience the outcome, they experience the journey. The path is the product.
Most benchmarks collapse an entire agent run into one bit. AgentLens argues that bit throws away everything that matters: how the agent followed instructions, wielded its tools, verified its own work, recovered from mistakes, and communicated while doing it. Two agents can both "pass" a task — one gracefully, one by flailing through five wrong turns and a confusing monologue. Your users feel that difference. Your metrics don't.
The Mental Model: Score the Trajectory, Not the Destination
Think of it as the difference between a verdict and a transcript.
- A verdict ranks models on a leaderboard.
- A transcript tells you why — and lets you fix things.
AgentLens combines both:
- Formal verification — where an objective check exists (did the tests pass?), use it.
- LLM-written trajectory reviews — where they don't, generate a readable explanation of why the score is what it is.
- Side-by-side comparisons — because "better than yesterday's version" is often the only question that matters.
The payoff: every run yields not a number, but an account. Numbers rank. Accounts diagnose.
The Framework: Evaluate Across Five Trajectory Dimensions
When assessing any agent — coding or otherwise — grade the journey along these axes:
| Dimension | The Question It Answers |
|---|---|
| Instruction-following | Did it do what was asked, or what was convenient? |
| Tool use | Were tool calls purposeful or scattershot? |
| Self-verification | Did it check its own work before declaring victory? |
| Error recovery | When it stumbled, did it recover — or spiral? |
| Communication | Would a human trust the running narration? |
A high pass rate with low scores on recovery and communication signals an agent that gets lucky, not one that's reliable.
The Actionable Shift: Treat Evaluation as a Product Pipeline
AgentLens's real move is organizational, not academic. They run it as a nightly evaluation pipeline — the same way engineers run continuous integration for code.
This reframes evaluation from a one-time launch ritual into standing infrastructure. Three uses worth stealing:
- Diagnose behavior — understand how your agent thinks, not just its win rate.
- Compare versions — did v4 actually improve on v3, or just trade one failure mode for another?
- Catch regressions — spot the silent degradation where the score holds but the experience rots.
The lesson generalizes far beyond code: if you ship an agent, you need a nightly transcript-level review, not a quarterly leaderboard check.
What This Means for You
If you build, buy, or rely on AI agents:
- Stop trusting single-number benchmarks. Ask what happened during the run.
- Instrument the five dimensions. Even manual spot-checks beat blind faith in pass rates.
- Make evaluation continuous. The failures that hurt users most are the ones that creep in between releases.
The deepest idea here is quietly radical: the explanation is the evaluation. A score you can't interrogate is a score you can't trust — and can't improve.
Users live inside the trajectory. Your metrics should too.
Sources & Further Reading
- AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation — https://arxiv.org/abs/2607.06624