AgentLens: Why the Journey Matters More Than the Destination

The Insight That Changes How You Evaluate AI

A passing test tells you nothing about whether you'd want to work with the agent that passed it.

Most coding-agent benchmarks collapse a rich, multi-step interaction into a single bit: pass or fail. But nobody who actually uses an agent experiences a bit. They experience the trajectory — the fumbling, the recoveries, the moments of clarity, the tone of the explanations. AgentLens is built on a deceptively simple correction: evaluate the whole journey, not just the arrival.

This reframe matters far beyond code. It's a model for evaluating any complex process where the outcome hides the story.

The Trajectory Lens: Five Dimensions of Real Quality

When you watch how an agent works rather than just whether it succeeds, five behaviors reveal themselves:

1. Instruction-following — Does it do what you asked, or what it assumed?
2. Tool use — Does it reach for the right instrument, or flail?
3. Self-verification — Does it check its own work before declaring victory?
4. Error recovery — When it stumbles, does it diagnose and adapt, or spiral?
5. Communication — Does it explain itself in a way a human can trust?

Two agents can both "pass." One arrives cleanly. The other guesses, breaks three things, silently patches them, and gets lucky. The score is identical. The trust is not.

The Framework: Pair the Objective with the Legible

AgentLens's core method is a mental model worth stealing for your own evaluation work:

• Formal verification where an objective check exists. Did the code compile? Did the tests pass? Hard, binary, cheap.
• LLM-written trajectory reviews where judgment is required. Why did this run score as it did? A readable explanation, not a number.
• Side-by-side comparisons to surface relative quality that absolute scores obscure.

The principle generalizes: use objective checks where they exist, and structured judgment where they don't — but never pretend one is the other. Most evaluation failures come from forcing a subjective quality into a binary metric, or leaving a measurable one to vibes.

From Ranking to Diagnosing

Here's the leap. A benchmark that only ranks models answers which is best? A benchmark that explains itself answers why, and what to fix?

AgentLens is used three ways in production:

• Diagnose behavior — understand how a model fails, not just that it does.
• Compare versions — track whether your own agent is actually improving.
• Catch regressions — a nightly pipeline flags when a "better" release quietly got worse in ways a pass-rate would miss.

Actionable takeaway: The next time you build or choose an evaluation, ask a single question — does this output tell me what to change? If your metric produces a leaderboard but no lessons, you've built a scoreboard, not a diagnostic.

The Meta-Lesson

The move from "did it pass?" to "how did it get there?" is the difference between measurement and understanding. It costs more — reviews are slower than boolean checks. But the cost buys you explanations you can act on, and explanations compound where scores merely accumulate.

Practice this week: Take one process you currently judge by outcome — a hire, a decision, a deliverable. Write down the trajectory it took. You'll often discover that the outcome was right for reasons that won't repeat, or wrong for reasons you can fix.

Evaluate the journey. The destination was never the whole story.

Sources & Further Reading

• AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation — https://arxiv.org/abs/2607.06624