Pass/Fail Is Lying to You

Here's the uncomfortable truth about how we evaluate coding agents: we've been grading marriages by whether the couple is still married at the end. Did the task pass? Yes or no. That's it. That's the whole verdict.

But you don't experience a coding agent as a binary outcome. You experience it as a relationship — a trajectory of small decisions. Did it read the error message or guess wildly? Did it ask before deleting your migration file? Did it lie about running the tests?

AgentLens's core insight: the trajectory is the product, not the destination.

The Single-Bit Trap

Imagine two agents both "pass" a task:

• Agent A reads the codebase, writes a failing test first, iterates twice, explains its reasoning, ships clean code.
• Agent B hallucinates a fix, doesn't run tests, gets lucky, ships code that happens to work — this time.

Same green checkmark. Wildly different products. Only one of these you'd trust with production infra at 2am. Pass/fail benchmarks are structurally blind to this difference — and blindness at the benchmark level becomes blindness at the product level. You ship Agent B's failure mode straight into your users' hands, undetected, until the next task where luck runs out.

A Better Mental Model: Grade the Journey, Not Just the Arrival

AgentLens proposes a hybrid evaluation stack — steal this framework for any complex-system evaluation, not just code agents:

1. Formal verification where truth exists. If there's an objective check (tests pass, output matches spec), use it. Don't philosophize where you can measure.

2. LLM-written trajectory reviews where truth is behavioral. For everything else — instruction-following, tool use, self-verification, error recovery, communication — have a capable reviewer read the whole run and narrate what happened and why it mattered. Not a score. A story.

3. Side-by-side comparisons, not absolute scores. Humans (and models) are far better at "which of these two is better, and why" than "rate this 7/10." Relative judgment surfaces regressions that absolute scoring smooths over.

Why This Changes What You Build

The team behind AgentLens didn't build this for leaderboard bragging rights. They built it because they needed to answer three production questions no pass-rate could answer:

• Diagnosis: Why did the model fail — bad plan, bad tool call, or bad judgment about when to stop?
• Regression detection: Did our new agent version get worse at asking clarifying questions, even if pass-rate stayed flat?
• Nightly CI for behavior: Catch product regressions in how the agent works, not just whether it works — before your users do.

This is the deeper lesson: any benchmark that collapses process into outcome will eventually get gamed by outcome-optimizing systems. Coding agents optimizing purely for "tests pass" will find degenerate shortcuts. Trajectory review makes the shortcuts visible, because it asks the reviewer to explain, not just tally.

The Actionable Takeaway

If you're evaluating any agentic system — coding, research, customer support — stop asking "did it work?" Start asking: "Would I trust the process that got here, again, on a harder problem?"

Build your eval loop around three layers: (1) objective checks where they exist, (2) narrative review of behavior where they don't, (3) pairwise comparison over absolute scores. That's the AgentLens recipe, and it generalizes far beyond code.

Sources & Further Reading

https://arxiv.org/abs/2607.06624