The Final Answer Is the Least Interesting Agent Trace

An agent can produce the right answer after calling the wrong tool, leaking private context into a query, retrying a payment, ignoring a policy, and finally guessing correctly. A final-answer grader marks the run as a success.

That score is not false. It is incomplete. Agent reliability lives in the path between task and outcome.

Outcome is necessary, not sufficient

Final state matters. A coding agent should make the tests pass. A support agent should resolve the request without violating policy. A research agent should produce a grounded report. If the outcome is wrong, the run failed.

But a correct outcome does not tell you whether the path was reproducible, efficient, or safe. Tool-using benchmarks increasingly make this distinction. τ-bench evaluates agents that interact with users and tools under domain policies. TRAJECT-Bench adds diagnostics for tool selection, arguments, and dependency order rather than scoring only the last answer.

What to record

A useful agent trace connects intent, action, observation, and environment state:

run
├─ task + acceptance criteria
├─ model and harness versions
├─ step 1
│  ├─ visible context
│  ├─ proposed tool + arguments
│  ├─ policy decision
│  ├─ tool observation
│  └─ state diff
├─ step 2 ...
├─ final state
├─ final response
└─ stop reason + budget usage

Capture references to large artifacts rather than copying everything into one event. Preserve stable identifiers so a grader can connect a tool call to the exact state it changed.

Do not rely on hidden model reasoning as your observability surface. The actionable trace is the externally visible contract: inputs, tool proposals, approvals, observations, state transitions, and outputs.

Grade the layers

Separate scores answer different engineering questions:

A run can pass outcome and fail efficiency. It can pass tool selection and fail argument safety. Keeping the dimensions separate makes regressions diagnosable.

Check deterministic invariants first

Use code-based graders wherever the environment exposes a fact:

• database row matches the expected value,
• only authorized files changed,
• no tool call exceeded its declared scope,
• required approval happened before the mutation,
• the final artifact parses and its tests pass,
• no irreversible action was executed twice.

Use model graders for semantic judgments that code cannot express cheaply: whether a summary is faithful, a clarification was appropriate, or an explanation meets a rubric. Calibrate those graders against human labels and retain disagreements. A model judge is a measurement instrument, not ground truth.

Anthropic's guide to agent evals recommends combining deterministic, model-based, and human grading based on the task. The same layered approach applies inside a trajectory.

Measure repeatability, not the lucky run

Agent execution is stochastic and environments are noisy. One pass says that success was possible. It does not estimate how often users will see it.

Run each task several times and report both capability and consistency:

• pass@k: did at least one of k attempts succeed?
• pass^k: did all k attempts succeed?
• median and tail cost, latency, and tool-call count,
• frequency of policy violations and human interventions,
• failure clusters by trajectory stage.

Capability is useful during exploration. Consistency is what production feels like.

A better standard

Keep the final outcome as the primary product score. Then require the trajectory to satisfy invariants for safety, efficiency, ordering, and communication.

The final answer tells you what happened once. The trace tells you whether the system was in control while it happened—and whether you should trust it with the next task.