Agent Trajectory Evaluation

Agent trajectory evaluation scores the whole run, not only the final answer: context, decisions, tool calls, observations, state writes, approvals, artifacts, costs, and final outcome.

Why final-output evals are not enough

A chat answer can be judged at the end. An agent changes the world on the way to its answer. It reads files, calls tools, edits code, spends money, writes memory, asks for approvals, and leaves artifacts. The final message can look correct while the trajectory is unsafe, wasteful, irreproducible, or impossible to debug.

The Eval Loop (Slop Is an Output Problem) captures the core Kevin thesis: slop is an output problem and quality needs measurement. Agent trajectory evaluation extends that loop to autonomous work. The unit of evaluation becomes the run, not the sentence.

What counts as a trajectory

OpenAI's Agents SDK tracing maps directly onto this model: traces and spans capture agent runs, model generations, function calls, handoffs, guardrails, and custom events. Source: OpenAI Agents SDK tracing docs, https://openai.github.io/openai-agents-python/tracing/, 2026-06-18

Metric families

IRL benchmark map

The state of the art is moving from "did the final answer match?" toward "did the agent take a good path through a real environment?" SWE-bench, Terminal-Bench, and tau-bench all push in that direction by involving repositories, terminals, APIs, policies, and user interactions.

Evaluation loop

1. Capture traces for real runs.
2. Label failure modes from traces, not imagination.
3. Convert recurring failures into rubric items and regression cases.
4. Score both final outcome and path quality.
5. Gate risky deploys or autonomous actions on the score.
6. Feed production failures back into the suite.

This is the trajectory version of the The Eval Loop (Slop Is an Output Problem) compounding arrow.

For non-symbolic trace quality, use BINEVAL-style atomic questions over each trace element: "did the agent read the nearest instructions?", "did it preserve the user's latest constraint?", "did a destructive action have an approval?", "did the verification command prove the edited surface?" The binary verdicts make a failed trajectory easier to repair than a single opaque run score. Source: arXiv:2606.27226

For Kevin's own agents, Agent Self-Improvement Eval Library is the first concrete receipt schema. It records events, commands, touched files, promoted pages, and hardening notes so a run can be checked after context compaction. The receipt is not the whole trace; it is the reviewable minimum needed to prove the agent followed the brain-first, skill-first, artifact-promotion, and closeout rules.

Leo Di Donato's "point another Codex at its trace logs?" reply is the practical test for trajectory evidence: a trace is good only if an independent reviewer or second model can reconstruct the actions, source choices, edits, and verification from it. If the trace requires the original agent to narrate what happened, the evaluation surface is still too weak. Source: X/@leodido, 2026-06-26

Kevin-stack implication

For wiki and agent work, a "done" claim should have trajectory evidence:

• relevant wiki pages searched or read,
• sources verified when facts are current,
• edits linked to the right pages,
• doctors/index/qmd rebuilt,
• tests or checks run,
• log updated,
• residual risk stated.

For Agent Machines, trajectory evaluation should become a product surface: run traces, tool calls, skill invocations, approvals, machine lifecycle, artifacts, and cost should be first-class review data.

OpenEnv is relevant if that product surface needs train/eval compatibility rather than only observability. Its simulation mode keeps MCP-backed tool calls inside the step() loop, so rewards, termination, and trajectory logging stay attached to the same environment actions. Production mode then exposes MCP directly for live clients without publishing the simulation control surface. Source: Hugging Face, https://huggingface.co/docs/openenv/guides/simulation-vs-production, accessed 2026-06-20

---

Timeline

• 2026-07-04 | Added the independent trace-audit test: trajectory evidence should be clear enough for another model or reviewer to inspect the run without relying on the original agent's final summary. Source: X/@leodido, 2026-06-26
• 2026-06-29 | Added the local self-improvement receipt runner as the first concrete Kevin-stack trajectory fixture. Source: scripts/run-agent-self-evals.ts; User request, 2026-06-29
• 2026-06-26 | Linked Awesome Evals as the external source map for trajectory/world-state grading, benchmark integrity, verifiers, pass@k/pass^k, and agent-eval infrastructure. Source: GitHub benchflow-ai/awesome-evals, 2026-06-26
• 2026-06-28 | Added BINEVAL-style binary questions as the rubric decomposition method for non-symbolic agent trace quality. Source: arXiv:2606.27226
• 2026-06-20 | Added OpenEnv as a watchlist item for trajectory-compatible agentic RL/eval environments, especially the simulation-vs-production split around MCP tool calls. Source: User request, 2026-06-20; Hugging Face docs, https://huggingface.co/docs/openenv/guides/simulation-vs-production
• 2026-06-18 | Created as the missing bridge between eval-loop, observability, benchmarks, and concrete agent runtimes. Source: whole-wiki graph audit, 2026-06-18