Awesome Evals

BenchFlow-maintained source map for building and evaluating AI agents: papers, practitioner posts, talks, tools, benchmarks, and runnable eval patterns.

What it is

benchflow-ai/awesome-evals is an annotated evals library rather than a plain awesome list. The README says the collection is assembled from recursive citation crawling, practitioner-web discovery, transcribed talks/podcasts, and gap audits; its sections cover eval motivation, benchmark design, LLM-as-judge, verifiers, trajectory grading, safety/adversarial evaluation, and the eval/RL-environment market. Source: GitHub README, 2026-06-26

The repo also includes a PATTERNS.md playbook for runnable patterns such as LLM-as-judge, pass@k/pass^k, error analysis, trajectory/world-state grading, CI gating, and verifiable rewards. That makes it a better source for agent evaluation design than a search query or scattered bookmark thread. Source: GitHub README, 2026-06-26

Add BINEVAL to the short list of concrete methods to check when the team needs an interpretable LLM-as-judge rubric: it decomposes criteria into binary questions, then aggregates inspectable verdicts. Source: arXiv:2606.27226, 2026-06-25

Add Red Queen Gödel Machine when the eval target is a self-improving agent. It is not just another benchmark source; it is a warning that a static judge becomes the thing the agent learns to satisfy. Use it to decide when a suite needs held-out cases, evaluator-evolution rules, or negative fixtures.

Inspected thread taxonomy

Xiangyi Li's screenshot thread is valuable because it is a taxonomy for choosing eval methods, not just a list of links. The inspected images organize the field into: why evals matter; evals as capability and RL-environment construction; model / harness / skill decomposition; observable grading surfaces; eval infrastructure; benchmark integrity; reward design; LLM-as-judge bias; agent trajectory and world-state evaluation; safety/adversarial evals; and talks/podcasts for deeper source review. Source: X/@xdotli, 2026-06-24

The operational takeaway for Kevin's stack: choose the grader by the object being graded. Use deterministic assertions for structured output, state diffs for environment-changing agents, trajectory match or trajectory judges for tool-use paths, binary LLM judges only after error analysis, and evaluator-evolution rules when the agent is improving itself. This feeds agent-eval-library, The Eval Loop (Slop Is an Output Problem), and Agent Trajectory Evaluation.

Version snapshot

As of 2026-06-30, GitHub reports benchflow-ai/awesome-evals on default branch main, not archived, with 593 stars, 41 forks, topics agent-evaluation, benchmarks, evals, llm-evaluation, and rl-environments, and no GitHub release. git ls-remote resolves HEAD to 54a392b65862165e2d5eef41d9d04c1675b4d66f; no release tags were returned, so treat the repo as commit-versioned rather than semver-versioned. The current README still reports 443+ curated links and 146 deep reading notes, while PATTERNS.md is the runnable practitioner playbook. Source: GitHub API and git ls-remote, 2026-06-30

Routing

Use this page when a task touches:

• evaluating agent output, tool use, trajectories, or world-state changes
• deciding whether a benchmark is valid or saturated
• turning production failures into new eval cases
• building CI gates for AI output
• comparing judgeable vs verifiable tasks

For implementation, route through The Eval Loop (Slop Is an Output Problem) first, then use ai-test-kit for deterministic AI SDK tests or Agent Trajectory Evaluation for agent-run traces. The current ai-test-kit route is latest=2.0.1 with ai peer support for 5.x || 6.x; use it for model/stream/UI fixture behavior, not provider-wire integration tests. Awesome Evals is the research map, not the runner.

Stack fit

This should feed Kevin's ai-agents.md, code.md, and backend.md packs because it sharpens the proof loop: an AI product is not "working" until its output has a benchmark, scorer, threshold, failure ledger, and replay path.

It is also a useful guardrail against overfitting to leaderboards. The repo's starter set includes sources on cost-aware agent evaluation, benchmark contamination, criteria drift, and process discipline, which match Kevin's no-sus preference for proof over vibes.

---

Timeline

• 2026-06-26 | Promoted from Kevin's GitHub stars harvest. GitHub API reported 452 stars, topics agent-evaluation, benchmarks, evals, llm-evaluation, and rl-environments; README describes 443+ curated links and 146 deep notes. Source: raw/github/benchflow-ai_awesome-evals-repo-2026-06-26.json; GitHub README, 2026-06-26
• 2026-06-30 | Deep-reviewed the full Xiangyi Li screenshot thread and promoted its taxonomy into the eval skill routing: final-output evals, trajectory/world-state grading, harness decomposition, eval infrastructure, benchmark integrity, verifier/RL environments, and LLM-as-judge bias. Current repo snapshot: main at 54a392b, 593 stars, 41 forks, no releases/tags. Source: X/@xdotli, 2026-06-24; GitHub API, 2026-06-30
• 2026-06-30 | Added direct X thread anchors for Xiangyi Li's eval-library screenshot replies that feed this page's source map. Source: X/@xdotli, 2026-06-24
• 2026-06-29 | Linked Red Queen Gödel Machine as the evaluator-evolution lens for self-improving agents. Source: arXiv:2606.26294; X/@omarsar0, 2026-06-28
• 2026-06-28 | Linked BINEVAL as the current binary-question LLM-as-judge method surfaced by Kevin's X bookmark batch. Source: X/@omarsar0, 2026-06-27; Source: arXiv:2606.27226