AI evals for agents: what they are and how to write them

This guide defines the vocabulary of AI evals (task, trial, grader, transcript, outcome, suite) and explains how agent evals differ from LLM evals: they grade a whole run, including tool calls, arguments, resulting state and final reply, not just one response. It covers the three grader types (code, model/LLM judge, human), the distinction between capability and regression evals, and pass@k versus pass^k, then walks through writing an agent eval in seven steps with a worked refund example.