How to Bootstrap Agent Evals with Synthetic Queries

Checking agent outputs isn't enough. The real failures hide in trajectories: which tools got called, in what order, with what inputs. This article walks through a pattern for building evals when you don't have production data yet. You define the dimensions your agent varies along, generate structured tuples across them, and turn those into natural-language test queries. Run them, read the traces, write down what broke. Those notes become goals that shape the next batch of queries. Repeat until the failures vanish.