A PaperBridge long-tail question
How do you build an AI agent evaluation dataset before you have production logs?
Do not pretend a synthetic dataset can predict the future user distribution. Build a small, explainable launch gate around every product promise and costly failure. Each case should define the initial state, user goal, allowed tools, expected environment state, prohibited actions, and grading method. Prefer database, file, API, or test assertions over a second model whenever the result can be checked mechanically. After launch, add real failure traces by type while keeping the original seed cases as regression tests. This is not a permanent gold standard; it is a contract for the minimum acceptable behavior at launch.
1. Start from product promises and failure cost, not random personas
List the tasks the agent explicitly promises to complete at launch: find an order, change a booking, produce a report, or modify code. Give each promise a normal case, a missing-information case, and a tool-failure case.
Then list outcomes that must never happen: unauthorized actions, corrupt customer data, privacy leakage, duplicate payment, or an irreversible change without confirmation. A rare but expensive failure still belongs in the Day-0 gate.
2. Write each case as a verifiable state transition
For a tool-using agent, a final answer that sounds right is not enough. Save the database, file, or application state before the task and define the state that should exist afterward. Both τ-bench and OSWorld emphasize execution outcomes rather than text-only judgment.
A case should contain the request, initial environment, tools and permissions, success state, prohibited state, timeout and stopping condition, and deterministic grader. Reserve human or model judging for qualities such as tone or explanatory clarity that cannot be reduced to a state check.
3. Cover failure layers without manufacturing a giant fake dataset
Separate task understanding, planning, tool selection, arguments, state recovery, policy compliance, and final verification. AgentBench's multi-environment results show that long-term reasoning, decision-making, and instruction following are distinct failure sources; one success rate hides where the system is weak.
Keep the seed set small enough that a person can explain and repair every failure. Add a case only when it covers a new promise, risk, or observed failure—not merely another paraphrase generated to inflate the dataset.
- Happy path: complete information, available tools, clear goal.
- Ambiguous path: missing fields, conflicting goals, mid-task changes.
- Tool path: timeout, partial success, duplicate response, stale state.
- Safety path: insufficient permission, privacy request, irreversible action, prompt injection.
- Recovery path: consistent state after retry and no duplicate side effect.
4. Repeat trials and measure reliability, not one demo
One success on one run is not production reliability. Repeat critical cases and report both single-run success and the probability that repeated trials all succeed. τ-bench uses pass^k to expose inconsistency across interactions.
Track cost, latency, tool calls, human-escalation rate, and whether failures stop safely. HELM's multi-metric approach applies here: accuracy is only one dimension, while robustness, efficiency, and safety also determine launch readiness.
5. Let real traces take over coverage after launch
Define production failure labels in advance: unfinished goal, bad tool arguments, authorization failure, state drift, false refusal, and user abandonment. Review traces on a schedule and minimize each new failure into a reproducible case.
Synthetic cases can be replaced by realistic ones, but retain the original high-cost risk seeds. Production frequency and severity are not the same; rare payment or privacy failures should remain permanent regression gates.
Minimum Day-0 evaluation contract
- List every launch promise and its costly failure modes.
- Store initial state, goal state, allowed tools, and prohibited actions.
- Write execution-based graders first; add human or model judgment only where needed.
- Cover happy, ambiguous, tool-failure, safety, and recovery paths.
- Repeat critical cases and report reliability, not the best run.
- Track success, cost, latency, tool calls, and safe stopping.
- Promote real traces by failure type while retaining risk regression seeds.
Primary research and official documentation
These sources support the facts. Workflow and comparison guidance is PaperBridge's synthesis of research, official documentation, and engineering practice.