Can Claude Build Your Evals?

Anthropic has released a new tool aimed at automating evaluation (eval) design and hill climbing with Claude. The tool is integrated into the Claude API as a skill, offering `build_eval` and `hill_climb` functionalities. The blog post introducing the tool outlines principles of good eval design, including mirroring production tasks, improving performance with stronger models and more thinking, and ensuring there's "passable headroom at the frontier." This latter point means that the most capable models should not score 100% on evaluations, allowing for reliable judgment of performance changes. Evals should not be saturated or impossible, with tasks being challenging but achievable by humans. The concept of adversarial sampling is also discussed, suggesting that focusing on areas where models fail can be more sample-efficient. The idea is to sample in the "valleys" of a model's capability surface, though there's a caveat that sampling based solely on human judgment of difficulty might not always align with AI capabilities. The `build_eval` command is presented as a guided workflow where Claude interviews the user, builds the eval within their codebase, and pauses for approval at key points. It helps sample inputs by looking at production transcripts, bug reports, manually written cases, and synthesized cases from the codebase. It can also prioritize production traffic or generate synthetic data anchored to real examples. Claude generates a simple page to show inputs, awaiting user confirmation. After input validation, Claude proposes a suitable grader: programmatic verification (code) or an LLM as a judge if code isn't feasible. Claude then grades a sample of cases and asks for user feedback on whether they would have scored them differently. The importance of reviewing scored transcripts before trusting the evaluator is emphasized, as misconfigured evaluations are a common failure mode. Once validated, the skill reports the evaluation set size, runs a baseline, and provides a score with a confidence interval. The output includes cases, the grader, a runner, JSON lines, full transcripts per case, and a plain page listing each case's score with a link to its transcript. Diagnostic checks are performed during baseline runs. Claude reruns a grader on the same output to check for verdict consistency and monitors for timeouts, API errors, and cutoff answers to prevent infrastructure noise from being mistaken for model variance. If the baseline scores 95% or higher, the skill warns the user, suggesting that hill climbing should focus on cost or latency rather than quality. The demonstration of the `build_eval` tool involved using a dataset of 100 traces from a real estate AI assistant. The user provided these traces in a `traces.json` file, as they did not have application code to share. Claude was prompted to build an eval using this data. Claude's initial analysis of the 100 traces identified several potential failure modes: * **Call transfer issues:** 14 traces involved call transfers, with subcategories like "break the prompts protocol" (10 traces), "re-asks" (6 traces), and "transfer without confirmation" (1 trace). * **Voice call hygiene:** Specific traces were flagged for triggering "end call allowed" and "transfer." * **Tour scheduling:** 35 traces related to booking tours, with 7 indicating the tool rescheduled but no trace showing an actual cancel or reschedule tool action, leaving old bookings intact. * **Date errors:** A specific instance of "Saturday, January 27th" was flagged. * **Made-up capabilities and facts:** Promises of virtual tours without tool support and bathroom layout claims without data. * **Email placeholder issues:** 12 replies ending with a literal "your name." Claude then asked which behavior should become the eval. The recommendation was "call transfer rules." The user expressed a preference to see the data first before committing to an eval, but Claude proceeded to build the eval for call transfer rules. The tool then presented inputs in a markdown file (`inputs.md`) and suggested skimming them to verify the calls and labels. This format was criticized for being difficult to read and not user-friendly, especially compared to HTML viewers. The user manually reviewed some traces in the markdown file and found issues, including repetitive and annoying bot behavior, and instances where the bot asked to connect the user multiple times. Claude then proceeded to build a code grader for call transfer rules. The grader included checks for specific phrases like "Would you like me to connect you?" and conditions like "no transfer happened." The user found these checks to be too brittle and too specific, particularly the "protocol okay is one if no transfer happened" rule, which lacked clarity. The complexity and potential for bundling multiple issues into a single eval were highlighted as problematic, making interpretation difficult. The tool then presented a scoring interface where users could agree or disagree with Claude's verdicts on individual traces. The user found this process tedious and backward, preferring to review raw data first to form their own understanding of what constitutes good performance before engaging with an automated eval. They noted that the interface for providing feedback (editing markdown or interacting with prompts) was cumbersome and lacked modern UI elements like HTML viewers. The user expressed significant friction in the process, particularly with the markdown file for reviewing traces and the subsequent labeling and feedback mechanism. They felt that the tool pushed them towards fixing an eval rather than building a good understanding of the data and the product's problems first. The lack of clarity on whether the tool created one eval or multiple, and the difficulty in prioritizing issues without seeing their relative frequency or impact, were also major concerns. Ultimately, the user concluded that while the tool showed some promise in identifying issues, the user experience and workflow were not efficient or intuitive. They felt that a more straightforward approach, such as using a data viewer to examine raw traces and take notes, would be more beneficial for someone new to evals or a new dataset. The current process felt like trying to fix a poorly constructed eval rather than building a robust one from the ground up. The user would fail the eval based on their experience, stating they wouldn't use it themselves. The final verdict from the user was that building a data viewer and manually inspecting traces would be a more effective starting point for improving an AI product.