7 Most Common AI Eval Mistakes (From Teaching 5K+ Engineers & PMs)

Here are the seven common mistakes to avoid when building Evals: * **Creating Evals Before Looking at Data:** * Brainstorming potential issues without data leads to infinite possibilities and wasted time. * You'll end up with a dashboard of irrelevant metrics. * Always examine real data to understand what "good" looks like and identify actual problems to evaluate. * Example: An AI leasing assistant failed to handle a prospect's budget objection, missing an opportunity to suggest other properties. * **Using Generic Metrics Off-the-Shelf:** * Generic metrics like "helpfulness" or "conciseness" are not actionable. * Scores on a scale of 1-5 are meaningless without clear definitions. * Focus on specific, observable issues relevant to your product. * Better metrics: "Did we handle sales objection when a qualifying opportunity was presented?" * Metrics should correlate to observed data, be reproducible, and address non-trivial issues. * Aim for binary pass/fail outcomes, not scores, for easier alignment. * Use human labels to verify and build confidence in your LLM judge. * **Not Validating LLM Judges Against Human Labels:** * Do not assume an LLM judge is good just because you wrote a prompt. * Segment data into train (20%), dev (40%), and test (40%) sets. * Source examples from the train set for prompt writing. * Tune the LLM judge against the dev set to maximize agreement. * Use the test set to ensure generalization and avoid overfitting. * Measure accuracy, precision, and recall (TPR, TNR), not just accuracy. * Crucially, align and measure your LLM judge against human labels. * **Not Designing Products for Evals:** * Design products with evaluation in mind from the start. * Avoid systems that only provide a final answer, making evaluation difficult and expensive. * For expert users, design for transparency: show sources, intermediate calculations, and trust signals. * Example: A data agent should show how it computed a number, not just the number itself. * This approach also improves user experience. * **Using Likert Scales (1-5) with LLM Judges:** * Avoid carrying over the intuition of "shades of gray" to eval construction. * Evals should mirror product and business decisions: "Is this good enough to ship? Yes or no?" * Likert scales create massive complexity and require extensive human alignment. * Scope evals to single, specific errors for easier binary scoring. * Binary pass/fail outcomes for specific issues (e.g., politeness, scheduling errors) are more actionable and cost-effective. * **Having the Wrong People Review Data:** * Developers are often not domain experts. * Domain experts (lawyers for legal assistants, doctors for medical assistants) should review data. * Create custom annotation apps to facilitate expert review. * This ensures data is evaluated by those who understand the context and can identify good or bad outcomes. * **Relying Too Much on Automated Approaches:** * Automated agents can miss many errors due to lack of context. * Looking at the data yourself is the best way to understand what is good and bad. * Leading automated eval vendors often miss obvious issues like sales objections, formatting errors, interruptions, and human handoff problems. * You must look at your own data to refine your understanding of "good" and ensure your evals are effective.