I Ranked Every AI Eval Concept From Worst to Best

Here's a summary of the transcript, focusing on the key points and ranking of eval concepts: **Key Eval Concepts Ranked:** * **S Tier (Essential):** * **Error Analysis:** A structured approach to reviewing data to identify application issues. Crucial for effective evaluation. * **Custom Annotation Tool:** Building a dedicated tool to streamline data review, removing friction and improving clarity. This offers significant advantages over generic methods. * **Reading Traces Yourself:** Directly examining your data is invaluable. There's no substitute for looking at the actual outputs. * **Validating Judges Against Human Labels:** Essential for trusting any automated judge (like an LLM). Human labels are the benchmark for accuracy. * **Domain Experts Writing Prompts:** Domain experts, not just developers, should be involved in crafting prompts to ensure relevance and accuracy for the specific product. * **A Tier (Highly Recommended):** * **Binary Pass/Fail Metrics:** Preferring binary outcomes (pass/fail) over scaled ratings (e.g., 1-5) simplifies decision-making and cuts through noise. * **Running Always Passing Evals Less Often:** If an eval consistently passes, it may be losing its value. Re-evaluate its necessity to optimize resources. * **Benevolent Dictator for Annotation:** Appointing a single trusted individual to make final annotation decisions reduces complexity and cost compared to committee-based annotation. * **B Tier (Good Practice):** * **Designing for Verification:** Structuring your application to easily show intermediate outputs to experts aids in evaluation and debugging. * **C Tier (Acceptable, but not ideal):** * **Writing Evals Before Building Anything:** Brainstorming potential failures upfront can be a starting point, but it's inefficient due to the vast number of possibilities. Data should guide prioritization. * **Deciding on a Rubric Before Reading Data:** Similar to brainstorming, this top-down approach is less effective than analyzing actual data first. * **D Tier (Suboptimal):** * **Evals Always Passing 100%:** If evals consistently pass, they aren't challenging enough or revealing new issues. Evals should sometimes be difficult to pass. * **Only Devs Annotate Data:** Unless building a coding agent, domain experts should annotate data, not developers. * **Immediately Writing an Eval for Every Failure:** While it's good to capture critical failures, not every minor issue warrants an immediate, costly eval. Prioritize based on impact and fixability. * **F Tier (Avoid):** * **Generic Metrics:** Scores like "helpfulness" or "conciseness" from external tools are usually uninformative and a waste of time. * **Off-the-Shelf LLM Judges:** Pre-built judges with generic prompts are rarely suitable for specific product needs. * **Prompt Optimization Without Evals:** Optimizing prompts without clear eval criteria leads to unfocused improvements and potential "slop." * **Public Benchmarks as a Substitute for Product Evals:** Public benchmarks are useful for model selection but don't reflect product-specific failures. * **One Judge for Everything:** A single, broad judge prompt is too scoped and unlikely to be effective. * **Eval Based Only on Brainstorm Failures:** Relying solely on brainstormed failures without data analysis is inefficient. * **Off-the-Shelf Guardrails:** Using generic, pre-made guardrails is generally ineffective as they aren't tailored to your product. * **One-to-Five Ratings:** Scaled ratings often hide complexity and hinder clear decision-making compared to binary outcomes. * **Picking an Eval Vendor First:** Focus on process and data analysis before selecting tools. * **Rugg and Bird Score for Answer Quality:** String similarity metrics are not suitable for evaluating AI answer quality. * **Letting AI Do All Your Evals:** AI can assist, but it cannot catch all nuanced issues requiring human judgment. The presenter offers a detailed eval guide at bit.ly/eval-guide for further information.