How to Use Decision Models for Evals (Jev & More)

This video explores how to choose and tune decision models for evaluation (evals) tasks, focusing on performance and cost. Six models were tested: Jev 1.13, Perplexity Decider 1.1, Liquid D1 and D1 Ultra, Cloudflare's Clef Flash, and OpenAI's Decisions. The experiment used 100 user session logs from a real AI assistant for property managers, with over half containing issues like failed human handoffs, incorrect appointment booking, or poor communication. **Key Concepts and Steps:** * **Decision Thresholds:** Models output a pass/fail flag and a score from 0 to 1. The default threshold is often 0.5, but it's crucial to tune this. * **Precision:** The proportion of flagged errors that are actual errors (minimizing false alarms). * **Recall:** The model's ability to find all actual errors (minimizing missed issues). * There's a trade-off: higher precision means lower recall, and vice versa. The optimal balance depends on business needs (e.g., cancer screening prioritizes recall, criminal law prioritizes precision). * **Visualizing Model Scores:** Plotting score distributions for passes and fails helps understand model separation. The goal is for passes and fails to have distinct score ranges. OpenAI Decisions performed best in separating these in the example. * **Precision-Recall Curves:** These curves visualize precision and recall at different thresholds for each model. * **Minimum Thresholds:** Before tuning, set minimum acceptable precision and recall levels (e.g., 80% for both). This filters out underperforming models. In the example, only OpenAI Decisions, Liquid D1 Ultra, and Perplexity met the 80% minimum. * **Choosing a Threshold:** After meeting minimums, select a specific threshold based on the desired balance between precision and recall. OpenAI Decisions offered the most flexibility here. * **Tuning the Decision Threshold:** * **Select a Metric:** Choose a metric that reflects the importance of precision and recall. The F1 score (equal weighting) is a common choice, but F-beta scores allow for custom weighting (beta < 1 for more precision, beta > 1 for more recall). * **Assign Value:** Consider assigning dollar values to catching failures and the cost of false alarms to guide metric selection. * **AI Assistance:** Use AI agents with specific prompts to tune thresholds, maximizing the chosen metric while respecting minimums and using cross-validation to avoid overfitting. Stratified five-fold cross-validation is recommended for robust evaluation. * **Impact of Tuning:** Threshold tuning can significantly improve performance. In the example, some models showed minor improvements, while others benefited more. * **Comparing Performance and Cost:** * **Visualization:** Plot performance (e.g., F1 score) against cost. * **Pareto Frontier:** Identify models on the Pareto frontier, representing the best trade-offs between performance and cost. Models below this line are suboptimal. Liquid D1 Ultra offered similar performance to Perplexity but at a higher cost, making it less appealing. * **Interpreting Probability Scores (Confidence):** * **Calibration:** Raw probability scores from models may not accurately reflect true confidence. Plotting predicted probabilities against observed failure rates (probability calibration chart) reveals calibration issues. * **Calibration Techniques:** Methods like Platt scaling or isotonic regression can recalibrate scores to better match reality. AI can assist in generating these plots and applying calibration techniques. * **Relevance for Evals:** If only binary pass/fail decisions are needed, probability calibration might be unnecessary. However, it's important if scores are used for other decision-making. **Recap and Recommendations:** 1. **Set Minimum Precision/Recall Thresholds:** Filter out unsuitable models. 2. **Choose a Metric:** Select a metric (e.g., F1, F-beta) that aligns with your business priorities for precision and recall. 3. **Tune the Threshold:** Optimize the decision threshold based on your chosen metric and minimums, using cross-validation. 4. **Check Probability Calibration:** If using scores as confidence estimates, verify and potentially calibrate them. 5. **Visualize Cost vs. Performance:** Plot models to identify the best trade-offs and make informed choices. The process is iterative; revisiting steps and refining prompts can lead to better model selection and optimization.