How To Test AI Agents With Simulations

Here's a summary of the provided transcript, focusing on key insights about evaluating and simulating AI agents: **1. The Evolution from Chatbots to Agents and its Impact on Evals:** * Traditional evals for chatbots focused on simple input-output pairs with predefined correct responses (e.g., "What is the capital of the US? -> Washington DC"). * The shift to AI agents introduces significant complexity: * **Agent components:** Agents now involve skills, tool definitions, memory, sub-agents, and their respective prompts. * **External dependencies:** Agents are deeply intertwined with the real world, interacting with internal databases, other services, and external contexts (e.g., emails). * **Context sprawl:** Not all context resides within the code repository anymore, making traditional prompt-based evals insufficient. * This increased complexity means agents are harder to evaluate, and single input-output pairings quickly become inadequate, as most issues arise from unforeseen interactions. **2. The Limitations of "More is More" Evals:** * The strategy of accumulating thousands of input-output eval pairs (e.g., 4,000-6,000) is often ineffective for agents. * **Scalability issues:** Agent capabilities increase faster than the ability to create new evals. * **Maintenance burden:** Running thousands of evals for every code change (e.g., adding a tool) is impractical and time-consuming. * **Fragility:** Evals themselves can break when underlying harnesses or agent architectures change, leading to a false sense of security. Many companies struggle with maintaining functional evals. **3. Raindrop's Initial Focus: Unsupervised Real-time Production Monitoring:** * Raindrop initially focused on monitoring agents in production to catch critical issues in real-time. * This "unsupervised real-time issue detection" relies on three ingredients: * **Codebase/Prompts:** Understanding expected behavior based on the agent's definition. * **Baseline Behavior:** Detecting drifts from established patterns after changes. * **Feedback on Production Data:** Learning from user interactions with alerts and summaries derived from traces to identify relevant issues. * The goal is to provide high-signal issue detection, ensuring teams react to almost every alert. * **Importance of Traces:** Directly showing traces with annotations is crucial for debugging, as AI-generated summaries can be misleading or "slop." **4. The Need for Prevention: Introducing Simulations:** * While production monitoring is effective for catching issues, it doesn't prevent them. * The stakes for AI agents are rising in critical domains like healthcare, defense, and finance, making prevention essential. * **OpenAI's Approach:** OpenAI (and other foundation labs) use simulations extensively, replaying past traffic against new model changes to understand their impact. * **Core Concept of Simulations:** Replay real, historical traffic against a new change (model, tool, prompt, infrastructure) to observe differences. This answers: "What would my change change? Does it fix the issue? Does it cause new ones?" * **Targeted Testing:** Simulations are highly targeted to specific changes (e.g., deleting a tool) by replaying traces where that tool was previously used. **5. Addressing Multi-turn and Simulation Challenges:** * **Multi-turn Conversations:** Differentiates between long, continuous conversations (hard to simulate user behavior) and turn-by-turn interactions where the agent's world state needs simulation. * **Simulating the Agent's World:** The key is to simulate the entire environment the agent interacts with (databases, services, Slack, GitHub, etc.) as it existed when the historical trace ran. * **Why not just cache tool results?** Agents can detect simple caching and become "simulation aware," refusing to perform or behaving differently. * **Swiss Cheese Analogy for Simulation:** * **Historical Tool Calls:** Use previous tool call results as "windows" into the world state at that time. * **World State Changes:** Recognize that the real world changes over time; directly connecting to current production databases for historical replays would yield incorrect results. * **Hybrid Approach:** Generate upfront simulations using past results and transformations, while also employing an LLM to generate correct data on the fly when needed. * **Simulation Awareness:** A critical metric is whether the agent believes it's in the real world. If an LLM can statistically determine a trace is simulated, it indicates poor simulation fidelity. Agents can learn to detect evals and simulations, influencing their behavior. * **Simulating Infrastructure:** Recreate as much infrastructure as possible (e.g., SQL parsers) to use actual code for validation. Network calls are often the boundary for simulation. * **Fidelity of Simulation:** * Connect to GitHub, check out the agent's code into a sandbox, and spin up sandboxes for the agent's own sandbox environments. * **Alignment Flow:** Run simulations against production traces from "main" to ensure the simulation environment accurately reflects current production before testing PRs. The answer to "how do you know the simulation is faithful?" is "simulations." * **Evaluating Faithfulness:** Use LLMs to blindly guess if a trace is simulated vs. real. If the LLM's accuracy is above chance, the simulation is not faithful. * **Data Simulation:** * Use the "Swiss cheese" approach, leveraging past tool call results. * For new data sources, simulate and guess what the database would have contained. * Allow users to connect read-only "MCPs" (Multi-Cloud Platform) to pull realistic production data and schemas into the world-building process, especially for entirely new data sources. **6. Practical Aspects of Simulations:** * **Number of Simulations:** Not thousands for every change. It depends on the size of the change and the selection of scenarios. A small change might need 5-10 targeted traces; a large rewrite might need 50+. Focus on selecting traces that exercise the changed components. * **What to Look For (Diffs):** * Changes in final response. * New tool errors or usage patterns. * Changes in response trajectory or step count. * Differences in cost and latency. * **Diffs vs. Evals:** * **Diffs:** Focus on "what changed?" (e.g., comparing old vs. new traces). LLMs can then judge if the change is good or bad. * **Evals:** Pure functions that run over traces, returning a score or pass/fail. They align the organization on definitions of "good" or "bad." * Examples: Verbosity (is the response too long?), consistency (is the answer the same as before?), cost (is it within 20% of previous cost?), step count (within 20% of previous steps?). * Evals should scale to any arbitrary trace, unlike input-output pairs. * Can be used to check for consistency (e.g., "is the answer the same?"). * Can also be used with "bad" traces to confirm fixes (e.g., "does the new output still exhibit the same issue?"). * **Gradient of Evals:** Start with deterministic metrics (e.g., length, step count) before moving to LLM-based judgments for subjective qualities (e.g., conciseness, writing quality). **7. Common Simulation Pitfalls (Foot Guns):** * **Replaying against live systems:** Leads to simulation awareness and incorrect data. * **Only serving cached tool responses:** Agents will detect this. * **Only replaying failures:** Can lead to overfitting; include a random sample of successful traces. * **Ignoring nondeterminism:** Replay traces multiple times. * **Not looking at traces:** Crucial for understanding what's happening; human review of a few traces is more informative than any eval score. **8. When to Use Simulations:** * **Start with local testing:** Manually run the agent, look at traces, and use an LLM to identify issues or improvements. * **As soon as you run evals for agents:** Some form of simulation is needed due to agent complexity. * **Consider building your own:** If the agent is simple, with few tools or straightforward data sources, you might be able to create a custom solution. * **Avoid slow feedback loops:** Simulations offer a tighter feedback loop than pushing to production, monitoring, and iterating. * **Complementary to Production:** Production remains the ultimate source of truth, but simulations prevent bad things from happening before deployment.