Evaluating Agents
1. Evaluating Agents
How do you know your agent actually works? And more importantly, how do you measure whether a prompt change, model swap, tool update, or harness modification made things better or worse? Traditional software testing breaks down pretty quickly with AI agents because of non-determinism. The same input can produce different outputs across runs, and a lot of agent failures are hard to catch. The response could come back, but be low quality, unsafe, incomplete, off policy, or subtly incorrect. And one of the biggest mistakes people make is only evaluating single-turn interactions. Most agents look great on turn one, but the failures usually appear after longer interactions when context accumulates, summaries compress information incorrectly, tools get repeatedly called, and the model slowly gets off the rails over time. That's why evaluation is a core part of Harness engineering. Good harnesses need mechanisms for measuring reliability, detecting regressions, validating workflows, and stress testing behavior over time. One of the most common patterns to use for this is LLM as a judge. You run your agent against many test cases, then use another model to evaluate the outputs against a rubric you define. The judge model scores the results and explains its reasoning. Strands ships with an evals SDK that helps structure these types of evaluations. To install it, run pip install strands agents evals. The SDK gives you building blocks for defining test cases, running experiments, simulating conversations, generating evaluation data sets, and scoring agent behavior. Let me start with a very simple evaluation so you can see the basic structure. We define some telemetry capturing up front, and then we have a task function that creates a simulator, passing in the specific case to simulate and max turns defined. Then we create the customer service agent. Notice how we pass in callback handler is none to suppress the output, but otherwise this is using the same customer service agent that we've been using over the last few videos. Then we run a multi-turn conversation loop to simulate a back and forth with a customer. Then we map those traces to each session and return the response. Then we define a few different cases to evaluate. Each case includes an input prompt and an expected outcome defined in metadata. Next we define an evaluator using a rubric. You can supply your own rubric in plain language, which tells the judge model how to score the response, but I'm using two different built-in evaluators that look at helpfulness and if the outcome was met. There are multiple built-in evaluators you can use, or you can write your own custom ones. Then we bundle everything into an experiment and run the evaluation suite. Let's go ahead and run it. The evaluation suite has run and we can see that our pass rate is only about 66% on average, so we have some work to do for our customer service agent, or we may have to evaluate how our evaluations are being run. But this is great data for us to start with. We can dig in to the results for each of these by expanding the results here in the command line. And if I scroll up, you can see the first test case that we had was for refund request, and this had a score of 83%, so this was looking pretty good. It did pass. And you can see here the reason that the judge model provided for why it ended up passing this, and you can also see the input that was passed in. For the next one we have order tracking, which got scored at a 0.5. And if we were to read through this, we would be able to figure out why it scored it that way. And then finally, for account issue, it scored zero because there actually was an error here. So, it seems that the account was having issues and it couldn't look it up. So, this is some awesome data that we could then take to then go debug, run more test cases, and figure out if the problem is with our agent, our evals, or maybe both need some work. Now, another challenge with evals is coverage. When you first start building an agent system, you often don't even know what cases you should be testing yet. That's where the experiment generator can be helpful. Instead of manually creating every test case yourself from scratch, the generator can inspect your agent's tools, workflows, and capabilities, and automatically bootstrap candidate evaluation cases for you. This is useful for discovering missing coverage areas or quickly generating an initial evaluation suite that you can later refine manually. It's not a replacement for carefully designed evals, but it's useful for getting started and expanding your test surface area. Now, let's also briefly talk about trajectory evaluation. Sometimes you don't just care about the final answer, but you also care whether the agent followed the correct workflow. For example, maybe your support agent must look up the customer, retrieve order history, validate refund eligibility, and then process the refund. Trajectory evaluators let you validate whether the expected sequence of actions actually occurred by evaluating the path the agent took to get there. You can also create completely deterministic evaluators in pure Python because not every evaluation needs another model call. For example, you might validate JSON schema correctness or response length. These are fast, deterministic, and inexpensive compared to LLM judges. In practice, production eval systems often combine both approaches. Deterministic assertions and LLM judges for subjective quality evaluation. As you improve prompts, add tools, introduce memory systems, change orchestration patterns, or swap models, you rerun your eval suites to detect regressions. You can even wire these evaluations directly into a CI/CD pipeline, so deployments fail automatically if reliability scores drop below acceptable thresholds. Also, production users will always expose edge cases that you didn't anticipate. So, your eval suite should continuously evolve as new failure modes are discovered. And one final thing to be aware of, evals can become expensive. The LLM as a judge pattern requires additional model calls for scoring, and large simulation runs can consume significant tokens very quickly. But despite the tradeoffs, evals are one of the most important investments you can make when building production agent systems. Without them, you really don't know whether your agent is improving or getting worse over time. In the next video, we'll take everything we've built across this course and deploy it to a production runtime using Amazon Bedrock Agent Core.2. Final slide
Finally, deploy your harness to the cloud with Amazon Bedrock AgentCore.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.