Iterating Based on Monitoring Insights
1. Iterating Based on Monitoring Insights
You now know how to read traces. Now, let's use them to improve your agent. The pattern is simple. Spot an issue, update instructions, retest, verify the fix. Let's walk through a real example. In the last video, we found an issue. When asked about sentiment in lost deals, the agent searched conversations without first identifying which deals were actually lost. It should query metrics first, then search conversations for those specific deals. Let's fix this. The issue is in how the agent plans multi-step queries. It needs clear guidance on when to combine tools in a specific order. I'm in Snowflake. Navigate to AI and ML, then Agents. We'll click on our Sales Intelligence agent. And let's go ahead and edit. Navigate to the Orchestration tab. We need to add guidance for questions that require sequential tool use. Find the section about tool selection. Now, we're going to add this instruction. For questions that filter unstructured data by structured criteria, always query the Sales Metric View tool first to identify the relevant records. Then use those results to guide your sales conversation search tool query. For example, if you're asked about lost deals, first query metrics to find deals where the win status is false, then search conversations for those specific customers. This instruction is specific. It describes a pattern, gives a rationale, and provides an example. Specific instructions produce consistent behavior. Click Save. Now, let's test the fix. Go ahead and open your agent in Snowflake Intelligence. Ask the same question. What was the sentiment in our lost deals? Now, let's wait for the response. Now, let's go back to monitoring. And let's give it a refresh. Now, let's find our new interaction and open the trace. Let's look at the first planning span. The agent now plans to use Cortex Analyst first to identify lost deals. Then, after it finds those details, it actually goes back into a planning phase to go ahead and use Cortex Search. That's exactly what we wanted. Check the tool execution status. If it's not working, go ahead and use Cortex Search. Tool execution spans. The first span is Cortex Analyst. The SQL filters for deals where win status equals false. And this is what returns small biz solutions as a lost deal. Then, we go into the planning phase for it to understand it needs to look at qualitative data. And then, it uses Cortex Search to find that information. The search query now specifically targets small biz solutions conversations. It's not doing a generic search for lost deal sentiment. It's searching conversations for the customer we know actually lost. The response is now more accurate. It discusses specific concerns from the small biz solutions conversations, which is the deal we actually lost. Before, it might have included sentiment from deals that were still pending or even won. That is the iteration loop. Spot the issue in the trace, update the instructions, retest, verify the trace, and that's one cycle complete. Let me show you another common pattern. Sometimes the issue isn't tool selection, it's response quality. Let's look for one of those. Let's go to Snowflake Intelligence once again. Let's ask, give me a quick summary of our pipeline. Now, let's check the response. It might be longer than you want for a quick summary. Let's go check the trace. Let's go ahead and refresh the page. So, let's check the trace. It's not doing a generic search for lost deal sentiment. So, let's check the trace. The planning seems fine. It wants to give us a quick summary. Tool execution seems fine as well. The issue is response generation. The agent retrieved good data, but didn't respect the quick part of our request. This is a response instruction issue, not an orchestration issue. Go back to edit, scroll to response instructions. We're going to add, when users ask for a quick, brief, or summary response, limit your answer to three or four sentences. We'll head back to Snowflake Intelligence. Let's start a new chat. Again, we're going to ask, give me a quick summary of our pipeline. The response should be more concise now. Let's go ahead and check the trace to confirm the response generation span, showing a shorter output. When you are going through and iterating your agents, here are some areas to identify issues and where to fix them. Wrong tool selection, faulty tool selection, and incorrect tool selection. Let's go back to Snowflake Intelligence. We're going to ask, give me a quick summary of our pipeline. The response should be more concise now. Let's go ahead and check the trace. The response should be more concise now. When you are going through and iterating your agents, here are some areas to identify issues and where to fix them. If it's a short answer question, fix the orchestration instructions. If it's a poor response format, fix with response instructions. If it's mixing clarification, fix it with a boundary instruction. Monitoring reveals which type of fix you need. I have one more tip. Keep notes on the issues you find and the fixes you make. Over time, you'll build a library of instruction patterns that work for your use cases. The agents you build will never be perfect on day one. That's normal. What matters is having a systematic way to improve them, and monitoring gives you visibility. Instructions give you control, and together, they let you iterate towards reliability. In the next video, we'll shift from evaluation to deployment. You'll learn how MCP, or Model Context Protocol, lets you make your agent accessible with tools like Cursor or Cloud Desktop. [2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.