Using the AI_COMPLETE function
1. Using the AI_COMPLETE function
Hello again. In this video, we will use Cortex AI_COMPLETE function to build a multi-turn chat application. Let's get started. First, sign into your Snowflake account if you haven't already. Navigate to Projects on the left panel. Select Workspaces. Click on the notebook titled Using LLM Functions and select the Connect button at the top to start the notebook session. This will take a couple minutes. First, we import the active Snowpark session. Then we set the context for the rest of the notebook cell to use Ski Gear Support DB and Ski Gear Support Schema. Look at the code below using AI_COMPLETE. Here we are using Python to call the AI_COMPLETE function. As we saw before, AI_COMPLETE can be used with different models. In this case, we're using Llama 3.1 70 billion parameter model for its multilingual capability. If your application needs to interact with a wide range of languages, say French, German, Spanish, Italian, Portuguese, Arabic, Hindi, Russian, Chinese, Japanese, or Korean, you could use Mistral Large 3 or the Llama 3.1 70 billion model. If you're building a summarization, extraction, or question and answer use case involving lengthy documents or an extremely large knowledge base, it is useful to turn to a model with larger context windows, such as Jamba Large 1.7, which boasts an impressive context window of 256,000 tokens, or even a larger Claude Sonnet 4.6, which offers a context window size of 1 million tokens. For use cases requiring a low latency or high throughput that only need low to moderate reasoning, small models below 10 billion parameter size can be an ideal choice, such as the Llama 3.2 1 billion or 3 billion parameter models. Not all models are available in every region, but for this course, we are working in AWS US West 2, Oregon region, which supports the widest set of models. As always, check the Snowflake documentation for latest availability of models by region. Let's get back to talking about the arguments used by AI_COMPLETE. Model is a single string that represents the model AI_COMPLETE will use. The prompt or history can be either a column of prompts or a single string that are sent to the LLM. Here we're asking the Llama model how snowflakes get their unique patterns. Another thing to note is that AI_COMPLETE is stateless, which means it does not retain any state from one call to the next. In order to reference all the previous model responses and prompts, you must pass all the previous inputs as part of the prompt or history to provide a stateful conversational experience. These must be passed in chronological order for proper reference. This allows for a more complex multi-turn conversation that retain memory of the previous stages of the conversation. Please note that with each iteration, more tokens are being consumed and this can accumulate over larger conversations. It turns out that talk is not in fact really cheap. Now let's look at the system and user roles in AI_COMPLETE. When we call AI_COMPLETE, we must ensure that each role key has an associated context key. The two roles we look at are the system and the user. The system prompt is an initial plain English prompt to the large language model to provide it with background information and instructions for response style. For example, here we instruct the model to answer programming questions in a style of a rancher. If you ever provide a system prompt, there can only be one and it must be the first message in the array of messages. Then the user prompt contains the meat of the request for which we expect a response from the model. For example, here we ask the model about the role of semicolons in JavaScript. You will notice that we've also added additional parameters to our call to AI_COMPLETE using options. For now, we'll start with setting guardrails to true, which activates Cortex Guard. Once you activate Cortex Guard, large language models responses associated with harmful content such as violent crimes, hate, sexual content, self-harm, and others will be automatically filtered out and the model will return a response filtered by Cortex Guard message. Under the hood, Cortex Guard is currently powered by Llama Guard model from Meta and it works by evaluating the responses of a language model before the output is returned to the application. Now that we are utilizing options, we need to know the two key differences. First, when we utilize options, the prompt argument must be an array of objects representing a conversation in chronological order where each object contains a role and a content key. Second, when options are present, we will get a full response object from AI_COMPLETE. Next, let's look at how AI_COMPLETE can be used for multi-turn conversations in a chat. We mentioned before that although AI_COMPLETE is stateless, we can choose to pass in the conversation's history, which can enable multi-turn conversations. Let's look at the messages with history list. First, we start with a system prompt. Then we define a user prompt that is just the question the user asked the model. Next comes the model's response captured under the system role. This way, we can keep appending our question to the model and responses in the message with history variable and recursively call the AI_COMPLETE function to build a multi-turn chat application. For this multi-turn chat experience, it would be great if the user could read the output text similarly to how they would read a real conversation, word by word or in LLM speak, token by token. Another word for this is streaming. We can enable streaming with the keyword argument stream by setting it to true. When enabled, a generator function is returned that provides the stream output as it is received. Each update is a string containing the new text content since the previous update. Doing so dramatically reduces the perceived latency to the user because each token, as soon as it's generated, appears on the output screen rather than waiting for the entire response to be generated. Please note, streaming is enabled for Python only, not SQL. The reason for this is that SQL, as mentioned before, is meant to optimize for batch processing scenarios where latency of an engine response is not a concern. When we want a chatbot to feel more creative and interesting, we can turn to my favorite parameter, temperature and top P. Temperature is a value from 0 to 1 that we can pass to the model to control the randomness of the output. More technically, a low temperature makes the model more confident in its top choices. Let's try it by adding a high temperature to our same chat with the Western Rancher about JavaScript. Oh, the response is a little bit more fun. As an alternative to using temperature, we can also use top P. Top P achieves similar results, but in a different way, by restricting generated tokens to those fitting in a cumulative distribution threshold. Moving from chat experiences to more task-based actions, it can actually be useful to constrain the number of tokens an LLM can generate. If you want to limit the number of output tokens, we can use max tokens. This can be very useful where we need to constrain the output, but we should be aware that small token values may result in truncated responses. An example of this would be where you want the LLM to only respond with either a number of 0 to 5 or only words yes or no. In many cases, when we are asking the LLM to perform constrained tasks like this, we want to do so in a batch. The easiest way to use an LLM in a batch is through SQL. In this example, we use AI_COMPLETE to critique an entire column of transcripts in a Snowflake table called Call Transcripts. Now let's talk about the output of AI_COMPLETE. When we specify options, AI_COMPLETE will return a JSON string containing choices, created, model, and usage. The choices key return an array of responses. Messages hold the generated responses. It also contains three subkeys under usage, completion tokens, prompt tokens, and total tokens used for this completion. We saw earlier that Cortex AI_COMPLETE supports a number of foundation models. However, what is the right model to use for your use case? We recommend you start with the large model to learn the art of possible, then scale down to find the smallest models that satisfy your latency and compute budget. Smaller models can be really useful for a simpler, more constrained task. Only when we need to perform more complex tasks or handle large amounts of context do we turn to state-of-the-art models. If we choose a model that is too small of a context window, we may result in a context overflow. Let's review what we've covered. We looked at Cortex AI_COMPLETE, which generates a response from a given prompt and a choice of an LLM. We looked at how we can use a series of options to guide the output, how to pass prompts, and how to build stateful applications. We talked about what to consider when choosing models, options like max tokens, temperatures, and top P, and cost considerations. In our next video, we'll get some hands-on experience using task-specific functions that Cortex offers.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.