What is Cortex Fine-tuning?
1. What is Cortex Fine-tuning?
What is Cortex fine-tuning? Welcome back! Let's talk about what we're going to do in this video. By the end of this video, you'll be able to fine-tune a large language model to accomplish a new task. You can fine-tune an LLM to respond in a new style which are not possible with an out-of-the-box LLM. You will be also able to distill the behavior of a large model to a small model so that way you get the same performance at a lower cost. Excited? I am. Now, you might be thinking, I can ask all sorts of questions to ChatGPT and other models every day and get the right answers back. If these models work just fine, why do we need to fine-tune them? Well, general-purpose large language models such as GPT are great, but they're just that, general purpose. Imagine a high school student who has a broad education covering a wide range of subjects such as mathematics, science, history, and languages. This student has a general understanding of many areas, but is not an expert in any specific field. Now, picture this student deciding to pursue a specialized career such as becoming an electrician or a chef. General-purpose LLMs are similar to this high school student with a broad knowledge. They've been trained on vast amounts of data and can provide information and answers across various subjects, but they're not specialists in any particular area. Fine-tuning these LLMs is similar to sending that high school student to a trade school. At that trade school, the student focuses intensely on a specific skill set as electrical work or culinary arts. Through the specialized training, the student becomes highly proficient in that particular field, much like how fine-tuning an LLM allows it to become highly skilled and accurate in a specific domain or task. Now, just as a trade school student can perform complex tasks on their chosen field with greater precision and expertise, the fine-tuned LLM can handle complex tasks within its specialized area with much higher accuracy and effectiveness. That makes sense, right? Say, occasionally, these models don't give me a straight answer to my questions, so I have to word the question differently to get the right answer. Can we simply experiment with these different prompts to get the model to respond the right way? Why would we fine-tune? Well, you can certainly play with different prompts. This approach can be effective in some cases. However, fine-tuning allows models to learn from new examples and potentially acquire new knowledge not part of their original training set. In contrast, prompt engineering limits models to their original knowledge and capabilities and cannot introduce new knowledge. In addition, the time involved in crafting different prompts and experimenting them is a factor as well. Having said all that, the case for prompt engineering versus fine-tuning can depend on several factors. How highly specialized and domain-specific is the task? What are the accuracy requirements? What is the level of customization needed? And more. Once you decide to fine-tune is your way forward for your specific use case and requirements, this video is how you can fine-tune those general-purpose LLMs using your own data using Snowflake. To run fine-tuning in Snowflake, we use Snowflake's Cortex fine-tuning function. Cortex fine-tune is a fully managed service which lets us take a general-purpose LLM and customize them to our specific needs. As mentioned above, fine-tuning a model allows us to train the model so that it can take complete new tasks, respond in a particular format, or use specific tones or styles. For example, you can teach a model to use a particular brand voice or always generate a JSON with a particular set of keys. Another reason we might want to fine-tune our LLM before deploying it is to achieve a more reliable output formatting that matches the expected or required format. This is great if you want to have consistent email formatting or information that is an output in a bullet or table form for reports, or a specific format such as JSON. Consider an example where we need to extract a condition and intervention from detailed medical notes. We carefully construct the following prompts shown here, which asks to extract only conditions and interventions from medical notes. When we ask this of the Llama 3.1, we get exactly the response we're looking for, containing only information we need. However, if we need to deploy this extraction for an entire health system, it will be cost prohibitive to use the 405 billion parameter model. Cortex has a full range of sizes for LLMs, so we could try a 1 billion parameter model here and say a Llama 3.1 8 billion model. However, the 1 billion parameter model won't even attempt to do the extraction. To step up, we can also try Llama 3.1 70 billion model. This bigger model is more helpful, but it ends up extracting a lot more information than we need and is too verbose. It is a perfect case for fine tuning because we want to teach the small model to respond in a particular format, just like the large model. While this particular combination of models may end up with one being too small or other being too large, models are constantly updating and you might find a model with just the right amount of parameters for your use case. The current list of support models can be found in Snowflake's documentation. Another word for processing of teaching a small model to behave like a large model is distillation. Just like a chemical distillation where we can run a process to separate out the compounds, you want to form a larger volume of liquids. We distill the capability from a large model to a smaller model, so that way we get the benefits of the larger model, but the cost of running the smaller one. Now imagine the scenario from earlier where we need to extract information from medical notes. The large model worked well on our medical notes. This is great for prototyping. In the real world setting, when you want to extract information from millions of medical notes in production, the cost would blow up. With scenarios like this, it's a good case for using fine-tune function to distill functionality of a large model into a small model. Large models are better for high accuracy, low volume tasks, since they are more expensive to run. For high volume tasks, by fine-tuning a smaller model, we can achieve the performance as close to a larger model while training it for a very specific use case at a much lower cost and latency. With fine-tuning, however, we would incur the fine-tuning expenses upfront, but we can enjoy the lower ongoing costs, improve performance, and latency that makes it worthwhile. Everyone is happy again. Phew! Well, let's move on to talk about how Cortex fine-tuning function works. Cortex fine-tuning function leverages parameter-efficient fine-tuning, or PEFT, under the hood. Parameter-efficient fine-tuning is a technique that improves performance of a pre-trained model for a specific task by fine-tuning only a small subset of its parameters. It is a good choice when we need to adapt a model for a domain-specific task while keeping costs and resources low. This technique reduces the number of parameters being tuned and allows you to use examples to adjust the behavior of the model and improve the model's knowledge of domain-specific tasks. Sounds great, right? Let's quickly consider hardware requirements when dealing with LLMs. Due to the enormous size, fine-tuning large LLMs such as the Llama 3.1 405 billion model entails intensive hardware requirements that need innovative approaches to reduce the memory load to train billions of parameters. But using Cortex fine-tune leverages parameters-efficient fine-tuning. That process is much cheaper and faster. Let's quickly touch on the models that Cortex fine-tune supports. There are a number of models from the Llama and the Mistral families that range in size. These can be fine-tuned. Note that the models available for fine-tuning will change over time. Some models may be available in some regions, but not in others. So check the Snowflake documentation for the latest availability. Let's look back at what we covered in this video. We looked at how fine-tuning a pre-trained LLM can accomplish a new task or respond in a new style in a way an out-of-the-box LLM cannot. We also looked at how we can distill the behavior of a large model to a small model on how it produces responses with quality of the large model while getting the lower cost benefits of the small model. We talked about how ability to tune a model to respond in a specific style or format such as JSON can be very useful in certain cases. We covered how Snowflake Cortex under the hood runs fine-tuning using parameter-efficient fine-tuning techniques. And it reduces the compute costs and why this is much more efficient. In the next video, we will prepare a Snowflake environment to fine-tune a foundational model for a specific use case. See you in the next video.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.