Model Traing for Safety: Instruction Fine-tuning
1. Model Traing for Safety: Instruction Fine-tuning
Let's take some time to explore model training for safety and discuss what you can do to tune the model itself. Although harm prevention is important to mitigate unintentional behavior and related risks, it provides evasive methods that can reduce AI's helpfulness and capabilities. How do we train an AI model that more closely aligns with our safety values while at the same time, ensuring it still retains AI helpfulness and capabilities. The first simple idea is to filter the training data for safety purposes. By filtering toxic data out from training datasets, you can naturally reduce the possibility that the generative AI model generates harmful output. You can achieve this by using safety classifiers to filter through the data. However, while this approach works for reducing toxic outputs to some extent, it also causes another issue. By limiting the access to toxic sentences, the model may result in poor performance, especially for tasks related to underrepresented groups. This is partially because the automated filtering methods cause false positives, especially for marginalized groups. This results in a reduction in the ability of the model to generate text about these groups, even in a positive way. In this case, there is a trade-off between safety and fairness. Instead of removing toxic data, can we actually teach the concept of safety via a fine-tuning process? While it is still an emerging research topic, there are several known methods, including instruction tuning and reinforcement learning from human feedback, or RLHF. Instruction tuning is a generic fine-tuning method that uses a collection of datasets described via instructions. Incorporating safety-related datasets and instructions allows you to embed the safety concept into the model. Another approach is to use RLHF, this technique optimizes language models directly from human preference feedback. By providing safety-related feedback, you were able to help make the model more aligned to human values. Let's look a bit deeper into the instruction-tuning. Large language models are trained with massive pre-training datasets and pre-training tasks. These tasks are mainly designed to acquire general capabilities, handle language. Instruction-tuning usually follows the pre-training phase and aims to teach more task-related capabilities using a collection of specific test datasets, such as translation, reasoning, question answering, and so on. Each task has different inputs and outputs. For example, in the translation task, the target model gets instruction inputs as a prompt, such as translate this sentence to Spanish. The new office building was built in less than three months. Then it is supposed to provide an output accordingly. The model is trained based on the gap between the responses and the prescripted target outputs. You can include safety-related tests, such as toxic language detection in this instruction-tuning phase. During the tuning process, the model is asked to detect whether a given text has toxic meaning by replying toxic or non-toxic. This approach helps detoxify models significantly from testing. As shown in this graph, from a paper, you can see instruction-tuned PaLM models labeled as Flan-PaLM have a lower probability to generate toxic sentences even based on a prompt that elicits toxic continuation or response than the base PaLM model.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.