Model Traing for Safety: RLHF
1. Model Traing for Safety: RLHF
RLHF also tries to embed the concept of safety by optimizing the model using human feedback. RLHF works in the following way. It starts from training a reward model that returns preference scores on a given prompt. To train a reward model, The first step is to generate sets or pairs of responses using an LLM, which can be the target model itself, or another model. Then the responses are passed to a human moderator who ranks them or simply judges which response is preferable. The reward model is then trained using this human preference data set for the next step. For safety purposes, you would usually use a set of input prompts that are picked or designed to elicit harmful responses and the human moderator would evaluate not only the helpfulness of the responses, but also their harmlessness. In the next step, the target model, you want to tune receives prompts and then generates responses accordingly. The trained reward model evaluates the responses and returns the preference score, which is supposed to reflect the human preference it was trained on. Here, the target model is iteratively trained via a reinforcement learning strategy to internalize the human preference through the reward model. By using special prompts for safety and carefully building a human preference dataset for safety with human moderators, you can then embed the safety concept into the model. RLHF is a widely used technique for generated models. But at the same time, you may need to start thinking about how to scale this manual supervision process as AI models acquire general capabilities. Since human supervision can't be perfect and as AI grows, at some point it may start leveraging human incapability to hide undesired outcomes in supervision. Researchers are trying to find a way to scale the supervision approach by using AI in the process. One of these attempts is constitutional AI released by Anthropic. Constitutional AI uses AI evaluation in two types of tuning, supervise and reinforcement learning based tuning. Supervised learning incorporates the process of self critique and revises responses by AI itself. The revised responses are used later as supervised learning targets to fine tune the model. In the reinforcement learning process, AI itself works as the moderator. AI is used to evaluate and create the preference dataset that trains the reward model. Hence, this process is called reinforcement learning from AI feedback or RLAIF. In the entire tuning of constitutional AI, the only human oversight provided is through a list of rules or principles as a form of prompt for self critique and supervised learning and for self evaluation in reinforcement learning. The alignment of AI for safety is still a hot topic and as AI evolves, there will continue to be many interesting ongoing discussions on this.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.