Harms Prevention

1. Harms Prevention

Let's learn some methods to address harm prevention. Harm prevention means to avoid showing harmful content, which may still be generated despite best responsible generation efforts. Input and output safeguards are essential aspects of AI safety that play crucial roles in preventing harm. They act as protective measures to guard what goes into an AI system, as an input, and what is produced by the AI system, as an output to ensure that AI behavior aligns with safety standards and ethical principles. Machine learning-based classification systems that classify whether an input text is safe or not, it's called a safety classifier, while it is technically possible to build an original safety classifier by yourself. Practically, it is a challenging task since a safety classifier needs to be trained carefully with specially consolidated datasets and with special care and attention towards fairness. Fortunately, there are a few safety classifiers available in the AI industry to help people prevent outputting harm from a model. In 2017, Google's Jigsaw team released prospective API, one of the first safety classifiers in the world. Prospective API is designed to help flag harmful speech that pushes people out of online spaces, pushes the towards violence, or even worse. By the time 2021 came around, prospective API reached around 500 million requests daily due to high usage in many online platforms. Currently, prospective API is widely used for input safeguards to guard input for the AI and for output safeguards to monitor output. While prospective API was one of the first safety classifiers, safety classifiers now exist from other organizations, including moderation API from OpenAI and Llama guard from Meta. May people think of block lists when talking about input safeguards. In very simple situations, block lists can be used to identify and block unsafe prompts. However, this method is brittle and often not sustainable over time. A better long-term solution is using classifiers to tag each prompt with potential harms or adversarial signals. Then you can apply different strategies on how to handle the request based on the type of harm detected. Strategies included to block, rewrite or let the request pass through. If the input is overtly adversarial or abusive in nature, the system can block it from being sent to the model, and instead, output a pre-scripted response. For example, if you ask a chat bot how to rob a bank, the classifiers will recognize this input as a harmful prompt and suggest the system to block it. You will get a reply. I cannot help you with that from the chat bot. Along with why it is a wrong idea and a national support helpline to reach out to, if you believe you are in a difficult situation. If the input is likely to elicit an unsafe response, then prompt engineering, control tokens, or style transfers can be used to attempt to steer the model towards safer generation. Depending on how much the AI model has already been safety tuned or custom tuned for the application use. You can pass through the input with toxic data and trust the model to safely handle the generated output. A well tuned model can produce a more capable response which can be more insightful than the prefix errors or messages. It's worth emphasizing that understanding what kind of toxic data your safety tuned AI model can take is very important in this approach. Even if the input safeguards are in place and the model has been safety tuned, the model may still output unacceptable content. This is why output safeguards are equally important. With output safeguards, you can use classifiers to detect harmful output and perform one of the following actions on the output. Provide an error message, provide a pre-scripted output, reiterate another response with ranking. If the classifier determines that the output is harmful without a doubt, you can simply configure the system to produce an error message in order to avoid delivering responses that are unsafe, bias, or offensive. Using a chat bot as an example, if your Corti contains such content, it will output an error message or refusal to respond. We can also configure the system to output a pre-scripted or semi-scripted output. Sometimes a semi-scripted output conveys more information than a simple error message. Semi-scripted outputs are generated by the AI model to provide a certain level of information on why it is unsafe. For example, instead of delivering an error message, the AI model can output, I cannot help you with that because the information about the weapon can be used to harm people and it violates our policy. Since most generative AI models are designed to be able to create multiple outputs for the same prompt, sometimes it is helpful to use this capability to handle risky outputs. When an unsafe output is detected, we generate multiple additional answers using generative AI and rank them based on the safety score. The model then replies with the output that has the highest safety score. It's worth emphasizing that the final output needs to meet all safety criteria. Meanwhile, there are also some safety fairness trade offs to be aware of. Classifiers are trained on large datasets to learn what is harmful and what is not. An example of this is using large amounts of comments on Wikipedia discussion pages to train a classifier while having a group of workers to review each one for attributes like harassment or personal attack. However, human judgment of toxicity is inherently biased and often doesn't represent the full spectrum of harassment. Even with a large and robust dataset, the model may not always learn the right patterns from it. One concern is around protecting underrepresented groups. Many LLM developers are using classifiers with low toxicity thresholds to remove hate speech, meaning they are flighting more things as toxic when the model is uncertain, which increases the risk for false positives. This may end up eliminating the LLM's ability to say things about underrepresented communities in ways that reinforce negative effects on the historically disadvantaged groups. The performance of classifiers around non-English prompts is important. The models are also more likely to interpret innocent non-English phrases as hate speech and harassment. When hate speech is written in veiled ways, such as with slag or in non-English languages, classifiers can be easily fooled. Machine learning models will always make some mistakes, so it's essential to build in mechanisms for humans to catch and correct accordingly. Having a human involved in a process instead of full automation, also known as human in the loop is always important and highly recommended despite the fact that you can employ classifiers to automate input and output bar railing. Integrating human oversight into the validation and review of AI outputs is important for critical or high risk applications. This allows humans to intervene if necessary and provide feedback to the AI system.

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.