Safety Evaluation
1. Safety Evaluation
Let's explore safety evaluation. Without understanding the safety criteria, it is not possible to design and implement a safer AI system. Numerous key considerations should be taken into account. Let's check the major failure modes and considerations we need to understand. First of all, you need to understand your product and users to define the failure mode for your product. Different products should have different definitions and goals related to some sensitive characteristics. However, there are some common failure modes you should prohibit for safety purposes. Generative AI systems should be designed to avoid providing child sexual abuse material or CSAM. Content on or from generative AI products must not include child sexual abuse material, nor content that appears to abuse, sexualize, endanger or otherwise exploit children. Personally identifiable information or PII and sensitive personally identifiable information or SPII of specific individuals. Generative AI should not reveal an individual's personal information and sensitive demographic information of specific individuals. Hate speech. Generative AI should not generate hate speech. Defined as content that promotes violence, incites hatred, promotes discrimination, or disparages towards any groups. There are many other common failure modes we usually need to avoid. However, the actual nuance and the threshold depends on actual products. To control the balance, it is also very important to design a system that we can flexibly control and adjust based on the needs and contexts of users. Because it is important to ensure that your generative AI model is safe, with respect to the failure modes, this is where adversarial testing comes in. Adversarial testing is a method for systematically evaluating an ML model with the intent of learning how it behaves when provided with malicious or inadvertently harmful input. Adversarial testing can help teams improve models and products by exposing current failures to guide mitigation pathways such as fine tuning, model safeguards or filters. Moreover, it can help inform product launch decisions by measuring risks that may be unmitigated, such as the likelihood that the model will output policy violating content. But what is considered malicious input and inadvertently harmful input? An input is considered malicious when the input is clearly designed to produce an unsafe or harmful output. For example, an input that asks the text generation model to generate a hateful speech about a particular minority group would be considered malicious input. This is also known as explicitly adversarial queries. These types of queries may contain policy violating language or express policy violating points of view in an attempt to trick the model into saying something unsafe, harmful or offensive. An input is considered inadvertently harmful when the input itself may be innocuous but elicits a harmful output. For example, an input that asks the text iteration model to describe a person of a particular ethnicity and the model provides a racist output. This is also known as implicitly adversarial queries. They can contain culturally sensitive or contentious topics and might include information on demographics, health, finance, or religion. When malicious or harmful outputs occur, it is important to study the output to learn more about how the model performs when being exposed to such inputs. This is accomplished through what is called an adversarial testing workflow. There are four steps in the workflow. Find or create a test dataset, run model inference using the test dataset, annotate the model output, and analyze and report results. To conduct effective adversarial testing, find or create a test dataset. You should not use standard evaluation data sets, as these are typically designed to accurately reflect the distribution of data that the model will encounter in the product. For adversarial tests, the key to creating or finding a test dataset is to design a test that pushes the model's limits. To accomplish this, you should focus on out of distribution examples and edge cases that could lead to outputs which violate safety policies. The dataset should comprehensively cover safety dimensions, real world use cases, and be both lexically and semantically diverse. Lexical diversity focuses on the variety of unique words used within a piece of text or speech. Semantic diversity focuses on a range of meanings and ideas expressed within a text or speech. Next, run a model inference using the test dataset. The goal here is to generate model outputs based on the test dataset. In certain cases, generating multiple model outputs per adversarial query can help. Third, annotate the model's outputs to identify any policy violations. There are two methods for doing this , automatically and manually. Automatic annotations use machine learning and other AI techniques to automatically label or tag data that aligns with specific policy categories and failure modes. Manual nations involve human raters, either internal or external, who label and tag the data. This is accomplished using their own specific instructions and platforms designed for this task. When should you use automatic nations versus manual? A good example is the notation of data that contains hate speech. Sometimes, automatic notation accuracy may be low for signals that try to detect constructs that are not strictly defined, as in the case of hate speech. In this situation, it is critical to use human raters to check and correct classifier generated labels for which scores are uncertain. The final step is to summarize test results in a report. You can pick any type of chart of your choice to effectively communicate results in a report to your stakeholders and decision makers. These results can guide model improvements and inform model safeguards such as the creation or improvement of filters.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.