Responsible AI Practices
1. Responsible AI Practices
for responsible AI. AI is software, and so general best practices for software systems should always be followed when designing AI systems. There are also various considerations unique to machine learning for us to discover. Here are six recommended practices for developing AI with responsible AI principles in mind. Use a human centered design approach and to defy multiple metrics to assess training and monitoring. When possible, examine raw data directly, have awareness of the limitations of your dataset and model, test the AI system to ensure it's working as intended. Monitor and update the system continuously after deployment. Let's discuss the first recommended practice. Use a human centered design approach. The way that actual users experience your system is essential to assessing the true impact of its predictions, recommendations, and decisions. To use a human centered design approach, you should design features with appropriate disclosures built in, clarity and control are crucial to a good user experience. Consider augmentation and assistance. In some cases, it might be optimal for your system to suggest a few options rather than one to the user. Technically, it's much more difficult to achieve good precision at one answer versus precision at a few answers. Model potential adverse feedback early in the design process, followed by specific live testing and iteration for a small fraction of traffic before full deployment. Engage with a diverse set of users and use case scenarios and incorporate feedback before and throughout project development. This builds a rich variety of user perspectives into the project and increases the number of people who benefit from the technology. The next practice is to identify multiple metrics to assess, training and monitoring. The use of several metrics instead of a single one will help you to understand trade offs between different errors and experiences. This means you should define metrics including feedback from user surveys, quantities that track overall system performance and short and long term product health. For example, click through rate and customer lifetime value respectively, and performance sliced across different subgroups. Ensure that your metrics are appropriate for the context and goals of your system. For example, a fire alarm system should have high recall, even if that means the occasional false alarm. Next, the practice of directly examining your raw data. Machine learning models will reflect the data they are trained on so analyze your raw data carefully to ensure you understand it. In cases where this is not possible, like with sensitive raw data, understand your input data as much as possible while respecting privacy. For example, by computing aggregate anonymized summaries, this means that data should be accurate. Ask yourself, does my data contain any mistakes, for example, missing values, incorrect labels, data, and data samples to be representative? Ask yourself, is my data sampled in a way that represents my users and the real world setting? Training serving skews shouldn't happen. The difference between performance during training and performance during serving is a persistent challenge. During training, you may need to adjust your training data or objective function. During evaluation, continue to ensure that evaluation data is as representative as possible of the deployed setting. Data and model should be simple. Ask yourself, are any features in my model redundant or unnecessary? Is my model unnecessarily complex? For supervised systems, consider the relationship between the data labels you have and the items you're trying to predict. Bias should be minimized in your data. First, you want to ensure that your data is fairly representative of the entire population. When you develop AI, you should have awareness of the limitations of your dataset and model. This means you should not mistake correlation for causation. For example, your model might learn that people who buy basketball shoes are taller on average. But this does not mean that a user who buys basketball shoes will become taller as a result. Communicate the scope of the training set. For example, a shoe detector trained with stock photos can work best with stock photos, but has limited capability when tested with user generated mobile device photos. Communicate limitations to users where possible. For example, an app that uses machine learning to recognize specific bird species might communicate that the model was trained on a small set of images from a specific region of the world. By better educating the user, you might also improve the feedback provided from users about your feature or application. Learn from software engineering, best test practices and quality engineering to ensure that the AI system is working as intended and could be trusted. This means you should conduct rigorous unit tests to test each component of the system in isolation. Conduct integration tests to understand how individual NL components interact with other parts of the overall system. Proactively detect input drift by testing that data distributions are not changing in unexpected ways. Use a gold standard dataset to test the system and ensure that it continues to behave as expected by updating it regularly. Conduct iterative user testing to incorporate a diverse set of users' needs in the development cycles. Apply the quality engineering principle of poka-yoke. The principle pushes you to build quality checks into a system so that unintended failures either cannot happen or they trigger an immediate response. For example, if an important feature is unexpectedly missing, the AI system won't output a prediction. Lastly, you should monitor and update the system continuously after deployment. Continued monitoring ensures that your model takes real world performance and user feedback like happiness tracking surveys and the heart framework into account. This means that you should be ready for issues to occur. Any model of the world is imperfect, almost by definition, build time into your product roadmap to let you address issues. Consider both short and long term solutions to issues. Balance short term simple fixes with longer term learned solutions. Analyze the candidate model before deployment, specifically how it differs from the deployed model and how the update will affect the overall system quality and user experience.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.