Identify Bias - TFDV Tool
1. Identify Bias - TFDV Tool
Now lets talk about how to identify bias problems in AI systems. There are many possible causes of biases in AI systems. Lets look from these two perspectives. Data and model bias identification always starts with asking good questions. For example, be careful about whether the dataset has proper data distributions. Is it skewed towards a subgroup of the population? Does it have selection bias problems? Also, you should be careful about the proportions. Even if a dataset contains a sufficient number of data points for the minority population, the label proportion can be different from the one of the majority population and you can check for bias issues in different ways. Even after training the model. You can check the model performance in different subsets of the dataset to verify if it doesn't have any harmful biases. You can also check if it doesnt unnecessarily benefit or harm a specific population. To help us identify bias issues, we can leverage a few open source tools TensorFlow data validation is a useful tool for exploring and validating ML data at scale. The what if tool by Google helps you investigate your data and model bias through data and prediction visualization. And helps you apply counterfactual analysis and TensorFlow model analysis is designed to analyze model behavior by applying granular model evaluation. Although the focus here is on the TensorFlow ecosystem tools, the tools mentioned are not limited to the TensorFlow library and are compatible with other frameworks. More importantly, the idea itself is universal, so you can find other alternatives. If you prefer other libraries, you can even implement an alternative on your own after you feel you properly understand what to do. Lets start from TensorFlow data validation or TFDV. TFDV helps you understand data distribution and statistics. Although this tool is not developed for fairness purposes only, it is always the starting point to help you investigate. For starters, with TFDV you can easily calculate summary statistics from a dataset. Although this code example shows you the TF record format which is tensorflows own super efficient binary storage format. You can also directly use a CSB file or a dataframe. All you need is a path to your data. With TFDV you can also calculate statistics by data slices. For example, you can slice data by country and state to help understand data proportion differences in various locations. After you calculate your stats, simply visualize them by using the method. Visualize statistics the dashboard contains two tables, one for numeric features and one for categorical features. The dashboard also has a control panel. The tables and dashboard are separated due to different information being shown for the two data types. Now in the table, a row corresponds to a feature. It contains some calculated statistics about the values of that feature across the entire dataset, along with a number of charts to show the distribution of values. For numerical values, TFDV calculates mean, standard deviation, number of zeros, and the min, median, and max values. The tool automatically tags some stats that might be problematic by bolding them and coloring them red. For example, if a feature had a high number of zero values, also known as null values, it might be automatically tagged. For categorical features, you are given the number of missing values, number of unique values, top or most frequent category, frequency of the top category, and average strength length. Notice this new screenshot and how it is different from the previous screenshot here. Within the same view, you can see a direct comparison of multiple data slices represented by blue and orange bars. With TFDV, you can also automatically infer the schema given stats. This is particularly useful for datasets with lots of features where manually defining the schema can be a tedious task. Instead, automatically generate it using infer schema, then refine as needed. This schema is especially useful when you want to automate data validation steps in an automated mlops pipeline. After defining the ideal characteristics of the dataset, you can automatically check the data issues from the next mlops pipeline execution. Lastly, TFDV lets you easily detect and act on anomalies by using the schema defined. Anomalies can be out of domain unexpected features, a skew of the entire data set, or a proportional gap between multiple data slices. By using this capability in a machine learning pipeline, especially in an automated system, you detect data problems before you train models. Again, although TFDV offers useful functionalities if you prefer, you can also develop a similar data validation system by using different libraries.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.