Concept-based and example-based explanations
1. Concept-based and example-based explanations
Let's move on to the concept-based explanation. Concept-based explanations are a type of interpretability technique used to understand how machine learning models make predictions through higher-level concepts that are more meaningful to humans instead of individual input features. Let's say we have an image classification model that classifies the zebra class. If we can get an attribution score for user-friendly concepts, like stripedness or horse shapeness, that would be intuitive and informative. TCAV, or Testing with Concept Activation Vectors, stands as a pioneering approach in the field of concept-based explanations. It aims to provide explanations for arbitrary concepts, enabling a deeper understanding of how machine learning models utilize these concepts in their decision-making process. The TCAV methodology involves the following steps. Gather a user-defined set of examples that exemplify the concept of interest, such as strike patterns, along with a set of random examples. Collect, label training data instances representing the target class. For example, zebras, employed a trained target neural network to process the prepared concept data. In a feature space of an intermediate layer of the model, train a linear classifier to distinguish between the representations produced by the concepts examples and those from random examples. The concept activation vector is the vector perpendicular to the classification boundary and represented by the red arrow. For the class of interest, zebras, use the directional derivative to quantify the model sensitivity to the concept. By following these steps, TCAV enables the computation of concept attribution for a wide range of concepts, providing valuable insights into how machine learning models rely on these concepts for making predictions. TCAV is also useful for fairness testing purposes, such as ensuring that a class of CEO doesn't have a stronger concept attribution for male concepts over female concepts. TCAV provides flexibility regarding the choice of concepts. But in some situations, you might want to automate the concept selection process since creating a dataset for a concept can be complex. ACE, or Automatic Concept-based Explanation, offers this automation capability. In this approach, concept activation vectors are automatically created through the following steps. Split the target image into smaller crop segments, then rescale back to the original image resolution to ensure compatibility with the target neural network. Apply K-means clustering inside the feature space of an intermediate layer of the target neural network. Each cluster is regarded as an automatically created concept. Calculate concept activation vectors for each cluster to determine the attribution of each concept to the prediction. The calculation of the concept activation vector is the same as TCAV. Here are some output examples of the ACE method. Although we can't provide hand-picked concepts like stripedness or horse shapeness, it still gives us interpretable results. For example, it seems like the uniform logo and basketball surface are important concepts for the basketball class. Let's talk about the example-based explanation. The focus here is on explaining the model's results by looking at the training data. Example-based explanations is another interpretability technique that allows users to understand model behavior and how predictions are made. It relies on providing approximate nearest neighbor-based explanations. We can generate example-based explanations for multiple types of data such as image, tabular, and text. For images, example-based explanations will show you which other images in the training data appeared most similar to the new image that you want to classify. For example, for this image of a dog, the model is classifying it as a husky, and all similar examples in the training data were also labeled as huskies. This works very similar for tabular and text data. The focus is on explaining model results by showing the most similar examples from the training data. How can we compute the nearest neighbors? First of all, all neural network models create some embedding as intermediate representations of input regardless of the type, like image, text, or tabular. So to compute the nearest neighbors, we can investigate representation of a layer, usually a layer close to the output layer, but before the final output layer. First, compute embeddings of the training dataset. Then we can query the nearest neighbors by using a target image embedding. We can use this explanation in many ways. Let's say you have a classification model that misclassified this bird image as a plane. We used example-based explanations to retrieve other images in the training data which appeared most similar to this misclassified bird image for the model. Examining those, we identified that both the misclassified bird image and the similar images were dark silhouettes. This, in turn, signals a potential lack of images of birds with dark silhouettes in the training data, and an immediate action to gather more data with images of silhouetted birds in order to improve the model.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.