Overview of interpretability techniques

1. Overview of interpretability techniques

>> Lets talk about the role interpretability plays in understanding model behaviors. Here is a classification model of a husky dog. Now the machine learning or ML model provides a smart prediction, but how do we understand the reasoning behind it? If our model is very simple like a linear regression model, it is not difficult to understand the reasoning. If we can assume input features of Xs are numeralized, we can simply inspect each coefficient of W. Since linear regression involves a weighted sum of the input features, the magnitude of the coefficients corresponds to the relative importance of those features. For instance, in image classification, we can analyze the coefficients associated with each pixel. However, for complex tasks like image classification, we typically don't employ such simple models. Now, Deep Neural Network or DNN models such as convolutional neural networks or CNNs and transformers generally outperform linear regression models in image classification tasks. This is because DNNs can extract more intricate relationships between the vast number of pixels in an image. It does this by using many weight parameters. However, the increased complexity of DNNs makes it more challenging to comprehend their behavior in a way that allows for straightforward interpretation. Although BNN is a very popular and common approach, their widespread adoption is hindered by their black box nature, making them less interpretable than simpler models. This inherent tradeoff between complexity and interpretability poses a challenge in understanding the reasoning behind DNNs. It should be noted that complexity doesnt always lead to better performance. You should always consider using simple models when the performance gain of using a more complex model is small. To help understand very deep and complex models, a lot of modern interpretability techniques exist. There are many ways to categorize different types of interpretability techniques, so we classified them into several subcategories. Every interpretability technique first falls into one of two intrinsic or post hoc. Intrinsic is the ability of a model to be directly interpreted by examining its structure or learning process. This means that the models internals are inherently transparent and can be understood without the need for additional tools. This approach is only possible for simple methods such as linear regression models, where you merely need to look at the trained weights for each feature, decision forests, bayesian networks, and so on. Note that some models still require sophisticated visualization to become interpretable. Next, post-hoc. Post-hoc refers to methods that are applied after a model is trained. These methods aim to provide insights into the model's behavior and explain its predictions. This approach is necessary for non intrinsic models, which are often more complex, and it provides a consistent way to assess interpretability across different model types. Post-hoc methods can be applied to intrinsically interpretable models too. So intrinsically interpretable models remain among the most popular interpretability methods because they are fairly easy to understand. At the same time, care should be given when describing these models to stakeholders. Why? First, if your model isn't complex enough to accurately describe the underlying system and fit the data well, explanations aren't going to be useful because the models are not useful. Second, ML models learn patterns and correlations, not causal structures. Third, model internals and interpretable ML methods tell you about your model, not necessarily your data or the underlying data generation system. And lastly, heap explanations to claims about the model only. Be careful making inferences on underlying populations of interest. Intrinsic explanations vary in their characteristics, particularly in terms of linearity, monotonicity, and the ability to consider feature interactions. For example, intrinsic explanation of linear regression is linear and monotone, but it doesn't consider interactions because we simply look at coefficients of each feature. So now let's focus more on post-hoc techniques, which are more flexible in terms of model selections. Post hoc interpretation techniques can provide either local or global interpretability. Local techniques provide explanations for each data point. We could ask the model, why do you classify this image as a husky dog? Using a local technique, we can then delve into the reasoning behind a specific prediction for a particular data point. Global techniques provide more aggregated explanations, referring to the entire model's prediction space. Using a global technique, we can ask the model, which feature do you prioritize most in general? Global methods provide a more complete map of the prediction space, but can fail to accurately capture the nuances of individual predictions. It can also fail to effectively explain the model's behavior in specific regions of the prediction space. Now let's talk about the terms model agnostic and model specific. Both local and global interpretation methods fall into one of these two barriers. At a high level, model specific methods use the internal details of the model, whereas model agnostic methods examine the model's behavior by manipulating the data fed into the model. Model agnostic interpretability methods don't rely on model internals. Instead, they analyze how changes in input features affect the model's output predictions. This approach allows these techniques to be applied to a wide range of machine learning models. Model specific interpretability method is restricted to usage on specific types of machine learning models. For example, some techniques rely on the gradients of a neural network and can only be applied to differentiable models like deep neural networks. Interpretability techniques can be categorized based on the type of output they produce. Here's a breakdown of the major categories based on output types. Although this is not an exhaustive list, it covers major categories. Most of the major interpretability methods fall into the category called feature-based explanations. These provide attribution scores, masks, or some visualizations that highlight the importance of each input feature for a given task. For example, feature-based explanations might indicate which features are most relevant in classifying an image as a husky dog. Concept-based explanations are similar to the feature-based explanation, but it outputs relative important scores to broader different concepts instead of each feature. These concepts, such as long nose-ness or white fur-ness, are designed to be more intuitive and interpretable for humans. Example-based explanations provide very different types of output. Instead of returning the importance of feature or concept explanations, it provides data points that our target model thinks is similar to a given query data point. This kind of output is also useful to understand how models interpret data and extract meaningful embeddings from the input.

2. Let's practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.