Local explainability with LIME

1. Local explainability with LIME

LIME is another tool designed for local explainability.

2. LIME

LIME, which stands for Local Interpretable Model-agnostic Explanations, is specifically designed to explain the predictions of complex machine learning models. As the name suggests, this technique offers understandable explanations on a local, per-instance basis, making it agnostic to the type of model being analyzed.

3. Lime explainers

There are several types of LIME explainers, each tailored to different kinds of data: LIME Tabular Explainer for structured data,

4. Lime explainers

LIME Text Explainer for textual data,

5. Lime explainers

and LIME Image Explainer for image data. These explainers generate perturbations within the dataset around a specific instance to see how these changes affect the model’s output. They then use these variations to build a local model, which is a simplified model focused on just that instance. This local model helps to explain how individual features influence the prediction for that particular instance, rather than the entire training dataset.

6. Lime explainers

In this video, we will focus specifically on the LIME Tabular Explainer, which is used with tabular datasets, such as those in CSV files or dataframes.

7. Admissions dataset

Let's explore how to use LIME to explain two models trained on the graduate admissions dataset. A regressor that predicts the likelihood of admission and a classifier that determines whether a student will be admitted. While here we examine both models for easier comparison, in real-life applications we would choose one based on the specific data and question at hand. The features used by these models are stored in dataframe X.

8. Creating tabular explainer

To work with a tabular dataset, we start by importing LimeTabularExplainer from the lime.lime_tabular module. Next, we select an instance to explain. We use the same instance for both regression and classification. We initialize the explainer using X.values to pass the actual data values as an array, and X.columns for the feature names. For the regression task, we set the mode to 'regression' to indicate that we are predicting a continuous value. For the classification task, we switch the mode to 'classification'. We generate an explanation using the .explain_instance method, providing the instance to be explained as an array through instance.values, and the model's prediction function, which is regressor.predict for regression and classifier.predict_proba for classification.

9. Visualizing explanation

The explanation.as_pyplot_figure() method generates visual representations of explanations. Both regression and classification models show that higher GRE and TOEFL scores increase admission chances, while lower SOP, University Rating, and CGPA scores decrease them. In practice, the choice between these models would depend on the specific scenario.

10. SHAP vs. LIME

Contrasting this with SHAP, LIME's local explanations show the specific conditions and ranges that detail how features influence an individual prediction, enhancing transparency and trust in model predictions at a granular level. In AI, this level of insight is crucial for ensuring that complex models can be trusted.

11. Let's practice!

Time for some practice!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.