Model Interpretability

Feature importance, SHAP, and LIME

A prediction is only useful if you can explain why. Learn to open the black box and show stakeholders what drove each decision.

Before We Begin

What we are learning today

Imagine a doctor receives a diagnosis from an AI and asks "why?" If the model cannot answer, the doctor cannot trust it. Interpretability is the practice of making predictions explainable. We start with simple tools—feature importance from trees and linear coefficients—then level up to SHAP values, which assign each feature a fair share of the prediction, and LIME, which builds a tiny local explanation around any single prediction.

How this lesson fits

A model that works is not enough—it also has to be understandable, fair, and ready for the real world. This module covers the human side of AI: explaining predictions, confronting bias, and deploying models responsibly.

The big question

How do we make sure an AI system is not just accurate but also transparent, fair, and safe to deploy?

Explain a model's prediction using interpretability toolsIdentify sources of bias and propose mitigation strategiesDescribe the basics of serving, monitoring, and handling drift in production

Why You Should Care

Regulations like the EU AI Act increasingly require explanations for automated decisions. Beyond compliance, interpretability helps practitioners catch bugs, discover shortcuts the model learned, and build trust with non-technical stakeholders. A model you cannot explain is a model you cannot safely deploy.

Where this is used today

✓Banking: explaining why a loan was denied to comply with fair lending laws
✓Healthcare: showing doctors which symptoms drove a risk score
✓Hiring tools: auditing whether a screening model unfairly penalizes certain groups

Think of it like this

Think of a judge writing an opinion. They don't just say "guilty"—they explain which evidence led to the verdict and how much each piece mattered. SHAP and LIME are the written opinion for your model's decisions.

Easy mistake to make

Interpretability does not mean the model is simple. Even complex deep learning models can be explained post-hoc with tools like SHAP—the explanation method is separate from the model's complexity.

By the end, you should be able to say:

Distinguish global interpretability (overall feature importance) from local interpretability (one prediction)
Explain SHAP values as each feature's fair contribution to a prediction
Describe how LIME approximates a complex model locally with a simple one
Discuss when a simpler, inherently interpretable model might be preferred over a black box

Think about this first

A bank's model rejects your loan application. You ask "why?" and the bank says "the algorithm decided." Is that acceptable? What would you want to know?

Words we will keep using

feature importanceSHAPLIMEblack boxexplainabilitylocal explanationglobal explanation

Opening the Black Box

A doctor receives a diagnosis from an AI and asks "why?" If the model cannot answer, the doctor cannot trust it. Interpretability is the practice of making predictions explainable \u2014 not just accurate.

GlobalWhich features matter most across all predictions? (Feature importance, SHAP summary)

LocalWhy did the model make THIS specific prediction for THIS specific person? (SHAP waterfall, LIME)

Global Feature Importance

Across all loan applications, which features drive the model's decisions the most? This is the bird's-eye view.

Interactive: SHAP-Style Local Explanation

Select an applicant to see how each feature contributed to their specific prediction. Positive bars push toward approval; negative bars push toward denial. The waterfall shows how we get from the base value to the final score.

Prediction: DENIED (score: 31.7, threshold: 45)

Applicant #1

Income: $34272/mo

Credit Score: 502

Debt Ratio: 47%

Actual: Denied

SHAP vs LIME

SHAP (SHapley Additive exPlanations)

Based on game theory. Assigns each feature a fair share of the prediction, ensuring consistency and additivity.

Pros: Theoretically grounded, additive, consistent.
Cons: Can be slow for large models.

LIME (Local Interpretable Model-agnostic Explanations)

Perturbs the input slightly and fits a simple linear model around the neighborhood to explain one prediction.

Pros: Fast, model-agnostic, intuitive.
Cons: Can be unstable \u2014 different perturbations may give different explanations.

When to Choose Interpretable Models

Sometimes the best explanation is a simple model. In high-stakes domains (healthcare, criminal justice, finance), a transparent decision tree or linear model that stakeholders can read directly may be preferable to a slightly more accurate but opaque neural network.

EU AI Act increasingly requires explanations for automated decisions. A model you cannot explain is a model you cannot safely deploy.