Interpretability

Interpretability connects a fitted model’s behavior to features, examples, and decisions. A decision tree is directly readable as rules; a linear model exposes coefficients; a random forest usually needs post-hoc tools such as permutation importance or partial dependence.

Why interpret a model

Interpretability is not decoration after evaluation metrics. It is a debugging tool: which features drive predictions, where does the model rely on shortcuts, and which examples are near a decision boundary? Different questions call for different tools:

Method	Scope	What it answers
Model coefficients	global	the direction and size of each feature’s linear effect
Permutation importance	global	how much the model relies on each feature
Partial dependence	global	the average effect of a feature across its range
Local attributions (SHAP-style)	local	why this one prediction came out as it did

Measuring feature importance

Permutation importance for feature $j$ measures how much the model’s score drops when that feature’s link to the target is destroyed:

$$I_j=S(\hat{f},X,y)-S(\hat{f},{\pi }_j(X),y),$$

where $\hat{f}$ is the fitted model, $X$ the feature matrix, $y$ the labels, $S$ a chosen score (such as accuracy or $R^2$), and ${\pi }_j(X)$ is $X$ with the values in column $j$ randomly permuted. A large drop $I_j$ means the model relied heavily on feature $j$. For additive local explanations, many methods approximate the prediction at a point as $\hat{f}(x)\approx {\varphi }_0+{\sum }_j{\varphi }_j(x)$, where ${\varphi }_0$ is a baseline value and each ${\varphi }_j(x)$ is the contribution attributed to feature $j$.

Worked example

This snippet fits a random forest on Iris data and uses permutation importance to estimate which features the fitted model relies on most.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
import numpy as np
 
X, y = load_iris(return_X_y=True)
rf = RandomForestClassifier(n_estimators=80, random_state=17).fit(X, y)
r = permutation_importance(rf, X, y, n_repeats=5, random_state=17, n_jobs=1)
print("accuracy", round(rf.score(X, y), 3))
print("permutation_importance_mean", np.round(r.importances_mean, 3))

Observed output:

accuracy 1.0
permutation_importance_mean [0.016 0.011 0.163 0.52 ]

Permuting the fourth Iris feature hurts accuracy most, so this fitted forest relies heavily on it. Because the score is computed on training data here, treat the result as a mechanism demonstration, not a deployment audit.

Caveats

Feature importance is not causality. Correlated features can hide each other’s importance. Explanations should be computed on validation or production-like data, and they should be checked against data leakage because the most “important” feature may be an impossible shortcut.

References

• scikit-learn User Guide: Permutation feature importance
• scikit-learn User Guide: Partial dependence

Nav

Section — Classical Machine Learning

← Gradient Boosting Unsupervised Learning →