Class Imbalance

Class imbalance means the class prior $P(Y=k)$ is highly uneven. The problem is not rarity by itself; it is that default training objectives, thresholds, and evaluation metrics may optimize the majority class while missing the decision that matters.

Why accuracy misleads

Imbalance makes accuracy a trap. Suppose $1000$ transactions contain only $10$ fraud cases (the positive class) and $990$ legitimate ones. A model that always predicts “not fraud” produces this confusion matrix:

	actual fraud (10)	actual legit (990)
predicted fraud	$TP=0$	$FP=0$
predicted legit	$FN=10$	$TN=990$

Its accuracy is $\frac{TP+TN}{1000}=\frac{0+990}{1000}=0.99$, yet its recall on the class that matters is $\frac{TP}{TP+FN}=\frac{0}{10}=0$ — it catches no fraud at all. Under imbalance, classification is really a ranking and decision-cost problem: how many cases can be reviewed, and what does missing a positive cost?

Metrics under imbalance

Let $\pi =P(Y=1)$ be the prevalence of the positive class. A binary rule thresholds a model score $s(x)$ at a cutoff $t$, predicting positive when $\hat{y}=1\{s(x)\ge t\}=1$, where the indicator $1\{\cdot \}$ is $1$ when its condition holds. Writing $TP$, $FP$, and $FN$ for the true-positive, false-positive, and false-negative counts, minority-class precision and recall are

$$precision=\frac{TP}{TP+FP},recall=\frac{TP}{TP+FN}.$$

Balanced accuracy averages recall across classes; in binary classification it is $(TPR+TNR)/2$. Here sensitivity, or true positive rate, is $TP/(TP+FN)$; specificity, or true negative rate, is $TN/(TN+FP)$. Sensitivity asks how many actual positives were caught. Specificity asks how many actual negatives were correctly rejected.

PR-AUC summarizes the precision-recall curve for the rare positive class. A common discrete summary is average precision (AP):

$$\mathrm{AP}=\sum _j(R_j-R_{j-1})P_j,$$

where $P_j$ and $R_j$ are precision and recall after moving the threshold through the ranked predictions. AP is a weighted average of precision values: a precision value gets weight only when recall increases. That happens when the next item admitted by the moving threshold is an actual positive.

The ranked-list view is the easiest way to read AP:

1. Sort examples by predicted positive-class score from highest to lowest.
2. Start with a threshold above the highest score, so no example is predicted positive.
3. Move down the ranking one example at a time.
4. After each newly admitted example, recompute precision and recall.
5. Add area only when recall increases; false positives lower later precision but do not directly increase recall.

Suppose the positive class is rare fraud and the model ranks six transactions like this:

Rank	Label	Precision after this rank	Recall after this rank	AP contribution
1	1	$1/1$	$1/3$	$1/3\cdot 1.00$
2	0	$1/2$	$1/3$	$0$
3	1	$2/3$	$2/3$	$1/3\cdot 0.67$
4	0	$2/4$	$2/3$	$0$
5	0	$2/5$	$2/3$	$0$
6	1	$3/6$	$3/3$	$1/3\cdot 0.50$

There are three positives, so recall increases by $1/3$ each time a positive appears. The average precision is:

$$\mathrm{AP}=\frac{1}{3}(1.00+0.67+0.50)\approx \mathrm{0.72.}$$

This score rewards putting positives early. The false positives at ranks 2, 4, and 5 do not add AP area directly, but they reduce precision at the later positive ranks. That is why AP is often more diagnostic than ROC-AUC when the negative class dominates: it focuses on how clean the high-score review queue is for the rare class.

Class weighting changes empirical risk to ${\min }_f{\sum }_i{w}_{y_i}L(y_i,f(x_i))$, where $w_k$ is larger for rare or costly classes.

Comparing plain and weighted models

The snippet compares the same imbalanced dataset with and without class weighting. The point is not that weighting is always better; it is that it changes the precision-recall tradeoff.

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from sklearn.model_selection import train_test_split
 
X, y = make_classification(n_samples=500, n_features=6, n_informative=3,
                           weights=[0.95, 0.05], random_state=11)
Xtr, Xte, ytr, yte = train_test_split(X, y, stratify=y, random_state=11)
for name, est in [("plain", LogisticRegression(max_iter=1000, random_state=11)),
                  ("balanced", LogisticRegression(max_iter=1000, class_weight="balanced", random_state=11))]:
    est.fit(Xtr, ytr)
    pred = est.predict(Xte)
    p, r, _, _ = precision_recall_fscore_support(yte, pred, zero_division=0)
    print(name, "accuracy", round(accuracy_score(yte, pred), 3),
          "minority_precision", round(p[1], 3), "minority_recall", round(r[1], 3))

Observed output:

plain accuracy 0.952 minority_precision 1.0 minority_recall 0.143
balanced accuracy 0.776 minority_precision 0.08 minority_recall 0.286

The balanced model catches more minority examples but creates many more false positives. Whether that is better depends on intervention cost.

Caveats

Resampling before splitting leaks duplicates or synthetic information across folds. PR-AUC is usually more informative than ROC-AUC under extreme rarity, but it still does not choose an operating threshold. Reweighting can hurt calibration, so check probability reliability separately.

References

• scikit-learn User Guide: classification metrics
• scikit-learn User Guide: unbalanced problems

Nav

Section — Classical Machine Learning

← Calibration