Classification performance¶
Accuracy tells you whether a model got the answer right. For a model that outputs a probability — the chance an application defaults, the chance a customer responds — that is rarely the question. Such a model is used by ranking: score everyone, act on the top slice. What matters is whether the ranking is any good.
That is what these four report. They read a predicted probability against the ground truth, and they need three things a predicted class alone cannot give: which column holds the score, which holds the outcome, and which outcome the score is the probability of.
| Answers | Range | |
|---|---|---|
| ROC AUC | How often does a positive outrank a negative? | 0.5 coin flip → 1.0 perfect |
| KS | How far apart are the two populations at their furthest? | 0 → 1 |
| Gini | The same as AUC, rescaled | 0 coin flip → 1 perfect |
| Gains table | If I act on the top 10%, what do I get? | a table |
The curve they all come from¶
Sort every prediction by score, highest first — the order the model would actually be used in — and walk down it. At each point, record two things: the share of all positives you have collected so far, and the share of all negatives.
TPR 1.0 ┤ ╭─────────────
share │ ╭─────────╯
of │ ╭─────╯ the area under this curve is AUC
positives │ ╭───╯
│ ╭─╯ ↕ the widest vertical gap is the KS statistic
0.0 ┼─╯───────────────────────────────────
0.0 share of negatives 1.0 FPR
A model that ranks well collects positives fast and negatives slowly, so the curve bows toward the top left. A model that knows nothing collects both at the same rate and the curve is the diagonal.
- ROC AUC is the area under it. The diagonal has area 0.5, which is why 0.5 means "no better than guessing".
- KS is the widest vertical gap between the two cumulative shares. Credit risk reads it directly: how well does this scorecard separate the two populations?
- Gini is
2 × AUC − 1. Same information, rescaled so 0 is a coin flip.
KS is unsigned
A model that ranks positives below negatives separates the two populations perfectly — it is perfectly informative once you invert it. KS reports the separation it has rather than zero. Read it next to AUC, which is signed and tells you which way round the model runs.
The gains table¶
The three numbers above score the whole ranking. The gains table says what the ranking is worth at each cutoff, which is how a cutoff actually gets chosen.
The scored population is sorted highest first and cut into ten equal bands — deciles. Each row says what you get if you act on everything down to that band.
band score range count defaults rate cum%pop gain lift cumLift KS
1 0.58-1.00 5,000 668 13.37% 10.0% 28.2% 2.82 2.82 0.191
2 0.50-0.58 5,000 467 9.32% 20.0% 47.9% 1.97 2.39 0.293
3 0.44-0.50 5,000 364 7.29% 30.0% 63.2% 1.54 2.11 0.349
4 0.40-0.44 5,000 263 5.26% 40.0% 74.3% 1.11 1.86 0.360 ← KS peak
5 0.35-0.40 5,000 213 4.27% 50.0% 83.3% 0.90 1.67 0.350
…
10 0.00-0.12 5,000 3 0.06% 100.0% 100.0% 0.01 1.00 0.000
Reading across row 3: the top 30% of applications contain 63.2% of all defaults, and defaults are 2.11 times as common there as in the book overall.
| Column | Means |
|---|---|
count, defaults | Records in this band, and how many were the outcome you named |
rate | defaults / count — how often the outcome occurred in this band |
cum%pop | The "top N%" this row describes |
gain | Share of all positives captured down to here. This is the gains curve |
lift | rate ÷ overall rate — how much better this band is than picking at random |
cumLift | The same for everything down to here; the number a campaign is sized on |
KS | Separation so far. Its peak is the cutoff that splits the two populations best |
The last row is always the whole population, so gain is 100% and cumLift is exactly 1 — you have selected everyone, so you have no lift over selecting everyone.
Where the cutoff goes
The band where the KS column peaks is the score that separates the two populations best. In the table above that is band 4, so a cutoff around 0.40 maximises separation. Whether it maximises profit is a different question, and that is what cumLift is for.
Choosing the class¶
Every one of these is a report about a class: gains of what, precision for whom. You name the outcome the score is the probability of, and everything else counts as a negative.
If your ground truth is logged as 0/1 — the usual case for a binary outcome — you can leave it unset and it compares against 1. Otherwise give the value as it appears in your data, for example defaulted or responded.
Records whose ground truth has not arrived yet are excluded rather than counted as negatives. See Ground truth for how labels arrive after predictions.
What these need from your data¶
- A numerical score column. A predicted class cannot produce a curve; there is nothing to rank by. Naming a categorical column is refused rather than charted as a flat line.
- Both classes present. A ROC curve needs positives to compute a true positive rate and negatives for a false positive rate. A period with only one — a quiet hour, or ground truth that has not landed — reports no value and appears as a gap in the chart rather than as a zero.
Monitoring them¶
ROC AUC, KS and Gini can be watched by a monitor like any other metric, which is the usual way to find out that a model has degraded: alert when AUC falls below a threshold you set.
A monitor on these needs both columns and the class, because that is what the metric reads. A monitor that does not name them is refused when you save it rather than failing silently every time it runs.
The gains table is not a monitor — it is a table read at a point in time, not a number that crosses a threshold. Monitor the KS or the AUC, and read the table when one of them moves.
A monitor's threshold appears on the curve chart as a dashed line with the alerting side shaded. See Monitors on charts for how to read it, and for why a monitor can sometimes be named on a chart rather than drawn on it.
Accuracy of the numbers¶
The scores are not ranked one by one. Ranking every prediction is not something the query engine can do at this scale, so the predictions are counted into fine score bins and everything is computed from those counts. Two consequences worth knowing:
The curve is drawn with straight segments between bin edges. Within a bin, positives and negatives are indistinguishable. The area is computed with the trapezoid rule, which is exact for a straight segment, and the error falls as bins are added. Measured against exact rank-based AUC on a 200,000-prediction synthetic distribution, the default binning lands within 0.001. That is one distribution rather than a universal guarantee — a score squeezed into a narrow range bins more coarsely — but it is the shape the default was chosen for.
The gains bands are cut by population, not by score. A gains table only means anything if every band holds the same number of records; otherwise "the top 10%" means something different on every row. The bins are equal slices of the score scale, and a scorecard concentrates its population into a few of them, so the bands are cut wherever the running count reaches the next tenth — which usually falls inside a bin. That bin is split in proportion.
Because the bands sample the same curve at ten points where the KS statistic samples it at a hundred, the table's largest KS value can equal the charted KS but never exceeds it. In the table above the peak reads 0.360 against a charted KS of 0.364; the true peak sits between two decile cuts.