Drift¶

Drift answers a different question from a custom metric. Not "what is this number?" but "has the shape of this data changed?"

You do not write a formula for drift. You pick a column and an algorithm, and MPM charts a drift score per time bucket.

What is being compared¶

Every drift score compares a current distribution against a reference one. There are two ways to choose the reference, and they answer different questions.

   Against the previous period                Against a training dataset

     Mon → Tue → Wed → Thu                        ┌──────────┐
      └────┘ └────┘ └────┘                        │ training │
     each compared with the one before            └────┬─────┘
                                                       │  every period compared
     "did something change recently?"                  ▼  against the same baseline
                                                  Mon  Tue  Wed  Thu

                                                  "how far from launch conditions?"

If your model has a training dataset registered, that becomes the reference and every period is compared against it. Otherwise each period is compared with the one before.

How a distribution becomes a number¶

Both distributions are first reduced to bins — buckets with a proportion of the data in each. For a categorical column the bins are the categories; for a numerical column they are value ranges.

       bin:      b1     b2     b3     b4     b5
   reference:   0.10   0.25   0.30   0.25   0.10      each row sums to 1
     current:   0.05   0.15   0.25   0.35   0.20

A drift algorithm turns those two rows into one number. MPM offers four.

The four algorithms¶

Below, r[i] is the reference proportion in bin i, and c[i] the current one.

PSI — Population Stability Index¶

   PSI = sum over bins of  ( r[i] - c[i] ) * ln( r[i] / c[i] )

Every bin contributes, and contributions grow sharply when a bin changes by a large factor. The conventional reading in credit risk is: below 0.1 stable, 0.1 to 0.2 worth investigating, above 0.2 act. PSI is unbounded — there is no maximum score.

PSI is the metric most often named by name in model governance policies.

KL — Kullback-Leibler divergence¶

   KL = sum over bins of  r[i] * ln( r[i] / c[i] )

From information theory: how surprised you would be by the current data if you were expecting the reference. Asymmetric — swapping reference and current gives a different answer. Also unbounded.

EMD — Earth Mover's Distance¶

Imagine each distribution as piles of earth. EMD is how much work it takes to shovel one into the shape of the other.

Alone among the four, EMD accounts for how far the data moved, not just that it moved. Shifting values into a neighbouring bin costs little; shifting them across the whole range costs a lot. That makes it a natural fit for numerical columns, where bins have a real order.

KS — Kolmogorov-Smirnov¶

Compare the two cumulative distributions and take the single largest gap:

   KS = max over bins of  | running total of r up to i  -  running total of c up to i |

   cumulative
   proportion
      1.0 ┤                            ╭──────────  reference
          │                      ╭─────╯
          │               ╭──────╯
          │        ╭──────╯   ⇕ largest gap = the KS score
          │  ╭─────╯   ╭─────╯
          │ ╭╯   ╭─────╯                            current
      0.0 ┼─┴────╯────────────────────────────────
            b1    b2    b3    b4    b5    b6

Two properties follow:

• Bounded between 0 and 1. 0 means identical, 1 means no overlap at all. A bounded score is easier to reason about and to explain.
• Robust. It reports the single worst disagreement, so it cannot be inflated by many tiny differences, or dominated by one nearly-empty bin.

Choosing between them¶

A reasonable default: PSI when governance asks for PSI, KS when you want a score that behaves predictably, EMD for numerical columns where distance is meaningful.

Alerting on drift¶

You can create an alert on a drift chart the same way as on any other metric: choose the column and algorithm, then set the threshold and the comparison yourself.

   Alert when   KS on "label_response"   is greater than   0.3
                └─ metric ────────────┘  └─ operator ───┘  └ threshold ┘

The threshold is yours to choose, per alert, so different models can use different sensitivities. Nothing is fixed to a built-in value.

Because the four algorithms are on different scales, a threshold cannot be carried from one to another. PSI is unbounded and typically read against 0.1 and 0.2; KS cannot exceed 1 and its useful range sits well below that. Changing the algorithm on an existing alert means revisiting its threshold.

Once set, the threshold is drawn on the chart as a dashed line with the alerting side shaded, so you can see how close the metric runs to it. See Monitors on charts.

Which columns can drift¶

Any column, whatever its kind:

The third is easy to overlook and often the most informative — see Ground truth.

A note on precision¶

Numerical distributions are summarised approximately rather than by retaining every value, which is what makes drift affordable over billions of predictions. Two consequences are worth knowing:

• Two samples drawn from an identical distribution will not score exactly zero. There is a small amount of noise, so a stable model reads as slightly above zero rather than at it — worth knowing when choosing an alert threshold.
• Distributions with no overlap at all score close to, but not exactly, the theoretical maximum.

Neither affects how you read a chart. Both matter if you are comparing scores against thresholds computed elsewhere.