How MPM works¶

This page explains what MPM stores, how it turns that into charts, and the concepts the rest of the MPM guides assume. It does not require any knowledge of SQL or of the underlying database.

What MPM is for¶

A model in production makes predictions, and two things can go wrong without announcing themselves:

1. The world changes. The applicants arriving today do not look like the ones the model was trained on. The model still returns confident answers; they are just increasingly wrong.
2. The model gets worse at its job. It was 92% accurate at launch and it is 71% accurate now, and nobody notices until someone runs a report.

MPM notices both. It stores every prediction your model makes, and lets you chart it, alert on it, and investigate it over time.

What MPM stores¶

Each time your model makes a prediction, your application sends MPM an event with three kinds of content:

The third is special: it usually arrives later. You learn whether a transaction was really fraudulent days after predicting it, so labels are attached to an existing prediction afterwards, matched by prediction ID. See Ground truth.

Each model's events are stored in their own table, with a prefix on each column showing what it is:

   __time     timestamp   feature_amount  feature_country  prediction_score  label_response
   ─────────  ──────────  ──────────────  ───────────────  ────────────────  ──────────────
   08-01 09h  175…000     240.50          UK               0.87              1
   08-01 10h  175…600      18.00          IE               0.02              0
   08-01 11h  175…200     990.00          UK               0.94              (not yet)

Those prefixes are how MPM knows what each column is, and which charts it can appear in:

You use the full column name, prefix included, whenever you refer to a column — for example in a custom metric formula.

Two kinds of question¶

MPM answers two quite different questions, through different machinery.

"What is this number, over time?"¶

You write a formula and MPM charts it per hour or per day:

SELECT ACCURACY("label_response", "prediction_class") FROM MODEL

This is a custom metric, written in a small SQL-like language. MODEL is a keyword standing in for your model's table — you never name the table. See Custom metrics.

"Has the shape of this data changed?"¶

Drift asks something subtler: not "what is the average?" but "is this column's whole distribution different from what it used to be?"

Picture last month's transaction amounts and this month's as two histograms. Drift is a single number summarising how different those two pictures are:

   Reference (training, or the previous period)     Current period

        ▁▃▅█▅▃▁                                             ▁▃▅█▅▃▁
   ──────────────────────                          ──────────────────────
        ^ centred low                                       ^ shifted right

                        drift score: 0.34  →  "quite different"

You do not write a formula for drift. You choose a column and an algorithm. See Drift.

For a model that outputs a probability rather than a class, how well that probability ranks the outcome is a separate question — see Classification performance.

How a custom metric becomes a chart¶

   you type a formula
           │
           ▼
   the editor checks it as you type        ← errors are underlined immediately
           │
           ▼
   saved  (the formula text is stored, not a query)
           │
           ▼
   translated into a database query        ← time buckets, filters and names added
           │
           ▼
   the database runs it
           │
           ▼
   results become one line per metric per filter
           │
           ▼
   chart

Two steps are worth understanding, because they explain most of MPM's behaviour.

Everything is grouped into time buckets. A custom metric is always "this value, per hour" or "per day". You never write the grouping; MPM adds it. This is why a metric formula has to reduce to a single number — that number is what gets plotted for each bucket.

Filters become extra columns, not extra queries. The filters you set on a dashboard (its predicates) are applied to each metric separately, so one metric with three filters produces three lines on the chart from a single query.

Metrics that look backwards¶

Some questions cannot be answered one bucket at a time:

• "How many predictions in total so far this month?" — a running total.
• "What is the 7-day rolling average volume?" — a sliding window.

Both need values from other buckets. MPM supports these through window functions:

SELECT SUM("feature_amount") OVER (ORDER BY timeBucket) FROM MODEL

   buckets:    Aug 1   Aug 2   Aug 3   Aug 4   Aug 5
   daily:        10      20      30      40      50
   running:      10      30      60     100     150     ← the running total

Because a running total has to be built from the per-bucket values, only measures whose parts add up to the whole can carry a window — sums, counts, minimums and maximums. A running total of per-bucket averages is not an average of anything, so averages and rates are not accepted. See Window functions.

Alerting¶

Anything you can chart, you can monitor. A monitor watches a metric and fires when it crosses a threshold you set, using a comparison you choose — greater than, less than, or either of those with equality. Drift is monitored the same way as any other metric, with the threshold set per alert.

Putting it together¶

   your application
        │  prediction made                    ground truth arrives later
        │                                              │
        ▼                                              ▼
   ┌──────────────────────────────────────────────────────────┐
   │  event storage:  feature_*   prediction_*   label_*      │
   └──────────────────────────────────────────────────────────┘
        │                                              │
        │ custom metrics                               │ drift
        │ (a formula you wrote)                        │ (a column + algorithm you picked)
        ▼                                              ▼
   ┌──────────────────────────────────────────────────────────┐
   │                    charts and alerts                     │
   └──────────────────────────────────────────────────────────┘

Glossary¶

Where to next¶