When it comes to monitoring models in production, it is important to be able to track metrics with different levels of granularity. Often models will drift for only a subset of predictions rather that for all predictions. If the only metrics that are tracked are at a model level, then small drifts in subsets of data can be missed in natural variance of metrics.
MPM allows you to define Segments so that you can track and compare drift or feature distributions between subsets of data.
In the Settings tab for each model you will find a section called
MPM Segments. In this section you will be able to creat new Segments as well as view the Segments that have been created.
By clicking on the
Create Segment button, you will be able to define a new Segment based on any of the categorical input or output features of the model.
When creating a Segment, you will need to complete four fields: 1. Segment name: Name of the Segment 2. Feature breakdown: The categorical feature used to define the subsets of predictions, for example
country 3. Segment values: The unique values taken by the feature chosen previously, these are going to the segments for which to view the metrics by 4. Description: Optional description field
Segments are only computed for predictions with a timestamp that is greater than the creation time of the Segment.
Viewing metrics by Segment¶
In both the
Model Performance page and the
Features page you will be able to choose whether you want to see metrics across all predictions or for the Segments that you have defined.
To choose the Segments to display, you can click on the
Compare Segments dropdown. This will open up a dropdown in which you can select the subsets of data to display.