MPM segments
When it comes to monitoring models in production, it is important to be able to track metrics with different levels of granularity. Often models will drift for only a subset of predictions rather that for all predictions. If the only metrics that are tracked are at a model level, then small drifts in subsets of data can be missed in natural variance of metrics.
MPM allows you to define Segments so that you can track and compare drift or feature distributions between subsets of data.
Defining Segments¶
In the Settings tab for each model you will find a section called MPM Segments. In this section you will be able to creat new Segments as well as view the Segments that have been created.
By clicking on the Create Segment button, you will be able to define a new Segment based on any of the categorical input or output features of the model.
When creating a Segment, you will need to complete four fields: 1. Segment name: Name of the Segment 2. Feature breakdown: The categorical feature used to define the subsets of predictions, for example country 3. Segment values: The unique values taken by the feature chosen previously, these are going to the segments for which to view the metrics by 4. Description: Optional description field
Note
Segments are only computed for predictions with a timestamp that is greater than the creation time of the Segment.
Viewing metrics by Segment¶
In both the Model Performance page and the Features page you will be able to choose whether you want to see metrics across all predictions or for the Segments that you have defined.
To choose the Segments to display, you can click on the Compare Segments dropdown. This will open up a dropdown in which you can select the subsets of data to display.
