Skip to content

Send data to MPM

Start logging model predictions, as well as new models, simply by sending Comet your model's input features and predictions. There is no additional configuration required.

There are two main integration paths when it comes to MPM:

  • Sending data from the application making the predictions
  • Using log forwarding

If the inference service is written in Python and predictions are not tracked already, we recommend sending events straight from the application using our MPM Python SDK.

If you are using a managed inference service like Sagemaker or Seldon, you can simply enable Data Capture or Request / Response logging and forward these events to MPM using our Rest API.

Integration with Experiment Management

Using MPM in conjunction with Experiment Management and Comet's Model Registry allows you to track models from development all the way to production ! In order to do this, you simply need to ensure that the model name used when sending MPM events matches up with the model name in the Model Registry.

MPM can also be used standalone, in which case models are automatically created in the Comet Model Registry whenever you send the first event for a new model.

MPM Python SDK

The MPM Python SDK has been developed with production inference services in mind and includes a number of optimizations to make sure the logging overhead is kept at a minimum.

The MPM SDK can be installed using:

pip install comet-mpm

Once installed, logging events to MPM takes just three lines of code:

from comet_mpm import CometMPM

MPM = CometMPM(model_name='credit-scoring', model_version='1.2.0')

MPM.log_event(
    prediction_id="1",
    input_features=[{'feature_1': 10}],
    output_value=True,
    output_probability=0.5
)

Note

If you are using FastAPI, you will also need to implement a shutdown event as detailled here

MPM Rest API

To track model performance, MPM needs to have access to a model's input features, output features and labels. In addition to the Python API, this data can be sent via a Rest API.

The MPM Rest API includes three methods that can be used to upload MPM events:

  • https://www.comet.ml/mpm/events: Used to upload single events that contain input and output features
  • https://www.comet.ml/mpm/events/batch: Used to upload batches of events that contain input and output features
  • https://www.comet.ml/mpm/labels: Used to upload labels

Sending input and output features to MPM

The JSON payload for the https://www.comet.ml/mpm/events and https://www.comet.ml/mpm/events/batch POST endpoints should contain the following attributes:

NameTypeDescriptionRequiredExample
workspaceNamestringComet workspace in which to log the modelobject-detection
modelNamestringName of modelDemo model
modelVersionstringVersion of the model"1.0.0"
predictionIdstringUsed to identify a single prediction1
timestampintTimestamp in milliseconds1.62E+12
featuresobjectInput features to the modelSee example below.
predictionobjectPrediction made by the model, can include two fields: value and probabilitySee example below.

Here is an example payload:

{
    "workspaceName": "...",
    "modelName": "Credit Scoring",
    "modelVersion": "1.0.0",
    "timestamp": 1615922560000,
    "predictionId": "000001",
    "features": {
        "feature_1":0.34,
        "feature_2": "dog",
    },
    "prediction": {
        "value": "true",
        "probability": 0.86
    }
}

To test, you can use:

workspaceName=<workspace_name>
api_key=<api_key>

current_timestamp=$(date -v-1H +%s000)
payload='{"features": {"categorical_feature_0": "value_1", "numerical_feature_0": 0.5841119597210334}, "modelName": "test-model", "modelVersion": "1.0.0", "prediction": {}, "predictionId": "e347539b-a1df-432e-aa4a-3fe93805d3be", "timestamp": '$current_timestamp', "workspaceName": "'$workspaceName'"}'

curl -s -d "$payload" \
    -H "Content-Type: application/json"\
    -H "Authorization: $api_key"\
    -X POST https://www.comet.ml/mpm/events

Note

In the example above we have offset the timestamp by an hour as data about the current hour is not displayed in the MPM dashboard.

If you use the current timestamp, you will need to wait an hour for the data to appear in the Comet dashboard or use one of our Rest API methods to check the event has been correctly ingested.

Sending labels to MPM

Ground truth labels can be sent to Comet and are used to compute accuracy related metrics (Accuracy, Precision, Recall, F1-score, etc). You can send the labels at any time after a prediction has been logged to MPM, we will take care of updating the relevant metrics.

Labels can be sent to Comet using the https://www.comet.ml/mpm/labels POST endpoint with a JSON payload containing the following attributes:

NameTypeDescriptionRequiredExample
workspaceNamestringComet workspace in which to log the modelobject-detection
modelNamestringName of modelDemo model
modelVersionstringVersion of the model"1.0.0"
predictionIdstringUsed to identify a single prediction1
valueobjectValue of the label"true"

Note

Only labels that match up to a prediction stored MPM will be ingested, if the label cannot be matched up to a prediction on ingestion it will not be processed.

Learn more

Nov. 27, 2022