Skip to content

Send data to MPM

Start logging model predictions, as well as new models, simply by sending Comet your model's input features and predictions.

There are two ways to send events to MPM:

MPM relies on JSON events to track the performance of your models, to enable Comet to deliver a monitoring solution that is agnostic of any model or serving framework. Comet also integrates with a wide number of frameworks.

MPM events

To track model performance, MPM needs to have access to a model's input features and output features. This data can be sent to MPM in JSON format using MPM events or using our Python SDK.

Each log message can contain the following attributes:

NameTypeDescriptionRequiredExample
workspaceNamestringComet workspace in which to log the modelobject-detection
modelNamestringName of modelDemo model
modelVersionstringVersion of the model"1.0.0"
predictionIdstringUsed to identify a single prediction1
timestampintTimestamp in milliseconds1.62E+12
featuresobjectInput features to the modelSee example below.
predictionobjectPrediction made by the model, can include two fields: value and probabilitySee example below.

Here is an example payload:

{
    "workspaceName": "...",
    "modelName": "Credit Scoring",
    "modelVersion": "1.0.0",
    "timestamp": 1615922560000,
    "predictionId": "000001",
    "features": {
        "feature_1":0.34,
        "feature_2": "dog",
    },
    "prediction": {
        "value": "true",
        "probability": 0.86
    }
}

Note

If you want full lineage between Experiment Management and MPM, it is important to ensure that the model name and model version used in the MPM events payload matches the one in the model registry.

If, in contrast, you simply want to use MPM, then you can just send the events and Comet will create the model for you.

Configure how events are delivered to MPM

You can configure how the log messages are delivered to Comet's MPM: either through the MPM REST API or MPM SDK.

With the REST API

The endpoint for the REST API is deployment-specific and follows the following format:

https://www.comet.ml/mpm/events

with the API key based in the header.

To send a message, make a POST request:

curl -d '{<payload>}'\
     -H "Content-Type: application/json"\
     -H "Authorization: <api_key>"\
     -X POST https://www.comet.ml/mpm/events

With <payload> following the log format described above.

To test, you can use:

workspaceName=<workspace_name>
api_key=<api_key>

current_timestamp=$(date +%s000)
payload='{"features": {"categorical_feature_0": "value_1", "numerical_feature_0": 0.5841119597210334}, "modelName": "test-model", "modelVersion": "1.0.0", "prediction": {}, "predictionId": "e347539b-a1df-432e-aa4a-3fe93805d3be", "timestamp": '$current_timestamp', "workspaceName": "'$workspaceName'"}'

curl -s -d "$payload" \
    -H "Content-Type: application/json"\
    -H "Authorization: $api_key"\
    -X POST https://www.comet.ml/mpm/events

With the SDK

To use the SDK, make sure you have installed it first.

Install the SDK

pip install comet-mpm

See the MPM SDK.

Use MPM SDK in your Inference Server code

Here are two examples, using popular frameworks.

Configure with Flask

from uuid import uuid4

from comet_mpm import CometMPM
from flask import Flask, jsonify, request

app = Flask(__name__)
MPM = CometMPM()

...

@app.post("/prediction")
def prediction():
    prediction_id = str(uuid4())

    input_features = get_input_features(request)
    prediction, probability = get_prediction()

    MPM.log_event(
        prediction_id=prediction_id,
        input_features=input_features,
        output_value=prediction,
        output_probability=probability,
    )

    return jsonify({"prediction": prediction, "probability": probability})

Configure with FastAPI/asyncio

from uuid import uuid4

from comet_mpm import CometMPM
from fastapi import FastAPI, Request

app = FastAPI()
MPM = CometMPM(asyncio=True)

...

@app.on_event("startup")
async def startup_event():
    MPM.connect()


@app.post("/prediction")
async def prediction(request: Request):
    prediction_id = str(uuid4())

    input_features = get_input_features(request)
    prediction, probability = get_prediction()

    await MPM.log_event(
        prediction_id=prediction_id,
        input_features=input_features,
        output_value=prediction,
        output_probability=probability,
    )

    return {"prediction": prediction, "probability": probability}


@app.on_event("shutdown")
async def shutdown_mpm():
    await MPM.join()

Learn more

Jul. 11, 2022