skip to Main Content

Where AI Developers Build

Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring.

Trusted by the most innovative ML teams

Where AI Developers Build

Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring.

Trusted by the most innovative ML teams

Manage Any ML or LLM Lifecycle, From Training Through Production

Manage Any ML or LLM Lifecycle, From Training Through Production

Automatically track all your prompt engineering work. Run automated evaluations on your LLM responses to optimize your applications before and after they hit production.

Opik – Open Source LLM Evaluation

Log all your machine learning iteration to a single system of record. Make it easy to reproduce a previous experiment and compare the performances of training runs.

Comet Experiment Management

Track data drift on your input and output features after your model is deployed to production. Set customized alerts to capture model performance degradation in real time.

Model Production Monitoring

Create a centralized repository of all your model versions with immediate access to how they were trained. Promote models to downstream production systems with webhooks

Comet Model Registry

Know which exact dataset version a model was trained on for auditing and governance purposes. Leverage remote pointers to reference data already stored in the cloud. 

Comet Artifacts
Panels in Comet
Model Production Monitoring | Comet ML
model registry screen in Comet
Artifacts screen in Comet

Easy Integration

Add just a few lines of code to your notebook or script and automatically start tracking LLM traces, code, hyperparameters, metrics, model predictions, and more.

from comet_ml import Experiment
from transformers import Trainer

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Train your model 
trainer = Trainer(
    model = model,
    # ...configs
)

trainer.train()

# 3. View real-time metrics in Comet

ML Experiment Management

PyTorch logo Pytorch

from comet_ml import Experiment
import torch.nn as nn

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Create your model class 
class RNN(nn.Module):
    #... Define your Class 

# 3. Train and test your model while logging everything to Comet
with experiment.train():
    # ...Train your model and log metrics 
    experiment.log_metric("accuracy", correct / total, step = step)

# 4. View real-time metrics in Comet

pytorch lightning logo Pytorch Lightning

from pytorch_lightning.loggers import CometLogger

# 1. Create your Model

# 2. Initialize CometLogger
comet_logger = CometLogger()

# 3. Train your model 
trainer = pl.Trainer(
    logger=[comet_logger],
    # ...configs
)

trainer.fit(model)

# 4. View real-time metrics in Comet

Hugging Face logo Hugging Face

from comet_ml import Experiment
from transformers import Trainer

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Train your model 
trainer = Trainer(
    model = model,
    # ...configs
)

trainer.train()

# 3. View real-time metrics in Comet

Keras logo Keras

from comet_ml import Experiment
from tensorflow import keras

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define your model
model = tf.keras.Model(
    # ...configs
)

# 3. Train your model
model.fit(
    x_train, y_train,
    validation_data=(x_test, y_test),
)

# 4. Track real-time metrics in Comet

Tensorflow logo Tensorflow

from comet_ml import Experiment
import tensorflow as tf

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define and train your model
model.fit(...)

# 3. Log additional model metrics and params
experiment.log_parameters({'custom_params': True})
experiment.log_metric('custom_metric', 0.95)

# 4. Track real-time metrics in Comet

SciKit Learn logo Scikit-learn

from comet_ml import Experiment
import tree from sklearn

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Build your model and fit
clf = tree.DecisionTreeClassifier(
    # ...configs
)

clf.fit(X_train_scaled, y_train)
params = {...}
metrics = {...}

# 3. Log additional metrics and params
experiment.log_parameters(params)
experiment.log_metrics(metrics)

# 4. Track model performance in Comet

XGBoost logo XGBoost

from comet_ml import Experiment
import xgboost as xgb

# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define your model and fit
xg_reg = xgb.XGBRegressor(
    # ...configs
)
xg_reg.fit(
    X_train,
    y_train,
    eval_set=[(X_train, y_train), (X_test, y_test)],
    eval_metric="rmse",
)

# 3. Track model performance in Comet

icon with 2 arrows pointing back and forth Any framework

# Utilize Comet in any environment
from comet_ml import Experiment

# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Model training here

# 3. Log metrics or params over time
experiment.log_metrics(metrics)

#4. Track real-time metrics in Comet

OPIK LLM Evaluation

OpenAI logo OpenAI

import comet_llm

from openai import OpenAI

#1. Initialize Comet 
comet_llm.init(
    api_key="YOUR_COMET_API_KEY",
    project="openai-example",
)

#2. Send Prompts to OpenAI 
client = OpenAI()
response = client.chat.completions.create(
  model="gpt-3.5-turbo",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)

#3. View Prompt Responses in Comet

Langchain logo LangChain

import comet_llm

from langchain.callbacks.tracers.comet import CometTracer
from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage

#1. Initialize Comet 
comet_llm.init(
    api_key="YOUR_COMET_API_KEY",
    project="langchain_example",
)

#2. Send Prompts to LLM
callbacks = [CometTracer()]
chat_model = ChatOpenAI(callbacks=callbacks)

text = "What would be a good company name for a company that makes colorful socks?"
messages = [HumanMessage(content=text)]

chat_model.invoke(messages)

#3. View Prompt Responses and Chains in Comet

icon with 2 arrows pointing back and forth Any Framework

#Utilize Comet in any environment 
import comet_llm

#1. Initialize Comet 
comet_llm.init(
    api_key="YOUR_API_KEY",
    project="YOUR_LLM_PROJECT",
)

#2. Log prompt to Comet
comet_llm.log_prompt(
    prompt = "",
    output = "",
)

#3. View Prompt History in Comet 

Model Monitoring

icon with 2 arrows pointing back and forth Any Framework

# Utilize Comet in any environment
from comet_mpm import CometMPM

# 1. Create the MPM logger
MPM = CometMPM()

# 2. Add your inference logic here

# 3. Log metrics or params over time
MPM.log_event(
        prediction_id="...",
        input_features=input_features,
        output_value=prediction,
        output_probability=probability,
    )

An End-to-End Model Evaluation Platform

Comet’s end-to-end model evaluation platform for developers focuses on shipping AI features, including open source LLM tracing, ML unit-testing, evaluations, experiment tracking and production monitoring.

Easy Integration

Add just a few lines of code to your notebook or script and automatically start tracking LLM traces, code, hyperparameters, metrics, model predictions, and more.

from comet_ml import Experiment
from transformers import Trainer

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Train your model 
trainer = Trainer(
    model = model,
    # ...configs
)

trainer.train()

# 3. View real-time metrics in Comet

OPIK LLM Evaluation

OpenAI logo OpenAI



from openai import OpenAI
from opik.integrations.openai import track_openai

openai_client = OpenAI()
openai_client = track_openai(openai_client)

response = openai_client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Hello, world!"}
    ]
)

Langchain logo LangChain


from langchain_openai import ChatOpenAI
from opik.integrations.langchain import OpikTracer

# Initialize the tracer
opik_tracer = OpikTracer()

# Create the LLM Chain using LangChain
llm = ChatOpenAI(temperature=0)

# Configure the Opik integration
llm = llm.with_config({"callbacks": [opik_tracer]})

llm.invoke("Hello, how are you?")

icon with 2 arrows pointing back and forth LlamaIndex


from llama_index.core import VectorStoreIndex, global_handler, set_global_handler
from llama_index.core.schema import TextNode

# Configure the Opik integration
set_global_handler("opik")
opik_callback_handler = global_handler


node1 = TextNode(text="The cat sat on the mat.", id_="1")
node2 = TextNode(text="The dog chased the cat.", id_="2")

index = VectorStoreIndex([node1, node2])

# Create a LlamaIndex query engine
query_engine = index.as_query_engine()

# Query the documents
response = query_engine.query("What did the dog do ?")
print(response)

icon with 2 arrows pointing back and forth Any Framework


from opik import track

@track
def llm_chain(user_question):
    context = get_context(user_question)
    response = call_llm(user_question, context)
    
    return response

@track
def get_context(user_question):
    # Logic that fetches the context, hard coded here
    return ["The dog chased the cat.", "The cat was called Luky."]

@track
def call_llm(user_question, context):
    # LLM call, can be combined with any Opik integration
    return "The dog chased the cat Luky."

response = llm_chain("What did the dog do ?")
print(response)

ML Experiment Management

PyTorch logo Pytorch

from comet_ml import Experiment
import torch.nn as nn

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Create your model class 
class RNN(nn.Module):
    #... Define your Class 

# 3. Train and test your model while logging everything to Comet
with experiment.train():
    # ...Train your model and log metrics 
    experiment.log_metric("accuracy", correct / total, step = step)

# 4. View real-time metrics in Comet

pytorch lightning logo Pytorch Lightning

from pytorch_lightning.loggers import CometLogger

# 1. Create your Model

# 2. Initialize CometLogger
comet_logger = CometLogger()

# 3. Train your model 
trainer = pl.Trainer(
    logger=[comet_logger],
    # ...configs
)

trainer.fit(model)

# 4. View real-time metrics in Comet

Hugging Face logo Hugging Face

from comet_ml import Experiment
from transformers import Trainer

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Train your model 
trainer = Trainer(
    model = model,
    # ...configs
)

trainer.train()

# 3. View real-time metrics in Comet

Keras logo Keras

from comet_ml import Experiment
from tensorflow import keras

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define your model
model = tf.keras.Model(
    # ...configs
)

# 3. Train your model
model.fit(
    x_train, y_train,
    validation_data=(x_test, y_test),
)

# 4. Track real-time metrics in Comet

Tensorflow logo Tensorflow

from comet_ml import Experiment
import tensorflow as tf

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define and train your model
model.fit(...)

# 3. Log additional model metrics and params
experiment.log_parameters({'custom_params': True})
experiment.log_metric('custom_metric', 0.95)

# 4. Track real-time metrics in Comet

SciKit Learn logo Scikit-learn

from comet_ml import Experiment
import tree from sklearn

# 1. Define a new experiment 
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Build your model and fit
clf = tree.DecisionTreeClassifier(
    # ...configs
)

clf.fit(X_train_scaled, y_train)
params = {...}
metrics = {...}

# 3. Log additional metrics and params
experiment.log_parameters(params)
experiment.log_metrics(metrics)

# 4. Track model performance in Comet

XGBoost logo XGBoost

from comet_ml import Experiment
import xgboost as xgb

# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Define your model and fit
xg_reg = xgb.XGBRegressor(
    # ...configs
)
xg_reg.fit(
    X_train,
    y_train,
    eval_set=[(X_train, y_train), (X_test, y_test)],
    eval_metric="rmse",
)

# 3. Track model performance in Comet

icon with 2 arrows pointing back and forth Any framework

# Utilize Comet in any environment
from comet_ml import Experiment

# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")

# 2. Model training here

# 3. Log metrics or params over time
experiment.log_metrics(metrics)

#4. Track real-time metrics in Comet

ML Model Monitoring

icon with 2 arrows pointing back and forth Any Framework

# Utilize Comet in any environment
from comet_mpm import CometMPM

# 1. Create the MPM logger
MPM = CometMPM()

# 2. Add your inference logic here

# 3. Log metrics or params over time
MPM.log_event(
        prediction_id="...",
        input_features=input_features,
        output_value=prediction,
        output_probability=probability,
    )

An End-to-End Model Evaluation Platform

Comet’s end-to-end model evaluation platform for developers focuses on shipping AI features, including open source LLM tracing, ML unit-testing, evaluations, experiment tracking and production monitoring.

Where AI Developers Build

Run Comet’s end-to-end evaluation platform on any infrastructure to see firsthand how Comet’s reshapes your workflow. Bring your existing software and data stack. Use code panels to create visualizations in your preferred user interfaces.

Where AI Developers Build

Run Comet’s end-to-end evaluation platform on any infrastructure to see firsthand how Comet’s reshapes your workflow. Bring your existing software and data stack. Use code panels to create visualizations in your preferred user interfaces.

An AI Platform Built for Enterprise, Driven by Community

Comet’s end-to-end evaluation platform is trusted by innovative data scientists, ML practitioners, and engineers in the most demanding enterprise environments.

An AI Platform Built for Enterprise, Driven by Community

Comet’s end-to-end evaluation platform is trusted by innovative data scientists, ML practitioners, and engineers in the most demanding enterprise environments.

Get started today, free

No credit card required, try Comet with no risk and no commitment.

Get started today, free

No credit card required, try Comet with no risk and no commitment.

Back To Top