Where AI Developers Build
Comet provides an end-to-end model evaluation platform for AI developers, with best in class LLM evaluations, experiment tracking, and production monitoring.
Trusted by the most innovative ML teams
Manage Any ML or LLM Lifecycle, From Training Through Production
Easy Integration
Add just a few lines of code to your notebook or script and automatically start tracking LLM traces, code, hyperparameters, metrics, model predictions, and more.
from comet_ml import Experiment
from transformers import Trainer
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Train your model
trainer = Trainer(
model = model,
# ...configs
)
trainer.train()
# 3. View real-time metrics in Comet
ML Experiment Management
Pytorch
from comet_ml import Experiment
import torch.nn as nn
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Create your model class
class RNN(nn.Module):
#... Define your Class
# 3. Train and test your model while logging everything to Comet
with experiment.train():
# ...Train your model and log metrics
experiment.log_metric("accuracy", correct / total, step = step)
# 4. View real-time metrics in Comet
Pytorch Lightning
from pytorch_lightning.loggers import CometLogger
# 1. Create your Model
# 2. Initialize CometLogger
comet_logger = CometLogger()
# 3. Train your model
trainer = pl.Trainer(
logger=[comet_logger],
# ...configs
)
trainer.fit(model)
# 4. View real-time metrics in Comet
Hugging Face
from comet_ml import Experiment
from transformers import Trainer
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Train your model
trainer = Trainer(
model = model,
# ...configs
)
trainer.train()
# 3. View real-time metrics in Comet
Keras
from comet_ml import Experiment
from tensorflow import keras
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Define your model
model = tf.keras.Model(
# ...configs
)
# 3. Train your model
model.fit(
x_train, y_train,
validation_data=(x_test, y_test),
)
# 4. Track real-time metrics in Comet
Tensorflow
from comet_ml import Experiment
import tensorflow as tf
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Define and train your model
model.fit(...)
# 3. Log additional model metrics and params
experiment.log_parameters({'custom_params': True})
experiment.log_metric('custom_metric', 0.95)
# 4. Track real-time metrics in Comet
Scikit-learn
from comet_ml import Experiment
import tree from sklearn
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Build your model and fit
clf = tree.DecisionTreeClassifier(
# ...configs
)
clf.fit(X_train_scaled, y_train)
params = {...}
metrics = {...}
# 3. Log additional metrics and params
experiment.log_parameters(params)
experiment.log_metrics(metrics)
# 4. Track model performance in Comet
XGBoost
from comet_ml import Experiment
import xgboost as xgb
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Define your model and fit
xg_reg = xgb.XGBRegressor(
# ...configs
)
xg_reg.fit(
X_train,
y_train,
eval_set=[(X_train, y_train), (X_test, y_test)],
eval_metric="rmse",
)
# 3. Track model performance in Comet
Any framework
# Utilize Comet in any environment
from comet_ml import Experiment
# 1. Define a new experiment
experiment = Experiment(project_name="YOUR PROJECT")
# 2. Model training here
# 3. Log metrics or params over time
experiment.log_metrics(metrics)
#4. Track real-time metrics in Comet
OPIK LLM Evaluation
OpenAI
import comet_llm
from openai import OpenAI
#1. Initialize Comet
comet_llm.init(
api_key="YOUR_COMET_API_KEY",
project="openai-example",
)
#2. Send Prompts to OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
{"role": "user", "content": "Where was it played?"}
]
)
#3. View Prompt Responses in Comet
LangChain
import comet_llm
from langchain.callbacks.tracers.comet import CometTracer
from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage
#1. Initialize Comet
comet_llm.init(
api_key="YOUR_COMET_API_KEY",
project="langchain_example",
)
#2. Send Prompts to LLM
callbacks = [CometTracer()]
chat_model = ChatOpenAI(callbacks=callbacks)
text = "What would be a good company name for a company that makes colorful socks?"
messages = [HumanMessage(content=text)]
chat_model.invoke(messages)
#3. View Prompt Responses and Chains in Comet
Any Framework
#Utilize Comet in any environment
import comet_llm
#1. Initialize Comet
comet_llm.init(
api_key="YOUR_API_KEY",
project="YOUR_LLM_PROJECT",
)
#2. Log prompt to Comet
comet_llm.log_prompt(
prompt = "",
output = "",
)
#3. View Prompt History in Comet
Model Monitoring
Any Framework
# Utilize Comet in any environment
from comet_mpm import CometMPM
# 1. Create the MPM logger
MPM = CometMPM()
# 2. Add your inference logic here
# 3. Log metrics or params over time
MPM.log_event(
prediction_id="...",
input_features=input_features,
output_value=prediction,
output_probability=probability,
)
An End-to-End Model Evaluation Platform
Comet’s end-to-end model evaluation platform for developers focuses on shipping AI features, including open source LLM tracing, ML unit-testing, evaluations, experiment tracking and production monitoring.
Track and compare your training runs, log and evaluate your LLM responses, version your models and training data, and monitor your models in production – all in one platform.
Where AI Developers Build
Run Comet’s end-to-end evaluation platform on any infrastructure to see firsthand how Comet’s reshapes your workflow. Bring your existing software and data stack. Use code panels to create visualizations in your preferred user interfaces.
Infrastructure
An AI Platform Built for Enterprise, Driven by Community
Comet’s end-to-end evaluation platform is trusted by innovative data scientists, ML practitioners, and engineers in the most demanding enterprise environments.
Get started today, free
No credit card required, try Comet with no risk and no commitment.