Skip to content
To Legacy DocumentationTo Comet website
Doc version: 2.2.11

Self Serve

Basic Requirements

To deploy self hosted Comet application you will need the following:

  • Server with supported Linux distribution, adequate storage space and backup cabability.
  • HTTP/HTTPS connectivity between the Comet SDK (where you train experiments) and the Comet application
  • License token: This token is a string of characters that will be requested during the installation process. The license token is unique to your organization and is required to activate the application.

Installation Steps

Installing Cometctl

Fetching the cometctl utility:

curl -sL https://installer.comet.com/comet-installer.sh | sudo bash
This command pulls and executes a bash setup script that will provision cometctl Once this command completes you can run cometctl from the cli using the cometctl commmand.

Initialize the Comet repository:

cometctl init --license-token {provided token}

Include your provided license token as received from the sales department or via email. This command will initialize the comet-ml package repository, and takes care of modifying the appropriate system files to allow the comet packages to be installed and updated.

Install Comet Enterprise:

cometctl aio install

As part of the installation process, cometctl will prompt you to set your comet_base_domain, when it does, you can set it to the IP or domain from the basic requirements

After that, you should see a splash screen showing a successful install, with instructions on how to access the Comet GUI. This will be the same IP or domain from the basic requirements. If you did not set the domain, or would like to change it for any reason, run cometctl aio update-config and set or change the comet_base_domain variable to your new IP or domain.

If you made any changes, you must run cometctl aio update-config

Verify Installation

Open your browser from any machine within the network and visit http://{replace-me}.com Sign up to your Comet enterprise version by clicking "Log in" on the top right corner

Python SDK Configuration

On every trainer machine where you install the Comet SDK (i.e pip install comet_ml) you will also need to define your internal Comet installation

import os
os.environ["COMET_URL_OVERRIDE"]="http://{replace-me}/clientlib/"
from comet_ml import Experiment

Environment variable:

export COMET_URL_OVERRIDE=http://{replace-me}/clientlib/

Config file:

[comet]
url_override=http://{replace-me}/clientlib/

Create a test.py file with this code snippet to test the installation:

import os
os.environ["COMET_URL_OVERRIDE"] = "http://{replace-me}/clientlib/"
from comet_ml import Experiment
from time import sleep
experiment = Experiment(project_name="test-comet")
experiment.set_name("testing comet")
experiment.log_other("finished", False)
for y in range(10):
   experiment.log_metric("a", y)
   experiment.log_parameter("paramA", 0.31000005)
   sleep(0.1)
experiment.log_other("finished", True)

Run:

pip install comet-ml
python test.py

Server Requirements

Hardware requirements

For a single server deployment, also called all-in-one, we recommend the following specifications. Please note that these specifications may vary depending on usage, simultaneous training, and the number of users.

  • CPU: At least 16 vCPUs, such as an Intel Xeon or equivalent processor
  • Memory: At least 32 GB of RAM, with 64 GB recommended for optimal performance
  • Storage: At least 1 TB of root disk space, preferably SSD. The disk space allocation may be adjusted downward if you plan on storing experiment data on another partition or offsite.
  • Network: At least 10 Gbps network interface

If you're using a public cloud provider, the following instance types are recommended for use:

  • Amazon Web Services (AWS): m5.4xlarge or larger instance types
  • Microsoft Azure: D16 v3 or larger instance types
  • Google Cloud Platform (GCP): n1-standard-16 or larger instance types

The application supports both RedHat and Debian-based distributions, including:

  • Ubuntu 18.04, 20.04 and 22.04
  • Amazon Linux 2
  • RedHat Enterprise Linux / CentOS 8

If you have the option to choose any distribution from these options, we recommend selecting Ubuntu. This is because Ubuntu provides most of the software dependencies required by the application in its package manager, making the installation process smoother and faster.

Enabling SSL

Run cometctl aio enable-ssl

Setup either a cloud Load Balancer with an SSL Certificate attached or make use of LetsEncrypt. If using AWS, you must use an Application Load Balancer or any other LB operating at layer 7. Elastic and Classic load balancers will not work as they operate on layer 4.

If using LetsEncrypt, run: cometctl aio enable-ssl --letsencrypt

Upgrade

For Debian based distros

apt update && apt install comet-ml-installer comet-ml-prod comet-ml-frontend

For RPM based distros:

dnf install comet-ml-installer comet-ml-prod comet-ml-frontend

Run the following command after to restart all comet services

cometctl aio restart-services

Updating config

Invoke cometctl aio update-config which will open the relevant configuration file and run basic validation on saving.

Comet Onprem useful administration scripts:

Restart all Comet related services

cometctl aio restart-services

Performs a quick health check to verify if the services are up and running

cometctl aio healthcheck

Enabling SMTP

SMTP is enabled by default.
The Comet SMTP server is currently supported on a best-effort basis only.

If you want to provide your own SMTP server, run cometctl aio update-config and make sure to change the following values:

smtp_enabled: true
smtp_host: <host>
smtp_port: <port>
smtp_user: <user>
smtp_password: <password>
smtp_protocol: <smtp or smtps>
mail_from: <user@yourcompanymail.com>

For additional documentation, please contact Comet at support@comet.com.

Backups

We recommend setting up a policy on your cloud provider to rotate snapshots from your instance volume(s). AWS guide GCP guide Azure guide Bare Metal: We recommend you periodically take a backup of the disk or volume containing your experiment data using a method of your choosing.

Server Usage Report and BI Events

Comet collects minimal usage reports and stats on the installation. Firewall whitelist information is as follows

Port: 443
Host: stats.comet.com
IP: 3.210.8.13
Proto: TCP

Hostname and IP are statically set and should not change. A complete sample usage report can be found below:

{
    "usage_id": "2022-12",
    "report": [
      {
        "username": "benjamin",
        "email": "z@gmail.com",
        "created_at": 1662014273604,
        "uiUsageCount": 1,
        "uiUsageUpdateTs": 1662014275281,
        "sdkUsageCount": 0,
        "sdkUsageUpdateTs": 1662014275281,
        "suspended": false
      },
    ],
    "number_of_users": 1,
    "licenseKey": "licenseKey1234",
    "domain": "gmail.com",
    "backendVersion": "3.2.74",
    "frontendVersion": "NA"
  }

Below is an anonymized version of an actual BI event report:

{
  "anonymousId": null,
  "context": {
    "library": {
      "name": "analytics-python",
      "version": "2.2.1"
    }
  },
  "event": "new_experiment_be",
  "integrations": {},
  "messageId": "e400c2e6-d161-4225-a992-a3621fff95eb",
  "originalTimestamp": "2023-01-13T14:40:41.337309+00:00",
  "properties": {
    "backendVersion": "3.2.165",
    "client": "my-company",
    "domain": "my-company.com",
    "experimentKey": "c249967859b24153b3c8659aac9g3dab",
    "frontendVersion": "5.20.2",
    "licenseKey": "pffka9493j2bd963do98Tfabccf80997r",
    "on_prem": true,
    "optimizationId": null,
    "organizationId": "PJ9ULeGB7kqQ4G7J08DsrgpoS",
    "organizationName": "defaultOrganization",
    "projectId": "a6e22f54d84a548b3f1760bf6421602b",
    "projectName": "my-recommender-model",
    "pyVersion": "3.8.12",
    "sdkVersion": "3.31.22",
    "status": "true",
    "teamName": "my-team",
    "version": "3.31.22"
  },
  "receivedAt": "2023-01-13T14:40:41.956Z",
  "sentAt": "2023-01-13T14:40:41.714Z",
  "timestamp": "2023-01-13T14:40:41.579Z",
  "type": "track",
  "userId": "my-name",
  "writeKey": "zZVvzqFNeXDt2KSyeGofvrsKY6osd6Ht"
}

If you have any questions regarding Comet Usage and Events Reporting, please contact support@comet.com or your account manager.

Caveats and FAQ

Q: I'm seeing a prompt asking me to restart as part of the install A: On certain newer distributions of Linux, you may be prompted as part of cometctl init or cometctl install with an ncurses prompt asking whether you'd like to restart any services. If you do not know what to do here, you may hit enter and bypass this.

Q: The curl command fails to install! A: Certain distributions will prompt for a confirmation to install the packages pulled in by the script pulled in by the first command. Save the script first and run it as a separate command and hit Y when prompted to install the packages.

May. 24, 2023