{"id":4133,"date":"2022-10-20T13:32:35","date_gmt":"2022-10-20T21:32:35","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4133"},"modified":"2025-04-24T17:17:03","modified_gmt":"2025-04-24T17:17:03","slug":"keep-track-of-machine-learning-experiments-with-comet","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/keep-track-of-machine-learning-experiments-with-comet\/","title":{"rendered":"Keep Track of Machine Learning Experiments With Comet"},"content":{"rendered":"\n
\n

If you\u2019re a software developer, you\u2019ve probably heard of Git (a version control system that allows you to keep track of any changes you make to your code) and GitHub (a hosting platform for Git). But, it turns out that there is a similar platform for those of us in the machine learning and data science worlds. Comet<\/a> (the GitHub for machine learning models) provides tools to assist us with tasks such as model tracking and version control, as well as model production management. In this post, we will learn more about what Comet is and how Comet can assist us in tracking the progress of our machine learning models. Without further ado, let\u2019s get started!<\/p>\n

What is Comet?<\/h2>\n

Comet<\/a> is a platform for managing and optimizing your machine learning lifecycle, from experiment tracking to model production monitoring. What that means \u2014 in simple terms \u2014 is that Comet allows you to compare and audit your machine learning model\u2019s experiments and artifacts (including your model hyperparameters, metrics, and more) in an optimal and effective manner.<\/p>\n

Consider the case in which you\u2019re building a classification model and want to tune it using GridSearchCV<\/code>. While tuning the model, you want to keep track of model metrics such as a precision score and the ROC-curve visualization. Instead of logging each metric manually in a notebook or spreadsheet, which is very inefficient, Comet enables you to automatically track and optimize all of these metrics without even batting an eye.<\/p>\n

Comet is an excellent tool for successfully organizing your machine learning model experiments. It offers several perks, including the ability to integrate with many common machine learning packages and platforms (e.g. TensorFlow, Keras, PyTorch, etc.), and is compatible with several mainstream data science and machine learning programming languages (e.g. Python, R, Java, etc.). And, it is completely free for individual use.<\/p>\n

Who Can Use Comet?<\/h2>\n

While Comet is often used for enterprise business applications, it can also be extremely handy for small- and mid-scale personal projects. In fact, Comet is remarkably accessible to beginner, intermediate, and professional users alike, so, whatever your skill-level, don\u2019t be afraid to dive in!<\/p>\n

In what type of scenario might beginner-level individuals employ Comet? Say you\u2019re developing a model for a Kaggle competition and you want to keep track of each ROC-curve visualization as you modify the model\u2019s hyperparameters. Comet will help you to achieve this, and as a result, will help you obtain better model results, more efficiently.<\/p>\n

Now that we understand how Comet may be useful for your particular use-case, let\u2019s test it out and see what we can do with it.<\/p>\n

Creating Our First Model Tracking Experiment Demo With Comet<\/h2>\n

After we\u2019ve fitted the first version of our model on the training set, and evaluated it on the validation set, it\u2019s time to start tuning the hyperparameters to ensure we\u2019re getting the best performance possible. This process is commonly referred to as an experiment, because each time we tune the model, we are testing how well it performs based on that particular set of hyperparameters.<\/p>\n

Keeping track of many iterations of experiments can be chaotic and time-consuming. Consider saving the confusion matrix for each set of hyperparameters passed to your model, and then trying to remember which matrix corresponded to which set of hyperparameters.<\/p>\n

Comet can assist us in organizing and managing this process, and we\u2019ll do exactly this in our demo. We will use Comet to keep track of our experiments and then, among the experiments we\u2019ve tracked, we\u2019ll choose the optimal set of hyperparameters for our model.<\/p>\n

Dataset Used<\/h2>\n

For this classification task, I\u2019ll be utilizing a dataset that determines whether a customer is satisfied or dissatisfied with a product, based on a set of predictor variables. Now, without further ado, let\u2019s get started.<\/p>\n

Signing-up on Comet<\/h2>\n

The first step is to sign-up on the Comet platform in order to obtain an API key that will allow our code in our IDE to communicate with the Comet website. We can do so by clicking here<\/a>. After we\u2019ve signed up and verified our account, we should see something like this in our dashboard:<\/p>\n

\n
\"\"<\/figure>

Signing in to Comet<\/picture><\/p>\n

 <\/p>\n<\/div>\n

Now, in the upper-right corner, we can click on our profile and navigate to settings, where we can see our API key. We click on the API key tab and will see something like this:<\/p>\n

\"\"<\/figure>
<\/picture><\/div>\n

Locating our API key<\/p>\n

 <\/p>\n

Once we\u2019ve completed that, the next step is to install the Comet library locally which we can do by running either of the following commands:<\/p>\n

pip install comet_ml<\/span><\/pre>\n
\u2014 or \u2014<\/p>\n
conda install -c comet_ml comet_ml<\/span><\/pre>\n
Once we\u2019ve installed the comet library, it\u2019s time to open our Jupyter notebook.<\/p>\n<\/div>\n\n\n\n
\n
Is your current workflow siloed and impossible to track? It might be time for an MLOps strategy. We created a free guide on how to create effective ML teams just for you. Learn more today!<\/a><\/p>\n<\/div>\n\n\n\n
Setting the Model\/Algorithm Parameter<\/h2>\n\n\n\n
As previously mentioned, we will be focusing on hyperparameter optimization in this demo. We will use Comet to keep track of the hyperparameters used in each experiment, and we will then compare those experiments to see which set of hyperparameters performs the best.<\/p>\n\n\n\n
The first step is to specify which hyperparameters we want to tune, and which values (or range of values) we\u2019d like to test. We do this by defining a dictionary of model parameters (model_params<\/code>) where each key represents the parameter name and the value represents the value we want to pass to the parameter. The value of each parameter is further composed of two key-value pairs, representing the type<\/code> and values<\/code> of the parameter ranges. This is because Comet offers various ways of dealing with several types of values: integer, double or float, discrete (list of numbers), and categorical (list of strings). Let\u2019s take a look at the parameters we want to pass into our model to get a better idea of what this looks like.<\/p>\n\n\n\n
#Specify the hyperparameters we want to tune in our algorithm\/estimator\nmodel_params = {\n     \"n_estimators\": {\n         \"type\": \"discrete\",\n         \"values\": [50, 100, 150, 200, 250, 300]\n     },\n     \"criterion\": {\n         \"type\": \"categorical\",\n         \"values\": [\"gini\", \"entropy\"]\n     },\n     \"min_samples_leaf\": {\n         \"type\": \"discrete\",\n         \"values\": [1, 2, 3, 4]\n     }\n}<\/pre>\n\n\n\n
Specifying our hyperparameters<\/p>\n\n\n\n
In our example, we wish to pass a list of numbers to n_estimators<\/code> and categorical values to criterion<\/code>. Click here<\/a> to learn more about how Comet handles parameter values.<\/p>\n\n\n\n
The Comet Optimizer<\/h2>\n\n\n\n
After we\u2019ve done specifying the hyperparameters, it\u2019s time to instantiate our Comet optimizer. The Comet optimizer is used to identify the best set of hyperparameter values that will minimize or maximize a specific metric on the fly. Because of its connection with Comet\u2019s Experiment objects, Comet\u2019s optimizer offers many advantages over typical hyperparameter optimizer search services.<\/p>\n\n\n\n
# Initialize our optimizer\nopt = comet_ml.Optimizer(api_key=API_KEY, config=optimizer_dict)<\/pre>\n\n\n\n
Instantiating our optimizer<\/p>\n\n\n\n
The Comet optimizer provides three types of hyperparameter tuning algorithms: Grid search (similar to GridSearchCV from Scikit-learn), Random search (similar to Random Search in Scikit-learn), and Bayes optmization. According to the Comet documentation, the Bayes optimization algorithm is often the most efficient and effective one to utilize, and therefore we will be using it in this tutorial.<\/p>\n\n\n\n
The Comet optimizer config parameters accept a dictionary consisting of the following:<\/p>\n\n\n\n