Hyperparameter tuning is one of the most important tasks in a Data Science project lifecycle because it determines the performance of our Machine Learning model.<\/p>\n
Many tools and strategies can be used to perform hyperparameter tuning, including (but not limited to) the following well-known Python libraries:<\/p>\n
- \n
- Tree-based Pipeline Optimization Tool (TPOT)<\/li>\n
- Hyperopt-Sklearn<\/li>\n
- Auto-Sklearn<\/li>\n<\/ul>\n
In this article, I focus on the Comet Optimizer<\/a>, provided by Comet<\/a>. With respect to the previous libraries, Optimizer is already integrated with Comet Experiments, thus results of each test can be visualized directly in Comet.<\/p>\n
Comet<\/a> is a platform for Machine Learning experimentation, which provides a variety of features. You can track experiments and their results, collaborate with other users, and optimize experiments using Comet\u2019s algorithms. When using Comet, you can compare datasets and code changes as they relate to experiments. Trying to find the best dataset or the best model parameters? Comet can help.<\/strong><\/p>\n
In this article, I describe how to exploit Comet for hyperparameter tuning. The article is organized as follows:<\/p>\n
- \n
- Overview of Comet Optimizer<\/li>\n
- Configuration of Parameters<\/li>\n
- Running Experiments<\/li>\n
- Show Results<\/li>\n<\/ul>\n
Overview of Comet Optimizer<\/h1>\n
The Comet Optimizer is used to tune hyperparameters of a model by maximizing or minimizing a particular metric. The Comet Optimizer supports three different optimization algorithms including Grid<\/em>, Random<\/em>, and Bayes<\/em>. In addition Comet also provides a mechanism to specify your optimization algorithm. For details about each optimization algorithm, you can refer to the Comet official documentation<\/a>.<\/p>\n
The
Optimizer<\/code> class is the main class for hyperparameter tuning in Comet. I assume that you have already installed thecomet<\/code>Python package. If you have not installed it yet, you can refer to my previous article<\/a>.<\/p>\nBefore creating an
Optimizer<\/code>, you should specify some configuration parameters, which include the optimization algorithm, the metric to be maximized\/minimized, the number of trials, and the parameters to be tested:<\/p>\nconfig = {\"algorithm\": \"bayes\",\n \"spec\": {\n \"maxCombo\": 0,\n \"objective\": \"minimize\",\n \"metric\": \"loss\",\n \"minSampleSize\": 100,\n \"retryLimit\": 20,\n \"retryAssignLimit\": 0,\n },\n \"trials\": 1,\n \"parameters\": my_params,\n \"name\": \"MY-OPTIMIZER\"\n}<\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nInnovation and academia go hand-in-hand. Listen to our own CEO Gideon Mendels chat with the Stanford MLSys Seminar Series team<\/a> about the future of MLOps and give the Comet platform a try for free!<\/a><\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n\n\nConfiguration of Parameters<\/h1>\n
Similar to the other tools for hyperparameter tuning, the Comet Optimizer requires specifying for each parameter to be tested the range of possible values, with a syntax depending on the type of parameter (integer, categorical, and so on). For example, for an integer<\/strong>, the following configuration should be used:<\/p>\n
{\"PARAMETER-NAME\":\n {\"type\": \"integer\",\n \"scalingType\": \"linear\" | \"uniform\" | \"normal\" | \"loguniform\" | \"lognormal\",\n \"min\": MY-MIN-VALUE,\n \"max\": MY-MAX-VALUE,\n }<\/span><\/pre>\nAnd for a categorical parameter<\/strong>, the following configuration should be used:<\/p>\n
{\"PARAMETER-NAME\":\n {\"type\": \"categorical\",\n \"values\": [\"LIST\", \"OF\", \"CATEGORIES\"]\n }\n }<\/span><\/pre>\nFor example, to tune a scikit-learn K-Neighbors Classifier you could use the following parameter configuration:<\/p>\n
my_params = {\n 'n_neighbors<\/strong>':{\n \"type\" : \"integer\",\n \"scalingType\" : \"linear\",\n \"min\" : 3,\n \"max\" : 8,\n },<\/span> 'weights<\/strong>':{\n \"type\" : \"categorical\",\n \"values\" : ['uniform', 'distance'],\n },\n 'metric<\/strong>': {\n \"type\" : \"categorical\",\n \"values\" : ['euclidean', 'manhattan', 'chebyshev', 'minkowski']\n },\n 'algorithm<\/strong>': {\n \"type\" : \"categorical\",\n \"values\" : ['ball_tree', 'kd_tree']\n }\n}<\/span><\/pre>\nRunning Experiments<\/h1>\n
You can create an
Optimizer<\/code> object as follows:<\/p>\nfrom comet_ml import Optimizer<\/span>opt = Optimizer(config,\n api_key=\"MY-API-KEY\",\n project_name=\"MY-PROJECT-NAME\",\n workspace=\"MY-WORKSPACE\")<\/span><\/pre>\nA Comet Optimizer creates a different experiment for each combination of parameters to be tested. Thus, you can iterate over the list of experiments and fit a new model for each experiment.<\/p>\n
For example, for the K-Neighbors classifier of the previous example, you can write the following code:<\/p>\n
from sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.metrics import f1_score, precision_score, recall_score<\/span>for experiment in opt.get_experiments()<\/strong>:\n model = KNeighborsClassifier<\/strong>(\n n_neighbors=experiment.get_parameter(\"n_neighbors\"),\n weights=experiment.get_parameter(\"weights\"),\n metric=experiment.get_parameter(\"metric\"),\n algorithm=experiment.get_parameter(\"algorithm\")\n )\n loss = model.fit(X_train, y_train)\n experiment.log_metric(\"loss\", loss)<\/span> y_pred = model.predict(X_test)\n f1 = f1_score(y_test, y_pred)\n precision = precision_score(y_test, y_pred)\n recall = recall_score(y_test, y_pred)<\/span> experiment.log_metric(\"f1\", f1)\n experiment.log_metric(\"precision\", precision)\n experiment.log_metric(\"recall\", recall)<\/span><\/pre>\nIn the previous example, for each experiment, precision, recall, and f1-score are calculated and logged through the
log_metric()<\/code> function.<\/p>\nShow Results<\/h1>\n
When all the experiments finish, you can view results in Comet. For each experiment, you can view the configuration parameter as well as the performance metric, as shown in the following Figure:<\/p>\n
\n \n![\"Hyperparameter](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*0JfvD-u-l3N_BIzndUAT_w.png\")
<\/figure><\/picture><\/div>\n<\/div>\n Image by Author<\/figcaption>\n<\/figure>\n In addition, you can build a panel with a specific metric for all the experiments. For example, the following Figure shows the precision for all the experiments:<\/p>\n
\n \n![\"An](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*jsmKOHKAbky0ini3duNVNQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\n Image by Author<\/figcaption>\n<\/figure>\n I have highlighted the specific bar with the following parameters:<\/p>\n
- \n
- n_neighbors = 4<\/li>\n
- weight = uniform<\/li>\n
- algorithm = ball_tree<\/li>\n<\/ul>\n
By comparing all the results, you can easily choose the best parameters for your model, and use them in production.<\/p>\n
Summary<\/h1>\n
Congratulations! You have just learned how to tune hyperparameters in Comet! You can just exploit the ready-to-use
Optimizer<\/code> class! For more details, you can read the Comet documentation<\/a>.<\/p>\nThere are many other things you can do with Comet, such as using it in conjunction with Gitlab<\/a>, as I described in my previous article<\/a>.<\/p>\n
Now you just have to try it \u2014 have fun!<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Photo by Walter Lee Olivares de la Cruz on Unsplash Hyperparameter tuning is one of the most important tasks in a Data Science project lifecycle because it determines the performance of our Machine Learning model. Many tools and strategies can be used to perform hyperparameter tuning, including (but not limited to) the following well-known Python […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[9,7],"tags":[],"coauthors":[132],"class_list":["post-7792","post","type-post","status-publish","format-standard","hentry","category-product","category-tutorials"],"yoast_head":"\n
Hyperparameter Tuning in Comet - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hyperparameter Tuning in Comet\" \/>\n<meta property=\"og:description\" content=\"Photo by Walter Lee Olivares de la Cruz on Unsplash Hyperparameter tuning is one of the most important tasks in a Data Science project lifecycle because it determines the performance of our Machine Learning model. Many tools and strategies can be used to perform hyperparameter tuning, including (but not limited to) the following well-known Python […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-04T18:58:27+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:06:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs\" \/>\n<meta name=\"author\" content=\"Angelica Lo Duca\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Angelica Lo Duca\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Hyperparameter Tuning in Comet - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/","og_locale":"en_US","og_type":"article","og_title":"Hyperparameter Tuning in Comet","og_description":"Photo by Walter Lee Olivares de la Cruz on Unsplash Hyperparameter tuning is one of the most important tasks in a Data Science project lifecycle because it determines the performance of our Machine Learning model. Many tools and strategies can be used to perform hyperparameter tuning, including (but not limited to) the following well-known Python […]","og_url":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-04T18:58:27+00:00","article_modified_time":"2025-04-24T17:06:04+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs","type":"","width":"","height":""}],"author":"Angelica Lo Duca","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Angelica Lo Duca","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Hyperparameter Tuning in Comet","datePublished":"2023-10-04T18:58:27+00:00","dateModified":"2025-04-24T17:06:04+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/"},"wordCount":673,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs","articleSection":["Product","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/","url":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/","name":"Hyperparameter Tuning in Comet - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs","datePublished":"2023-10-04T18:58:27+00:00","dateModified":"2025-04-24T17:06:04+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*m0F4yTHUl6hCfXSs"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-in-comet\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Hyperparameter Tuning in Comet"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7792","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7792"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7792\/revisions"}],"predecessor-version":[{"id":15521,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7792\/revisions\/15521"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7792"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7792"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7792"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}