{"id":9449,"date":"2024-03-20T06:00:23","date_gmt":"2024-03-20T14:00:23","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=9449"},"modified":"2025-04-24T17:02:55","modified_gmt":"2025-04-24T17:02:55","slug":"hyperparameter-tuning-with-bayesian-optimization","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\/","title":{"rendered":"Hyperparameter Tuning With Bayesian Optimization"},"content":{"rendered":"\n
\n
<\/div>\n
\n
\n

What is Bayesian Optimization used for in hyperparameter tuning?<\/h4>\n
Photo by Abbas Tehrani<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n

 <\/p>\n

Hyperparameter tuning, the process of systematically searching for the best combination of hyperparameters that optimize a model’s performance, is critical in machine learning model development.<\/p>\n

While various techniques exist, such as grid search and random Search, Bayesian Optimization is more efficient and effective.<\/p>\n

This article explores the intricacies of hyperparameter tuning using Bayesian Optimization. We’ll cover the basics, why it’s essential, and how to implement it in Python.<\/p>\n

Let’s examine the code examples more thoroughly to understand better how to implement Bayesian Optimization for hyperparameter tuning in Python.<\/p>\n<\/div>\n<\/div>\n<\/section>\n\n\n\n

\n
\n
\n

Grid Search<\/h3>\n

Grid Search is the most straightforward method for hyperparameter tuning. It involves specifying a grid of hyperparameter values and exhaustively searching through all possible combinations.<\/p>\n

Suppose you have two hyperparameters: learning rate and batch size. You would define a set of possible values for each and then train a model for every possible pair<\/strong> of hyperparameters.<\/p>\n

This technique’s advantage is that it is simple and easy to implement, but it is computationally expensive and time-consuming, especially as the number of hyperparameters grows.<\/p>\n

Random Search<\/h3>\n

Random Search randomly samples the hyperparameter space a fixed number of times.<\/p>\n

Instead of trying all combinations like in Grid Search, it randomly selects a few and evaluates the model performance for those sets.<\/p>\n

This hyperparameter tuning technique is faster than Grid Search and can outperform it if the hyperparameter space is large, but it does not guarantee that the optimal set of hyperparameters will be found.<\/p>\n

Bayesian Optimization<\/h3>\n

Bayesian Optimization is a probabilistic model-based optimization algorithm. It builds a probability model of the objective function (i.e., the model’s performance for a given set of hyperparameters) and selects the most promising hyperparameters to evaluate the true objective function.<\/p>\n

The key advantage is that it considers past evaluations, making the Search more directed than random or grid searches.<\/p>\n

Let’s go through an example step by step, where we’ll see how to use Bayesian Optimization to tune an XGBoost classifier.<\/p>\n

1. Importing modules and loading the dataset<\/h4>\n
import<\/span> numpy as<\/span> np\nfrom<\/span> sklearn.datasets import<\/span> load_digits\nfrom<\/span> xgboost import<\/span> XGBClassifier\nfrom<\/span> skopt import<\/span> BayesSearchCV\nfrom<\/span> sklearn.model_selection import<\/span> train_test_split\n\n#Load the dataset and split it into training and test sets<\/span>\ndigits = load_digits()\nX = digits.data\ny = digits.target\nX_train, X_test, y_train, y_test = train_test_split(X, y,\n                          test_size=0.2<\/span>, random_state=42<\/span>)<\/span><\/pre>\n
In this example, we’re tuning an XGB classifier for the Digits<\/strong> dataset.<\/p>\n
2. Define the hyperparameter search space<\/h4>\n
We define the hyperparameter space using param_space<\/code>. Here, we have defined some hyperparameters, such as n_estimators<\/code> , which can vary between 50 and 100, and max_depth<\/code> , which can vary between 1 and 30 and so on.<\/p>\n
param_space = {\n 'learning_rate'<\/span>: (0.01<\/span>, 1.0<\/span>, 'log-uniform'<\/span>),\n 'max_depth'<\/span>: (1<\/span>, 50<\/span>),\n 'gamma'<\/span>: (1e-9<\/span>, 0.6<\/span>, 'log-uniform'<\/span>),\n 'n_estimators'<\/span>: (50<\/span>, 100<\/span>),\n 'degree'<\/span>: (1<\/span>, 6<\/span>),\n 'kernel'<\/span>: ['linear'<\/span>, 'rbf'<\/span>, 'poly'<\/span>]\n}<\/span><\/pre>\n
3. Initialize the Bayesian optimizer:<\/h4>\n
optimizer = BayesSearchCV(\n estimator=XGBClassifier(n_jobs=1<\/span>),\n search_spaces=param_space,\n scoring='accuracy'<\/span>,\n cv=3<\/span>,\n n_iter=50<\/span>,\n)<\/span><\/pre>\n
BayesSearchCV<\/code> takes care of the Bayesian Optimization loop. It iteratively updates the model and selects new hyperparameters to evaluate.<\/p>\n
Final Step<\/h4>\n
4. Fitting the model<\/h4>\n
#Fit the model<\/span>\noptimizer.fit(X_train, y_train)\n\n# After fitting, you can get the best parameters and score as follows:<\/span>\nbest_params = opt.best_params_\nbest_score = opt.best_score_\nprint<\/span>(f\"Best parameters: {best_params}<\/span>\"<\/span>)\nprint<\/span>(f\"Best accuracy score: {best_score}<\/span>\"<\/span>)<\/span><\/pre>\n
The BayesSearchCV<\/code> function performs Bayesian Optimization over the hyperparameters.<\/p>\n
Why Bayesian Optimization?<\/h4>\n
Using other optimization techniques, such as Grid Search, which evaluates all possible combinations, or Random Search, which samples them randomly, is computationally expensive and time-consuming.<\/p>\n
Bayesian Optimization makes intelligent choices based on past evaluations. It reduces the search time and improves the model’s performance by finding a better set of hyperparameters.<\/p>\n
This technique often outperforms Grid and Random Search, especially when the hyperparameter space is large and high-dimensional.<\/p>\n
However, the term “best” method can depend on your specific use case, the complexity of your model, and computational resources.<\/p>\n<\/div>\n<\/div>\n<\/section>\n\n\n\n
\n
<\/div>\n
\n
\n
Conclusion<\/h3>\n
The above example demonstrates how Bayesian Optimization can be practically applied to tune hyperparameters for a machine learning model. The technique is efficient and effective in finding the global optimum in the hyperparameter space.<\/p>\n
It’s a tool that every data scientist should have in their toolkit for building robust and optimized machine learning models.<\/p>\n<\/div>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"
What is Bayesian Optimization used for in hyperparameter tuning? Photo by Abbas Tehrani on Unsplash   Hyperparameter tuning, the process of systematically searching for the best combination of hyperparameters that optimize a model’s performance, is critical in machine learning model development. While various techniques exist, such as grid search and random Search, Bayesian Optimization is more […]<\/p>\n","protected":false},"author":88,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6,7],"tags":[],"coauthors":[185],"class_list":["post-9449","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-tutorials"],"yoast_head":"\nHyperparameter Tuning With Bayesian Optimization - Comet<\/title>\n<meta name=\"description\" content=\"Explore the intricacies of hyperparameter tuning using Bayesian Optimization: the basics, why it's essential, and how to implement in Python.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Hyperparameter Tuning With Bayesian Optimization\" \/>\n<meta property=\"og:description\" content=\"Explore the intricacies of hyperparameter tuning using Bayesian Optimization: the basics, why it's essential, and how to implement in Python.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-20T14:00:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:02:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8\" \/>\n<meta name=\"author\" content=\"Pralabh Saxena\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Pralabh Saxena\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Hyperparameter Tuning With Bayesian Optimization - Comet","description":"Explore the intricacies of hyperparameter tuning using Bayesian Optimization: the basics, why it's essential, and how to implement in Python.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization","og_locale":"en_US","og_type":"article","og_title":"Hyperparameter Tuning With Bayesian Optimization","og_description":"Explore the intricacies of hyperparameter tuning using Bayesian Optimization: the basics, why it's essential, and how to implement in Python.","og_url":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2024-03-20T14:00:23+00:00","article_modified_time":"2025-04-24T17:02:55+00:00","og_image":[{"url":"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8","type":"","width":"","height":""}],"author":"Pralabh Saxena","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Pralabh Saxena","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\/"},"author":{"name":"Pralabh Saxena","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/661df331deec9788343ef011c9467cc8"},"headline":"Hyperparameter Tuning With Bayesian Optimization","datePublished":"2024-03-20T14:00:23+00:00","dateModified":"2025-04-24T17:02:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\/"},"wordCount":593,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8","articleSection":["Machine Learning","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization\/","url":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization","name":"Hyperparameter Tuning With Bayesian Optimization - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#primaryimage"},"thumbnailUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8","datePublished":"2024-03-20T14:00:23+00:00","dateModified":"2025-04-24T17:02:55+00:00","description":"Explore the intricacies of hyperparameter tuning using Bayesian Optimization: the basics, why it's essential, and how to implement in Python.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#primaryimage","url":"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8","contentUrl":"https:\/\/cdn-images-1.medium.com\/max\/800\/0*IHHsC89aTxRv-6s8"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/hyperparameter-tuning-with-bayesian-optimization#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Hyperparameter Tuning With Bayesian Optimization"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/661df331deec9788343ef011c9467cc8","name":"Pralabh Saxena","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/af2f89cb395a3afe9b42605f70d9c6a7","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/1689749938719-96x96.jpg","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/09\/1689749938719-96x96.jpg","caption":"Pralabh Saxena"},"url":"https:\/\/www.comet.com\/site\/blog\/author\/pralabh-saxena2014gmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9449","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/88"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=9449"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9449\/revisions"}],"predecessor-version":[{"id":15371,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/9449\/revisions\/15371"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=9449"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=9449"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=9449"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=9449"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}