{"id":4428,"date":"2022-10-31T09:35:21","date_gmt":"2022-10-31T17:35:21","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4428"},"modified":"2025-04-24T17:16:48","modified_gmt":"2025-04-24T17:16:48","slug":"7-optimization-methods-used-in-deep-learning","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/","title":{"rendered":"7 Optimization Methods Used In Deep Learning"},"content":{"rendered":"\n
\n
\n
\n
\"\"<\/figure>
Photo by <\/picture>Jo Coenen – Studio Dries 2.6<\/a> on <\/picture>Unsplash<\/a><\/div>\n<\/div>\n<\/figure>\n

Optimization plays a vital role in the development of machine learning and deep learning algorithms \u2014 without it, our model would not have the best design. The procedure refers to finding the set of input parameters or arguments to an objective function that results in the minimum or maximum output of the function \u2014 usually the minimum in a machine learning\/deep learning context.<\/p>\n

In this article, we are going to cover the top seven most common optimization methods used in deep learning.<\/p>\n

#1 Gradient Descent<\/h3>\n

Getting a sound understanding of the inner workings of gradient descent<\/a> is one of the best things you could do for your career in ML\/DL. It\u2019s one of the most popular optimization algorithms and comes up constantly in the field. Gradient descent is a first-order, iterative optimization method \u2014 first-order means we calculate only the first-order derivative.<\/p>\n

To find the minimum of the function, steps are taken in proportion to the negative gradient of the function at its current point. The initial point on the function is random.<\/p>\n

\n
\n
\"\"<\/figure>
Example of minimizing J(w); [Source: <\/picture>MLextend<\/a>]<\/picture><\/div>\n<\/div>\n<\/figure>\n

Gradient descent isn\u2019t perfect though. An update to our parameters can only be made once we\u2019ve iterated through the whole dataset which makes learning extremely slow when the data is large. Various adaptions to gradient descent<\/a> were made to overcome this problem. They include mini-batch gradient descent<\/strong> and stochastic gradient descent.<\/strong><\/p>\n

#2 Momentum<\/h3>\n

The two variations of gradient descent mentioned above \u2014 stochastic and mini-batch \u2014 oscillate as they step towards the minimum of the objective function. These oscillations occur because variance is introduced at each step since an update to the parameters is made after iterating through every n<\/em> number of instances.<\/p>\n

\n
\n
\"\"<\/figure>
Conceptual view of how different variations of gradient descent step towards the minimum of an objective function; [Source: <\/picture>i2tutorials<\/a>]<\/picture><\/div>\n<\/div>\n<\/figure>\n

Momentum extends on gradient descent<\/a>, hence we typically refer to it as gradient descent with momentum. The technique seeks to overcome the oscillation problem by adding history to the parameter update equation. The idea behind momentum is that with an understanding of the direction required to reach the minimum of the function faster, we can make our gradient steps move in that direction and reduce the oscillations in irrelevant directions.<\/p>\n

The update parameter from the previous step, Vt\u22121, is added to the gradient step, \u25bd\u03b8J(\u03b8). How much information from historical steps depends on the value \u03b3, and the step size is controlled by the learning rate, \u014b.<\/p>\n

\n
\"\"<\/figure>
Introducing momentum to gradient descent<\/picture><\/div>\n<\/figure>\n

To update the parameter with momentum, we would use the following equation:<\/p>\n

\n
\"\"<\/figure>
Parameter update with momentum<\/picture><\/div>\n<\/figure>\n

This adjustment applied to mini-batch and stochastic gradient descent will help to reduce oscillations in each gradient step which in turn would speed up convergence.<\/p>\n

#3 Nesterov Accelerated Gradient (NAG)<\/h3>\n

Momentum-based optimization takes steps towards the minimum based on past steps, which reduces the oscillations we see in mini-batch and stochastic gradient descent. One problem arises when we use momentum is that we may miss the minimum value of the objective function. This is because, as we approach the minimum, the value for momentum is high.<\/p>\n

\u201cWhen the value of momentum is high while we are near to attain convergence, then the momentum actually pushes the gradient step high and it might miss out on the actual minimum value.\u201d \u2014 Hands-on Deep Learning Algorithms with Python,<\/strong> P.101<\/strong><\/p><\/blockquote>\n

Nesterov introduced an intelligent new method to help overcome this problem. The idea behind the method is to calculate the gradient of where the momentum would take us too, instead of calculating the gradient at the current position.<\/p>\n

\u201cThus, before making gradient step with momentum and reaching a new position, if we understand which position the momentum will take us to, then we can avoid overshooting the minimum value.\u201d \u2014 Hands-on Deep Learning Algorithms with Python,<\/strong> P.102<\/strong><\/a><\/p><\/blockquote>\n

Upon learning that momentum is going to push us past the minimum value, we could reduce the speed of momentum to try to reach the minimum value.<\/p>\n

To identify the position in which we\u2019d arrive at with momentum, we calculate the gradients with respect to the approximate position of where our next gradient step will be (the lookahead position):<\/p>\n

\n
\"\"<\/figure>
Calculating the gradient of where momentum will take us<\/picture><\/div>\n<\/figure>\n

We can rewrite the Vt equation presented in momentum as follows:<\/p>\n

\n
\"\"<\/figure>
Nesterov accelerated gradient mathematical expression.<\/picture><\/div>\n<\/figure>\n

The parameter update step remains the same.<\/p>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n
\n

Join 16,000 of your colleagues at Deep Learning Weekly<\/a> for the latest products, acquisitions, technologies, deep-dives and more.<\/p><\/blockquote>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n
\n

#4 Adaptive Gradient (AdaGrad)<\/h3>\n

Adaptive gradient uses parameter-specific learning rates which are constantly adapted in accordance with how frequently a parameter update is made during training.<\/p>\n

\u201cParameters that have frequent updates or high gradients will have a slower learning rate, while a parameter that has an infrequent update or small gradients will also have a slower learning rate. [\u2026] parameters that have infrequent updates implies that they are not trained enough, so we set a high learning rate for them, and parameters that have frequent updates implies they are trained enough, so we set their learning rate to a low value so we don’t overshoot the minimum.\u201d \u2014 Hands-on Deep Learning Algorithms with Python,<\/strong> P.103<\/strong><\/p><\/blockquote>\n

Let\u2019s see how this looks from a mathematical standpoint. For simplicity\u2019s sake, we are going to represent the gradient as g. <\/em>Thus, the gradient of parameter, \u03b8i<\/em>, at an iteration, t<\/em>, can be represented as:<\/p>\n

\n
\"\"<\/figure>
Representing the gradient of a parameter at iteration t<\/picture><\/div>\n<\/figure>\n

Our update equation can be rewritten as:<\/p>\n

\n
\"\"<\/figure>
Revised update equation<\/picture><\/div>\n<\/figure>\n

For each iteration, t<\/em>, updating a parameter, \u03b8i, <\/em>involves diving the learning rate by the sum of squares of all previous gradients of the parameter, \u03b8i <\/em>(epsilon is added to avoid division by zero error):<\/p>\n

\n
\"\"<\/figure>
Adaptive gradient<\/picture><\/div>\n<\/figure>\n

When the sum of squares past gradients is a large value, our learning rate will be scaled to a smaller number. In contrast, if the value is low, the learning rate will be divided by a lower value, thus resulting in a higher learning rate.<\/p>\n

#5 AdaDelta<\/h3>\n

Adadelta seeks to improve on the Agagrad optimization method. Recall that in Adagrad all the past squared gradients are summed together. This means that on every iteration the sum of the past squared gradients will increase, and when the squared past gradient value is high, we will be dividing the learning rate by a large number which causes the learning rate to decay. When the learning rate is a very small value, convergence will take longer.<\/p>\n

\u201cInstead of accumulating all past squared gradients, Adadelta restricts the window of accumulated past gradients to some fixed size w<\/em>.\u201d \u2014 An Overview of Gradient Descent Optimization Algorithms<\/strong><\/a>, Sebastien Ruder<\/strong><\/p><\/blockquote>\n

To avoid the inefficient computation of squaring and storing gradients from the window, w<\/em>, in each iteration, we take the exponentially decaying running average of gradients:<\/p>\n

\n
\"\"<\/figure>
The exponentially decaying running average of gradients<\/picture><\/div>\n<\/figure>\n

The \u03b3 we saw in momentum is similar to the one we see in this equation \u2014 it is used to decide how much information from the previous running average of gradients should be added. In Adadelta, it\u2019s referred to as the exponential decaying rate<\/strong>.<\/p>\n

Our final update equation is:<\/p>\n

\n
\"\"<\/figure>
Final update equation<\/picture><\/div>\n<\/figure>\n

Note<\/strong>: See An Overview of Gradient Descent Optimization Algorithms<\/strong><\/a>, for the mathematical proof.<\/p><\/blockquote>\n

 <\/p>\n

#6 RMSProp<\/h3>\n

RMSprop was also introduced to combat the decaying learning rate problem faced in Adagrad. We also use the exponentially decaying running average of the gradients:<\/p>\n

\n
\"\"<\/figure>
The exponentially decaying running average of gradients<\/picture><\/div>\n<\/figure>\n

We then divide the learning rate (0.9 is the recommended value), n<\/em>, by this running average of gradients.<\/p>\n

\n
\"\"<\/figure>
Update equation in RMSprop<\/picture><\/div>
 <\/strong><\/figcaption><\/figure>\n

#7 Adaptive Moment Estimation (Adam)<\/h3>\n

Adam is probably the most popularly used optimization algorithm for neural networks. It combines RMSprop and momentum by not only storing the exponentially decaying average of past squared gradients (like we see in RMSprop and Adadelta), Vt, <\/em>but also storing an exponentially decaying average of the past gradients, Mt<\/em>, similar to momentum.<\/p>\n

\u201cWhereas momentum can be seen as a ball running down a slope, Adam behaves like a heavy ball with friction, which thus prefers flat minima in the error surface[15]<\/a>.\u201d \u2014 An Overview of Gradient Descent Optimization Algorithms, Sebastien Ruder<\/strong><\/p><\/blockquote>\n

We compute the decaying averages of the past, Mt<\/em>, and past squared, Vt<\/em>, gradients as follows:<\/p>\n

\n
\"\"<\/figure>
Estimates of the first and second moments of the gradients<\/picture><\/div>\n<\/figure>\n

The first moment (mean), Mt<\/em>, and second moment (uncentered variance), Vt<\/em>, are both estimates of the gradients \u2014 hence the name of the method.<\/p>\n

\u201cWhen the initial estimates are set to 0, they remain very small, even after many iterations. This means that they would be biased towards 0, especially when \u03b21<\/em> and \u03b22 are close to 1.\u201d \u2014 <\/em>Hands-on Deep Learning Algorithms with Python,<\/strong> P.112<\/strong><\/p><\/blockquote>\n

We can counteract the biases by computing the bias-corrected first and second-moment estimates:<\/p>\n

\n
\"\"<\/figure>
Bias corrected first and second moments<\/picture><\/div>\n<\/figure>\n

Our final update parameter is given as:<\/p>\n

\n
\"\"<\/figure>
Final update parameter in Adam optimization<\/picture><\/div>\n<\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n

In this article, we covered: Gradient descent, Momentum, NAG, Adagrad, Adadelta, RMSprop, and Adam optimization techniques. This is by no means an extensive list but serves as a good foundation for anyone interested in learning more about various optimizers used in deep learning. For a more in-depth look at the optimizers discussed, I highly recommend An Overview of Gradient Descent Optimization Algorithms<\/em><\/a>, by Sebastien Ruder.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Photo by Jo Coenen – Studio Dries 2.6 on Unsplash Optimization plays a vital role in the development of machine learning and deep learning algorithms \u2014 without it, our model would not have the best design. The procedure refers to finding the set of input parameters or arguments to an objective function that results in the minimum […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[138],"class_list":["post-4428","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\n7 Optimization Methods Used In Deep Learning - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"7 Optimization Methods Used In Deep Learning\" \/>\n<meta property=\"og:description\" content=\"Photo by Jo Coenen – Studio Dries 2.6 on Unsplash Optimization plays a vital role in the development of machine learning and deep learning algorithms \u2014 without it, our model would not have the best design. The procedure refers to finding the set of input parameters or arguments to an objective function that results in the minimum […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-31T17:35:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:16:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k\" \/>\n<meta name=\"author\" content=\"Kurtis Pykes\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kurtis Pykes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"7 Optimization Methods Used In Deep Learning - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/","og_locale":"en_US","og_type":"article","og_title":"7 Optimization Methods Used In Deep Learning","og_description":"Photo by Jo Coenen – Studio Dries 2.6 on Unsplash Optimization plays a vital role in the development of machine learning and deep learning algorithms \u2014 without it, our model would not have the best design. The procedure refers to finding the set of input parameters or arguments to an objective function that results in the minimum […]","og_url":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2022-10-31T17:35:21+00:00","article_modified_time":"2025-04-24T17:16:48+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k","type":"","width":"","height":""}],"author":"Kurtis Pykes","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Kurtis Pykes","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"7 Optimization Methods Used In Deep Learning","datePublished":"2022-10-31T17:35:21+00:00","dateModified":"2025-04-24T17:16:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/"},"wordCount":1588,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/","url":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/","name":"7 Optimization Methods Used In Deep Learning - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k","datePublished":"2022-10-31T17:35:21+00:00","dateModified":"2025-04-24T17:16:48+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#primaryimage","url":"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k","contentUrl":"https:\/\/miro.medium.com\/max\/700\/0*HZIz4s0KrLsr0P4k"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/7-optimization-methods-used-in-deep-learning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"7 Optimization Methods Used In Deep Learning"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4428","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=4428"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4428\/revisions"}],"predecessor-version":[{"id":15661,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4428\/revisions\/15661"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=4428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=4428"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=4428"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=4428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}