{"id":4584,"date":"2022-11-10T17:49:59","date_gmt":"2022-11-11T01:49:59","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4584"},"modified":"2025-04-24T17:16:34","modified_gmt":"2025-04-24T17:16:34","slug":"vanishing-exploding-gradients-in-deep-neural-networks","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/","title":{"rendered":"Vanishing\/Exploding Gradients in Deep Neural Networks"},"content":{"rendered":"\n
\n

Building a Neural Network model can be very complicated and tuning the Neural Network model can make it even more confusing. One of the most common problems when working with Deep Neural Networks is the Vanishing and\/or Exploding Gradient Descent. In order to prevent this from happening, one solution is initializing weights.<\/p>\n

Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback. If either Vanishing or Exploding occurs, the loss gradient will either be too small or too large, meaning the network requires more time to converge.<\/p>\n

Background on Deep Neural Networks:<\/h1>\n

A Neural Network<\/strong> is a network of biological neurons. In the use of Artificial Intelligence, Deep Neural Network contains artificial neurons or nodes.<\/p>\n

Deep Neural Networks<\/strong><\/a> are made up of nodes that contain three different layers: an input layer, one or more hidden layers, and an output layer. Each node is connected to another node and is where computation happens.<\/p>\n

    \n
  1. Input Layer<\/strong> \u2014 receives the input data<\/li>\n
  2. Hidden Layer(s)<\/strong> \u2014 perform mathematical computations on the input data<\/li>\n
  3. Output Layer<\/strong> \u2014 returns the output data.<\/li>\n<\/ol>\n

    The nodes in the Neural Networks are made up of parameters which are called weights and used to calculate a weighted sum of the inputs.<\/p>\n

    Weight<\/strong> controls the strength of the connection between two neurons. The weight is a big factor in deciding how much influence the input has on the output.<\/p>\n

    Bias <\/strong>is to guarantee that there will always be activation in the neurons, even if the input is 0. Bias will always have a value of 1 and is an additional input into the next layer.<\/p>\n

    Feedforward and Backpropagation<\/h1>\n

    A Cost Function<\/strong> is a mathematical formula used to calculate the error, it is the difference between our predicted value and the actual value. In the ideal world, we would want a Cost Function of 0, telling us that our outputs are the same as the data set outputs.<\/p>\n

    Neural Network models use a Cost Function optimization algorithm called Stochastic Gradient Descent. Its aim is to minimize the Cost Function by incrementally changing the weights of the network. Aiming to produce a set of weights that is capable of making useful predictions. In order to begin the optimization process, the algorithm requires a starting point in the space of possible weight values.<\/p>\n

    Feedforward Network<\/strong> is the process in which the result in the next neuron now becomes the input for the next neuron as the information always moves in one direction (forward).<\/p>\n

    In Neural Network, there is also a process called Backpropagation, also abbreviated as \u201cbackprop.\u201d Backpropagation<\/strong> is the messenger who tells the Neural Network whether it made a mistake when it made a prediction.<\/p>\n

    Backpropagation goes through these steps:<\/p>\n

      \n
    1. The Neural Network makes a guess about data<\/li>\n
    2. The Neural Network is measured with a loss function<\/li>\n
    3. The error is backpropagated to be adjusted and corrected<\/li>\n<\/ol>\n

      The Process:<\/h1>\n

      As the input features are propagated, going through the various hidden layers consisting of different or same activation functions, we produce a sample of predictive probabilities.<\/p>\n

      The backpropagation algorithm moves towards the input layer and away from the output layer calculating error gradients.<\/p>\n<\/div>\n\n\n\n

      <\/div>\n\n\n\n
      \n

      Most projects fail before they get to production. Check out our free ebook<\/a> to learn how to implement an MLOps lifecycle to better monitor, train, and deploy your machine learning models to increase output and iteration.<\/p><\/blockquote>\n<\/div>\n\n\n\n

      <\/div>\n\n\n\n
      \n

      The gradient of the Cost Function in the Neural Network processes in relation to each parameter; weights and biases. The algorithm then takes a Gradient Descent steps towards the minimum cost and updates the value of each parameter in the Neural Network using these updated gradients.<\/p>\n

      \n
      \n
      \"\"<\/figure>
      Source: <\/picture>researchgate<\/a><\/div>\n<\/div>\n<\/figure>\n

      Why do the Gradients Vanish or Explode?<\/h1>\n

      Vanish<\/h1>\n

      Vanishing<\/strong> is when as backpropagation occurs, the gradients normally get smaller and smaller, gradually approaching zero. This leaves the weights of the initial or lower layers unchanged, causing the Gradient Descent to never converge to the optimum.<\/p>\n

      For example, Activation Functions such as the sigmoid function have a very prominent difference between the variance of their inputs and outputs. They shrink and transform a large input space into a smaller output space, which lies between [0,1].<\/p>\n

      Looking at the graph below of the Sigmoid Function, we can see that using larger inputs, regardless if they are negative or positive will classify at either 0 or 1. However, when the Backpropagation processes, it has no gradient to propagate backward in the Neural Network. The little gradient that does exist, will continuously keep diluting as the algorithm continues to process through the top layers, leaving nothing for the lower layers.<\/p>\n

      \n
      \"\"<\/figure>
      Source: <\/picture>Machine Learning Journey<\/a><\/div>\n<\/figure>\n

      Explode<\/h1>\n

      Exploding<\/strong> is the opposite of Vanishing and is when the gradient continues to get larger which causes a large weight update and results in the Gradient Descent to diverge.<\/p>\n

      Exploding gradients occur due to the weights in the Neural Network, not the activation function.<\/p>\n

      The gradient linked to each weight in the Neural Network is equal to a product of numbers. If this contains a product of values that is greater than one, there is a possibility that the gradients become too large.<\/p>\n

      The weights in the lower layers of the Neural Network are more likely to be affected by Exploding Gradient as their associated gradients are products of more values. This leads to the gradients of the lower layers being more unstable, causing the algorithm to diverge.<\/p>\n

      \n
      \n
      \"\"<\/figure>
      Source: <\/picture>Udacity<\/a><\/div>\n<\/div>\n<\/figure>\n

      Weight Initialization<\/h1>\n

      Weight Initialization is the process of setting the weights of a Neural Network to small random values that help define the starting point for the optimization of the model.<\/p>\n

      Weight Initialization Techniques:<\/h1>\n

      Initializing weights to zero<\/h3>\n

      If we initialize all our weights to zero, our Neural Network will act as a linear model because all the layers are learning the same thing.<\/p>\n

      Therefore, the important thing to note with initializing your weights for Neural Networks is to not initialize all the weight to zero.<\/p>\n

      Initializing weights randomly<\/h3>\n

      Using random initialization defeats the problem caused by initializing weights to zero, as it prevents the neurons from learning the exact same features of their inputs. Our aim is for each neuron to learn the different functionalities of its input.<\/p>\n

      However, using this technique can also lead to vanishing or exploding gradients, due to incorrect activation functions not being used. It currently works effectively with the RELU activation function.<\/p>\n

      Initializing weights using Heuristic<\/h3>\n

      This is considered the best technique to initialize weights for Neural Networks.<\/p>\n

      Heuristics serve as good starting points for weight initialization as they reduce the chances of vanishing or exploding gradients from occurring. This is due to the fact that the weights are neither too bigger than 1, nor less than 1. They also help in the avoidance of slow convergence.<\/p>\n

      The most common heuristics used are:<\/p>\n

      1. He-et-al Initialization.<\/h2>\n

      When using the RELU activation function, this heuristic is used by multiplying the randomly generated values of W by:<\/p>\n

      \n
      \n
      \"\"<\/figure>
      <\/picture>Source<\/a><\/div>\n<\/div>\n<\/figure>\n

      2. Xavier initialization<\/h2>\n

      When using the Tanh activation function, this heuristic is used by multiplying the randomly generated values of W by:<\/p>\n

      \n
      \n
      \"\"<\/figure>
      <\/picture>Source<\/a><\/div>\n<\/div>\n<\/figure>\n

      Conclusion<\/h1>\n

      The rise in Machine Learning and the implementation and application of models in our day-to-day lives raises concern about how to efficiently these models work.<\/p>\n

      A lot of time and money is spent on models that don\u2019t accurately produce the outputs we expected. Therefore, it\u2019s important to understand Cost Function and how Stochastic Gradient Descent minimizes the Cost Function by changing the weights of the network. This will benefit the overall process of building your model, producing accurate outputs to make the right conclusions and decisions, at a reduced cost.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

      Building a Neural Network model can be very complicated and tuning the Neural Network model can make it even more confusing. One of the most common problems when working with Deep Neural Networks is the Vanishing and\/or Exploding Gradient Descent. In order to prevent this from happening, one solution is initializing weights. Initializing weights in […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[139],"class_list":["post-4584","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nVanishing\/Exploding Gradients in Deep Neural Networks - Comet<\/title>\n<meta name=\"description\" content=\"Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Vanishing\/Exploding Gradients in Deep Neural Networks\" \/>\n<meta property=\"og:description\" content=\"Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-11T01:49:59+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:16:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp\" \/>\n<meta name=\"author\" content=\"Nisha Arya Ahmed\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nisha Arya Ahmed\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Vanishing\/Exploding Gradients in Deep Neural Networks - Comet","description":"Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/","og_locale":"en_US","og_type":"article","og_title":"Vanishing\/Exploding Gradients in Deep Neural Networks","og_description":"Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback.","og_url":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2022-11-11T01:49:59+00:00","article_modified_time":"2025-04-24T17:16:34+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp","type":"","width":"","height":""}],"author":"Nisha Arya Ahmed","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Nisha Arya Ahmed","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Vanishing\/Exploding Gradients in Deep Neural Networks","datePublished":"2022-11-11T01:49:59+00:00","dateModified":"2025-04-24T17:16:34+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/"},"wordCount":1282,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/","url":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/","name":"Vanishing\/Exploding Gradients in Deep Neural Networks - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp","datePublished":"2022-11-11T01:49:59+00:00","dateModified":"2025-04-24T17:16:34+00:00","description":"Initializing weights in Neural Networks helps to prevent layer activation outputs from Vanishing or Exploding during forward feedback.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#primaryimage","url":"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp","contentUrl":"https:\/\/miro.medium.com\/max\/700\/0*FrY8ou2OPu0w0wGp"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/vanishing-exploding-gradients-in-deep-neural-networks\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Vanishing\/Exploding Gradients in Deep Neural Networks"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4584","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=4584"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4584\/revisions"}],"predecessor-version":[{"id":15651,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4584\/revisions\/15651"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=4584"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=4584"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=4584"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=4584"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}