{"id":4560,"date":"2022-11-10T17:53:10","date_gmt":"2022-11-11T01:53:10","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4560"},"modified":"2025-04-24T17:16:27","modified_gmt":"2025-04-24T17:16:27","slug":"weight-initialization-in-deep-neural-networks","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/weight-initialization-in-deep-neural-networks\/","title":{"rendered":"Weight Initialization In Deep Neural Networks"},"content":{"rendered":"\n
\n
<\/figure>

<\/p>\n

Photo by Graphic Node<\/a> on Unsplash<\/a><\/p>\n

Very deep neural networks can suffer from either vanishing or exploding gradients<\/a>. This is because the main operation used to compute the derivatives as we propagate through a neural network model is matrix multiplication \u2014 thus, a network of n<\/em> hidden layers will multiply n<\/em> derivatives together.<\/p>\n

When the derivatives are large, the gradient will increase exponentially as we propagate through the model until it eventually explodes. In contrast, when the derivatives are small, the gradient will decrease exponentially as we propagate through the model until it eventually vanishes. Both scenarios lead to the same end (in different ways): the network failing to learn well.<\/p>\n

A partial solution to this problem is to be more careful when randomly initializing the weights of the network. This procedure is called weight initialization<\/strong>. The end goal of weight initialization is to prevent gradients from exploding or vanishing, thus, permitting a slightly more effective optimization process. It doesn\u2019t entirely solve the vanishing\/exploding gradient problem, but it\u2019s an important design choice and helps a lot when building deep neural networks.<\/p>\n

Let\u2019s consider the make_circles<\/code> dataset from scikit learn:<\/p>\n

import<\/strong> numpy as <\/strong>np\nimport<\/strong> pandas as <\/strong>pd\nimport <\/strong>matplotlib.pyplot as <\/strong>plt\nfrom<\/strong> sklearn.datasets import<\/strong> make_circles\nfrom<\/strong> sklearn.model_selection import<\/strong> train_test_split<\/span>import <\/strong>tensorflow as <\/strong>tf\nfrom <\/strong>mlxtend.plotting import <\/strong>plot_decision_regions<\/span>import <\/strong>warnings\nwarnings.filterwarnings(\"ignore\")<\/span># load data<\/em><\/strong>\nX_train, y_train = make_circles(n_samples=10000<\/em>, noise=.05<\/em>)\nX_test, y_test = make_circles(n_samples=100<\/em>, noise=.05<\/em>)<\/span># visualize data<\/em><\/strong>\nplt.subplots(figsize=(8, 5<\/em>))\nplt.scatter(X_train[:, 0<\/em>], X_train[:, 1<\/em>], c=y_train, cmap=plt.cm.Spectral)<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
We are going to build a 4 layer neural network and test how it performs with different weight initializations. The full code for this notebook can be found via Github<\/a>.<\/p>\n
Zero Initialization<\/h1>\n
Zero initialization is exactly how it sounds: all weights are initialized as 0. In such scenarios, the neurons in each layer would learn exactly the same thing. Our network would fail to break symmetry as a consequence.<\/p>\n
We can take this a step further and say: \u201cwhenever a constant value is used to initialize a model’s weights, you can expect it to perform poorly.\u201d This is because the outputs of all hidden units in the model would have the same influence on the cost, thus, resulting in identical gradients.<\/p>\n
Here\u2019s the results of our model when we initialized weights as zero.<\/p>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
Visualizing cost per epoch and decision boundary for a model with weights initialized at zero.<\/figcaption><\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Tips, tricks, and innovation \u2014 all in your inbox each week. Subscribe to the Comet newsletter<\/a> for the latest industry news gathered by our team of experts.<\/p><\/blockquote>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Random Initialization<\/h1>\n
Assigning weights random values breaks the symmetry. It\u2019s better than zero but it\u2019s not without some issues. Weights must not be initialized too high or too low because:<\/p>\n