The weight initialization technique you choose for your neural network can determine how quickly the network converges or whether it converges at all. Although the initial values of these weights are just one parameter among many to tune, they are incredibly important. Their distribution affects the gradients and, therefore, the effectiveness of training.<\/p>\n\n\n\n
In neural networks, weights represent the strength of connections between units in adjacent network layers. The linear transformation of these weights and the values in the previous layer passes through a non-linear activation function to produce the values of the next layer. This process happens layer to layer during forward propagation; through back propagation, the optimum values of these weights can be found out so as to produce accurate outputs given an input.<\/p>\n\n\n\n
In this article, we\u2019ll cover:<\/strong><\/p>\n\n\n\n We will not be covering mathematical derivations of these initialization approaches. If you are interested in this type of material, we do link to several great resources within and at the end of the article (in the \u2018Further Reading\u2019 section)<\/em> .<\/p>\n\n\n\n Improperly initialized weights can negatively affect the training process by contributing to the\u00a0vanishing<\/strong>\u00a0or\u00a0exploding gradient problem.\u00a0<\/strong>With the vanishing gradient problem, the weight update is minor and results in slower convergence \u2014 this makes the optimization of the loss function slow and in a worst case scenario, may stop the network from converging altogether. Conversely, initializing with weights that are too large may result in exploding gradient values during forward propagation or back-propagation (see more\u00a0here<\/a>).<\/p>\n\n\n\n Deeplearning.ai<\/a>\u00a0recently published\u00a0an interactive post<\/a>\u00a0where you can choose different initialization methods and watch the network train. Here\u2019s an example:<\/p>\n\n\n\n You\u2019ll notice how setting an initialization method that\u2019s too small barely allows the network to learn (ie. reduce the cost function) while an initialization method that\u2019s too large causes divergence (check the decision boundary).<\/p>\n\n\n\n In 2012, AlexNet, the winner of that year\u2019s ImageNet Large Scale Visual Recognition Challenge (ILVSRC), popularized the weight initialization approach of \u201cinitialization with Gaussian (normal) noise with mean equal to zero and standard deviation set to 0.01 with bias equal to one for some layers\u201d (see\u00a0Krizhevsky et al. 2012<\/a>).<\/p>\n\n\n\n However, this normal random initialization approach does not work for training very deep networks, especially those that use the ReLU (rectified linear unit) activation function, because of the vanishing and exploding gradient problem referenced earlier.<\/p>\n\n\n\n To address these issues, Xavier and Bengio (2010) proposed the \u201cXavier\u201d initialization which considers the size of the network (number of input and output units) while initializing weights. This approach ensures that the weights stay within a reasonable range of values by making them inversely proportional to the square root of the number of units in the previous layer (referred to as\u00a0fan-in<\/em>). See the diagram below on how to find the fan-in and fan-out for a given unit:<\/p>\n\n\n\n The choice of activation function ends up playing an important role in determining how effective the initialization method is. Activation functions are differentiable and introduce non-linear properties (i.e. curvature) into our neural networks that are crucial for solving the complex tasks that machine learning and deep learning are designed to tackle.<\/p>\n\n\n\n The activation function is the non linear transformation that we do over the input signal. This transformed output is then sent to the next layer of units as input. Some examples of these non-linear activation functions are:<\/p>\n\n\n\n Rectified Linear unit (ReLU) (and leaky ReLU) are commonly used since they are relatively robust to the vanishing\/exploding gradient issues. For activation functions like ReLU, Kaiming He et al. (2015) introduced a more robust weight initialization method that accounts for the fact that it\u2019s not symmetric (see performance differenced cited in the He et al. paper below). Both methods use a similar theoretical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly considering the type of the distribution.<\/p>\n\n\n\n Figure from\u00a0the He et al. (2015) paper<\/a>\u00a0showing how their refined initialization strategy (red) reduces the error rate much faster than the Xavier method (blue) for (P)ReLUs and accounts for the fact that it is not symmetric.<\/p>\n\n\n\n For an accessible proof of the Xavier and He initialization methods, see Pierre Ouannes\u2019 excellent post\u00a0\u2018How to initialize deep neural networks? Xavier and Kaiming initialization\u2019<\/a>.<\/p>\n\n\n\n It\u2019s important to note that weight initialization is\u00a0still an active area of research.\u00a0<\/em>Several interesting research projects have popped up including\u00a0data-dependent initializations<\/a>,\u00a0sparse weight matrices<\/a>, and\u00a0random orthogonal matrix initializations<\/a>.<\/p>\n\n\n\n One of the most interesting developments in this area is MIT\u2019s\u00a0Lottery Ticket Hypothesis<\/a>, which details how these large neural nets contain smaller \u201csubnetworks\u201d that are up to 10 times smaller than the full network. According to the research team, these subnetworks can learn just as well, making equally precise predictions sometimes faster than the full neural networks.<\/p>\n\n\n\n The authors of \u2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks<\/a>\u2019, Carbin and Frankle, tested their lottery ticket hypothesis and the existence of subnetworks by performing a process called\u00a0pruning<\/em>, which involves eliminating unneeded connections from trained networks based on their network prioritization or weight to fit them on low-power devices.<\/p>\n\n\n\n In fact,\u00a0TensorFlow<\/a>\u00a0recently announced a new weight pruning API:<\/p>\n\n\n\n Weight pruning means eliminating unnecessary values in the weight tensors. We are practically setting the neural network parameters\u2019 values to zero to remove what we estimate are unnecessary connections between the layers of a neural network. This is done during the training process to allow the neural network to adapt to the changes.<\/p>\n\n\n\n Read more about the weight pruning API\u00a0here<\/a>.<\/p>\n\n\n\n\n
Why is weight initialization important?<\/h3>\n\n\n\n
![\"\"](\"https:\/\/i0.wp.com\/blog.comet.ml\/wp-content\/uploads\/2019\/10\/deep-learning.gif?fit=769%2C376&ssl=1\")
<\/figure>\n<\/div>\n\n\n\nHow approaches for weight initialization have shifted?<\/h3>\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/fan-in-1-1024x585-1.jpg\")
<\/figure>\n<\/div>\n\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/epoch-error.jpg\")
<\/figure>\n<\/div>\n\n\n\nInteractive Demo<\/strong><\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2022\/06\/meme.jpg\")
<\/figure>\n<\/div>\n\n\n\n