{"id":4437,"date":"2022-10-28T09:08:36","date_gmt":"2022-10-28T17:08:36","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4437"},"modified":"2025-04-24T17:16:50","modified_gmt":"2025-04-24T17:16:50","slug":"activation-functions-in-neural-networks","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/","title":{"rendered":"Activation Functions In Neural Networks"},"content":{"rendered":"\n
\n
\n
\n
\"\"<\/figure>
Photo by <\/picture>John Smit<\/a> on <\/picture>Unsplash<\/a><\/div>\n<\/div>\n<\/figure>\n

An activation function plays an important role in a neural network. It\u2019s a function used in artificial neurons to non-linearly transform inputs that come from the previous cell and provide an output. Failing to apply an activation function would mean the neurons would resemble linear regression<\/a>. Thus, activation functions are required to introduce non-linearity into neural networks so they are capable of learning the complex underlying patterns that exist within data.<\/p>\n

In this article, I am going to explore various activation functions used when implementing neural networks.<\/p>\n

Note<\/strong>: See the full code in GitHub<\/a>.<\/p><\/blockquote>\n

Sigmoid Function<\/h3>\n
\n
\"\"<\/figure>
Mathematical expression for sigmoid function<\/picture><\/div>\n<\/figure>\n

The sigmoid function is a popular, bounded, differentiable, monotonic activation function used in neural networks. This means its values are scaled between 0 and 1 (bounded), the slope of the curve can be found at any two points (differentiable), and it\u2019s neither entirely increasing nor decreasing (monotonic). It has a characteristic \u201cS\u201d shaped curve which may be seen in the image below:<\/p>\n

# Sigmoid function in Python<\/em><\/strong>\nimport<\/strong> matplotlib.pyplot as<\/strong> plt\nimport <\/strong>numpy as <\/strong>np<\/span>x = np.linspace(-5<\/em>, 5<\/em>, 50<\/em>)\nz = 1<\/em>\/(1<\/em> + np.exp(-x<\/em>))<\/span>plt.subplots(figsize=(8<\/em>, 5<\/em>))\nplt.plot(x<\/em>, z<\/em>)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of the sigmoid function<\/picture><\/div>\n<\/figure>\n
This activation function is typically used for binary classification, however, it is not without fault. The sigmoid function suffers from the vanishing gradient problem<\/a>: a scenario during backpropagation in which the gradient decreases exponentially until it becomes extremely close to 0. As a result, the weights are not updated sufficiently which leads to extremely slow convergence \u2014 once the gradient reaches 0, learning stops.<\/p>\n
Softmax Function<\/h3>\n
The softmax function is a generalization of the sigmoid function to multiple dimensions. Thus, it\u2019s typically used as the last activation function in a neural network (the output layer) to predict multinomial probability distributions. In other words, we use it for classification problems in which class membership is required on more than two labels.<\/p>\n
\n
\"\"<\/figure>
Mathematical expression for softmax function<\/picture><\/div>\n<\/figure>\n
<\/div>\n
Hyperbolic Tangent (tanh) Function<\/h3>\n
The hyperbolic tangent function (tanh) is similar to the sigmoid function in a sense: they share the \u201cS\u201d shaped curve characteristic. In contrast, the hyperbolic tangent function transforms the values between -1 and 1. This means inputs that are small (more negative) will be mapped closer to -1 (strongly negative) and 0 inputs will be mapped near 0.<\/p>\n
\n
\"\"<\/figure>
Mathematical expression for tanh function<\/picture><\/div>
<\/figcaption><\/figure>\n
# tanh function in Python<\/em><\/strong>\nimport<\/strong> matplotlib.pyplot as<\/strong> plt\nimport <\/strong>numpy as <\/strong>np<\/span>x = np.linspace(-5<\/em>, 5<\/em>, 50<\/em>)\nz = np.tanh(x<\/em>)<\/span>plt.subplots(figsize=(8<\/em>, 5<\/em>))\nplt.plot(x<\/em>, z<\/em>)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of the tanh function<\/picture><\/div>\n<\/figure>\n
Ian Goodfellow’s book, Deep Learning (P.195)<\/em>, states \u201c\u2026the hyperbolic tangent activation function typically performs better than the logistic sigmoid.\u201d However, it suffers a similar fate to the sigmoid function, the vanishing gradient<\/a> problem. It\u2019s also computationally expensive due to its exponential operation.<\/p>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
How does the team at Uber manage to keep their data organized and their team united? Comet\u2019s experiment tracking. Learn more from Uber\u2019s Olcay Cirit<\/a>.<\/p><\/blockquote>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Rectified Linear Unit (ReLU) Function<\/h3>\n
We tend to avoid using sigmoid and tanh functions when building neural networks with many layers due to the vanishing gradient problem<\/a>. One solution could be to use the ReLU function which overcomes this problem and has become the default activation function for various neural networks.<\/p>\n
The ReLU activation function returns 0 if the input is 0 or less and returns the value provided as input directly if it’s greater than 0. Thus, the values range from 0 to infinity:<\/p>\n
\n
\"\"<\/figure>
Mathematical expression for ReLU function<\/picture><\/div>\n<\/figure>\n
In Python, it looks as follows\u2026<\/p>\n
# ReLU in Python<\/em><\/strong>\nimport <\/strong>matplotlib.pyplot as <\/strong>plt\nimport<\/strong> numpy as <\/strong>np<\/span>x = np.linspace(-5<\/em>, 5<\/em>, 50<\/em>)\nz = [max<\/strong>(0<\/em>, i) for i in <\/strong>x]<\/span>plt.subplots(figsize=(8<\/em>, 5<\/em>))\nplt.plot(x<\/em>, z<\/em>)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of ReLU function<\/picture><\/div>\n<\/figure>\n
Since the ReLU function looks and behaves mostly like a linear function, the neural network is much easier to optimize. It\u2019s also very easy to implement.<\/p>\n
Where the ReLU activation function suffers is when there are many negative values \u2014 they will all be outputted as 0. When this occurs, learning will be severely impacted and our model would be prohibited from properly learning complex patterns in the data. This is known as the dying ReLU problem. One solution to the dying ReLU problem is using Leaky ReLU.<\/p>\n
Leaky ReLU Function<\/h3>\n
Leaky ReLU is an improvement on ReLU that attempts to combat the dying ReLU problem. Instead of defining the ReLU function as 0 for all negative values of x, it is defined as an extremely small linear component of x.<\/p>\n
\n
\n
\"\"<\/figure>
Mathematical expression for leaky ReLU function<\/picture><\/div>\n<\/div>\n<\/figure>\n
The modification to the ReLU function alters the gradient for values equal to or less than 0 to be values that are non-zero. As a result, extremely small inputs would no longer encounter dead neurons.<\/p>\n
Let\u2019s see this in Python:<\/p>\n
# leaky ReLU in Python<\/em><\/strong>\nimport <\/strong>matplotlib.pyplot as <\/strong>plt\nimport<\/strong> numpy as <\/strong>np<\/span>x = np.linspace(-5<\/em>, 5<\/em>, 50<\/em>)\nz = [max<\/strong>((0.3<\/em>*0)<\/em>, i) for i in <\/strong>x]<\/span>plt.subplots(figsize=(8<\/em>, 5<\/em>))\nplt.plot(x<\/em>, z<\/em>)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of the Leaky ReLU function<\/picture><\/div>\n
<\/div>\n<\/figure>\n
Note<\/strong>: We typically set the \u03b1 value to 0.01. Rarely ever is it set to 1 (or close to it) since that would make Leaky ReLU a linear function.<\/p><\/blockquote>\n
Exponential Linear Unit (ELU) Function<\/h3>\n
The Exponential Linear Unit (ELU) is an activation function for neural networks. In contrast to ReLUs, ELUs have negative values which allow them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. [Credit: PaperWithCode<\/a>]<\/p>\n
\n
\"\"<\/figure>
Mathematical expression for ELU function<\/picture><\/div>\n<\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
# ELU function in Python\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.activations import elu<\/span>x = np.linspace(-5, 5, 50)\nz = elu(x, alpha=1)<\/span>plt.subplots(figsize=(8, 5))\nplt.plot(x, z)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of the ELU function<\/picture><\/div>\n<\/figure>\n
<\/div>\n
Swish Function<\/h3>\n
The swish function was announced as an alternative to ReLU by Google in 2017. It tends to perform better than ReLU in deeper networks, across a number of challenging datasets. This comes about after the authors showed that simply substituting ReLU activations with Swish functions improved the classification accuracy of ImageNet.<\/p>\n
\u201cQfter performing analysis on ImageNet data, researchers from Google alleged that using the function as an activation function in artificial neural networks improves the performance, compared to ReLU and sigmoid functions. It is believed that one reason for the improvement is that the swish function helps alleviate the vanishing gradient problem during backpropagation.\u201d<\/em>
\n\u2014 [Source<\/strong>: Wikipedia<\/a>]<\/p>\n
One drawback of the swish function is that it\u2019s computationally expensive in comparison to ReLU and its variants.<\/p>\n
\n
\"\"<\/figure>
Mathematical expression for the swish function<\/picture><\/div>\n<\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
# Swish function in Python<\/em><\/strong>\nimport<\/strong> matplotlib.pyplot as<\/strong> plt\nimport <\/strong>numpy as <\/strong>np<\/span>x = np.linspace(-5<\/em>, 5<\/em>, 50<\/em>)\nz = x * (1<\/em>\/(1<\/em> + np.exp(-x<\/em>)))<\/span>plt.subplots(figsize=(8<\/em>, 5<\/em>))\nplt.plot(x<\/em>, z<\/em>)\nplt.grid()<\/span>plt.show()<\/span><\/pre>\n
\n
\"\"<\/figure>
Visual representation of the Swish function<\/picture><\/div>\n<\/figure>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Many activation functions exist to help neural networks learn complex patterns in data. It\u2019s common practice to use the same activation through all the hidden layers. For some activation functions, like the softmax activation function, it\u2019s rare to see them used in hidden layers \u2014 we typically use the softmax function in the output layer when there are two or more labels.<\/p>\n
The activation function you decide to use for your model may significantly impact the performance of the neural network. How to Choose an Activation Function for Deep Learning<\/em><\/a> is a good article to learn more about choosing activation functions when building neural networks.<\/p>\n
Thanks for reading.<\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Photo by John Smit on Unsplash An activation function plays an important role in a neural network. It\u2019s a function used in artificial neurons to non-linearly transform inputs that come from the previous cell and provide an output. Failing to apply an activation function would mean the neurons would resemble linear regression. Thus, activation functions are required to […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[138],"class_list":["post-4437","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nActivation Functions In Neural Networks - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Activation Functions In Neural Networks\" \/>\n<meta property=\"og:description\" content=\"Photo by John Smit on Unsplash An activation function plays an important role in a neural network. It\u2019s a function used in artificial neurons to non-linearly transform inputs that come from the previous cell and provide an output. Failing to apply an activation function would mean the neurons would resemble linear regression. Thus, activation functions are required to […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-28T17:08:36+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:16:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th\" \/>\n<meta name=\"author\" content=\"Kurtis Pykes\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kurtis Pykes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Activation Functions In Neural Networks - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/","og_locale":"en_US","og_type":"article","og_title":"Activation Functions In Neural Networks","og_description":"Photo by John Smit on Unsplash An activation function plays an important role in a neural network. It\u2019s a function used in artificial neurons to non-linearly transform inputs that come from the previous cell and provide an output. Failing to apply an activation function would mean the neurons would resemble linear regression. Thus, activation functions are required to […]","og_url":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2022-10-28T17:08:36+00:00","article_modified_time":"2025-04-24T17:16:50+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th","type":"","width":"","height":""}],"author":"Kurtis Pykes","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Kurtis Pykes","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"Activation Functions In Neural Networks","datePublished":"2022-10-28T17:08:36+00:00","dateModified":"2025-04-24T17:16:50+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/"},"wordCount":1093,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/","url":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/","name":"Activation Functions In Neural Networks - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th","datePublished":"2022-10-28T17:08:36+00:00","dateModified":"2025-04-24T17:16:50+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#primaryimage","url":"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th","contentUrl":"https:\/\/miro.medium.com\/max\/700\/0*d31lOUMVCqEND9th"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/activation-functions-in-neural-networks\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"Activation Functions In Neural Networks"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4437","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=4437"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4437\/revisions"}],"predecessor-version":[{"id":15663,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4437\/revisions\/15663"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=4437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=4437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=4437"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=4437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}