{"id":4597,"date":"2022-11-10T17:47:27","date_gmt":"2022-11-11T01:47:27","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=4597"},"modified":"2025-04-24T17:16:43","modified_gmt":"2025-04-24T17:16:43","slug":"how-to-train-your-deep-learning-models-faster","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/","title":{"rendered":"How To Train Your Deep Learning Models Faster"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

Photo by Marc-Olivier Jodoin<\/a> on Unsplash<\/a><\/p>\n\n\n\n

\n

\n<\/p>

Deep learning is a subset of machine learning that utilizes neural networks in \u201cdeep\u201d architectures, or multiple layers, to extract information from data. Early neural networks employed relatively simple (or \u201cshallow\u201d) architectures, but today\u2019s deep learning neural networks can be incredibly complex.<\/p>\n

Neural networks are designed to work much like human brains and are comprised of individual neurons that resemble the structure of a biological neuron. A neuron receives input from other neurons, performs some algorithmic processing on it, and generates an output (which may or may not be fed into yet another neuron as input). Neural networks are therefore a combination of a series of algorithms that ultimately recognize underlying relationships and <\/strong>patterns in the data. In short, a neural network is an interconnected arrangement of mathematical functions.<\/p>\n

In 1957, the first neural network, called a Perceptron, was developed by Frank Rosenblatt. It was similar to modern-day neural networks, except in that it only had one hidden layer, as well as configurable weights and biases. Now, many decades later, multi-layer neural networks are widely used for solving incredibly complex problems.<\/p>\n

Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.<\/p>\n

\n
\n
\"\"<\/figure>
Multilayer Neural Network \u2014 Image Created By Author Using <\/picture>Canva.com<\/a><\/div>\n<\/div>\n<\/figure>\n

\n<\/p>

Training a deep learning neural network can take days, or even weeks, or more! However, there are some methods that we can use to train models faster, and we\u2019ll discuss a few of them in this article.<\/p>\n

It can takes days, or even weeks, to train a DL neural network, but the use of transfer learning, optimizers, early stopping, and GPUs can speed up this process significantly.<\/p>\n

Selecting the right optimizer<\/h1>\n

Optimization algorithms are responsible for reducing the loss, and increasing the evaluation metric (often, accuracy), of your deep learning model by controlling the learning rate<\/strong>. Training time depends largely on how quickly your model can learn optimal weights for your network, so selecting the right optimization method can reduce training time exponentially.<\/p>\n

There are many optimizers to choose from, including Gradient Descent, Momentum, Mini Batch Gradient Descent, Nesterov Accelerated Gradient, AdaGrad, RMSProp, and more. Many of these optimizers are mathematically complex, and even many experts in the field don\u2019t know all the details of all the optimization algorithms. One pragmatic approach is to choose a versatile algorithm and use that for most problems. And for this, the Adam optimizer<\/strong> is an excellent go-to option. It performs exceedingly well on a wide range of applications and compares favorably to other stochastic optimization methods.<\/p>\n

The Adam optimizer adjusts the learning rate as it performs gradient descent<\/em> to ensure reasonable values throughout the weight optimization process. This flexibility allows Adam to increase the learning rate where appropriate (speeding up the training process and avoiding local minima) and decrease the learning rate when approaching the global minimum.<\/p>\n

# Initializing the Adam optimizer\nimport tensorflow as tf<\/span>adam = tf.keras.optimizers.Adam(\n    learning_rate=0.001,\n    beta_1=0.9,\n    beta_2=0.999,\n    epsilon=1e-07,\n    amsgrad=False,\n    name='adam',\n    **kwargs\n)<\/span><\/pre>\n
learning_rate<\/strong>: floating-point value or a tf.keras.optimizers.schedules.LearningRateSchedule<\/a><\/code>scheduled value; defaults to 0.001<\/code>.
\nbeta_1<\/strong>: The exponential decay rate for the 1st-moment estimate; defaults to 0.9<\/code>.
\nbeta_2<\/strong>: The exponential decay rate for the 2nd-moment estimate; defaults to 0.999<\/code>.
\nname: <\/strong>Optional name for the operations created when applying gradients. Defaults to Adam<\/code>.
\nepsilon<\/strong>: A small constant for numerical stability; defaults to 1e-7<\/code>.
\namsgrad<\/strong>: Boolean; whether to apply the AMSGrad variant of this algorithm.<\/p>\n
<\/div>\n
You can learn about different optimizers in more detail here<\/a>.<\/p>\n
# Configuring the model for training\nimport tensorflow as tf\nimport keras<\/span>accuracy = tf.keras.metrics.Accuracy()\nadam = tf.keras.optimizers.Adam()<\/span>model.compile(\n  loss='binary_crossentropy',\n  optimizer= adam,\n  metrics=[accuracy]\n)<\/span><\/pre>\n
Making Use of Transfer Learning<\/h1>\n
The process of transferring the knowledge gained by one model to another model is called transfer learning.<\/p>\n
In transfer learning, we make use of a pre-trained model with pre-trained weights and freeze all the layers of that model except the final output layer. Then we replace the original output layer with our custom final layer, which includes the number of output classes we need our new model to identify.<\/p>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
<\/div>\n
\n<\/p>
There are many pre-trained models available in the Keras library<\/a>. These models have been trained extensively for long periods of time on massive labeled datasets. By freezing all but the output layer of one of these models, we are able to utilize the patterns learned in these massive training sessions, and apply them to our own use cases. And to use one is as simple as one line of code, which downloads the pre-trained model from Keras.<\/p>\n
<\/div>\n
from keras.layers import Dense, Flatten\nfrom keras.models import Model\nfrom keras.models import Sequential\nfrom keras.applications.vgg16 import VGG16<\/span># Image Size<\/strong>\nIMAGE_SIZE = [224, 224]<\/span># Pretrained Model\n<\/strong>vgg = VGG16(input_shape= IMAGE_SIZE + [3], weights ='imagenet',\n                include_top= False)<\/span># Freeze layers\n<\/strong>for layer in vgg.layers:\n  layer.trainable = False<\/span># Add custom output layer\n<\/strong>x = Flatten()(vgg.output)\nprediction = Dense(3<\/strong>, activation= 'softmax')(x)\nmodel_1 = Model(inputs= vgg.input, outputs=prediction)<\/span># Generate Model Summary\nmodel_1<\/strong>.summary()<\/span><\/pre>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
\n<\/p>
You can see that we have attached our new output layer classifying only three classes instead of 1,000.<\/p>\n
Transfer learning can save a lot of time and resources because we don\u2019t have to train the whole model from scratch.<\/p>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Big teams rely on big ideas. Learn how experts at Uber, WorkFusion, and The RealReal use Comet<\/a> to scale out their ML models and ensure visibility and collaboration company-wide.<\/p><\/blockquote>\n<\/div>\n\n\n\n
<\/div>\n\n\n\n
\n
Early Stopping<\/h1>\n
One epoch is one complete pass of training data through the neural network. During each epoch, each neuron has the opportunity to update its weights, so the more epochs you choose, the longer your training will be. Additionally, choosing too many epochs can lead to overfitting. On the other hand, choosing too few epochs can cause underfitting.<\/p>\n
Early stopping is a form of regularization that stops the training process once model performance stops improving on the validation set. It allows significantly decreases the likelihood of overfitting the model.<\/p>\n
Keras has a callback function designed to stop training early, once it has detected that the model is no longer making significant improvements<\/p>\n
from tensorflow.keras.callbacks import EarlyStopping<\/span>early_stopping = EarlyStopping(\n    monitor='val_loss',\n    min_delta=0,\n    patience=0,\n    verbose=0,\n    mode='auto',\n    baseline=None,\n    restore_best_weights=False\n)<\/span><\/pre>\n
We won\u2019t be utilizing all of these parameters for our example, but let\u2019s take a look at two we will use:<\/p>\n
monitor: <\/strong>metric to be used as a measure for terminating the training.
\npatience<\/strong>: number of epochs with no improvement after which training gets terminated.<\/p>\n
In the example below, we have set out model to stop training when the loss stops improving for two consecutive epochs.<\/p>\n
from tensorflow.keras.callbacks import EarlyStopping<\/span>early_stopping = EarlyStopping(monitor='loss', patience=2)<\/span>history = model.fit(\n    X_train,\n    y_train,\n    epochs= 100,\n    validation_split= 0.20,\n    batch_size= 50,\n    verbose= \"auto\",\n    callbacks= [early_stopping]\n)<\/span><\/pre>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
\n<\/p>
Early stopping will stop the neural network when it stops improving for the specified number of epochs, thus reducing the training time taken by the network.<\/p>\n
GPUs For Training<\/h1>\n
Ultimately, no matter how much you optimize your deep learning model, if you are training it on a CPU it will take you exponentially longer that training it on a GPU. GPUs can improve the overall training process by performing multiple computations simultaneously.<\/p>\n
There are two options through which you can access GPU power for your model. First, the cheapest option is to make use of cloud GPUs provided by big tech companies.<\/p>\n
Google Colab<\/strong><\/a> is a leading GPU provider, that provides you the opportunity to easily upload your Python notebook and train a model on their virtual machines. Once all files are uploaded, you can even walk away from your computer and track the training process on your mobile phone or tablet.<\/p>\n
Kaggle<\/strong> is another GPU provider that provides 30 hours per week of free GPU time.<\/p>\n
If you\u2019re willing to spend some big bucks on training your model, there are other platforms as well that will provide you will even more access hours to their cloud GPUs. One example is Good Cloud GPU ($300 for 850 hours).<\/p>\n
Another expensive option is to build a high-end computer system that has at least 16GB of ram and around 8 GB of high-end graphics like Geforce RTX from Nvidia. A system like this will take a serious bite out of your pocket, though. If you are ready to invest some money, the CUDA toolkit, is definitely worth considering.<\/p>\n
CUDA will create a path between your computer hardware and deep learning model. The NVIDIA CUDA Toolkit provides a development environment for creating high-performance GPU accelerated applications like computer games, deep learning models, etc. The NVIDIA CUDA Deep Neural Network library (cuDNN) is specially designed for training and building deep neural networks. You can read about the complete step-by-step installation process here<\/em><\/a>.<\/p>\n
GPUs can drastically increase the training speed of your deep learning models, but access to GPUs is limited, unless you have deep pockets.<\/p>\n
Conclusion<\/h2>\n
Although training a neural network takes lots of time, with the help of optimizers, transfer learning, and access to advanced hardware, we can reduce the training time significantly, while still creating a top-notch model.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Photo by Marc-Olivier Jodoin on Unsplash Deep learning is a subset of machine learning that utilizes neural networks in \u201cdeep\u201d architectures, or multiple layers, to extract information from data. Early neural networks employed relatively simple (or \u201cshallow\u201d) architectures, but today\u2019s deep learning neural networks can be incredibly complex. Neural networks are designed to work much like human […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6,7],"tags":[],"coauthors":[140],"class_list":["post-4597","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-tutorials"],"yoast_head":"\nHow To Train Your Deep Learning Models Faster - Comet<\/title>\n<meta name=\"description\" content=\"Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How To Train Your Deep Learning Models Faster\" \/>\n<meta property=\"og:description\" content=\"Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-11T01:47:27+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:16:43+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO\" \/>\n<meta name=\"author\" content=\"Abhay Parashar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Abhay Parashar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How To Train Your Deep Learning Models Faster - Comet","description":"Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/","og_locale":"en_US","og_type":"article","og_title":"How To Train Your Deep Learning Models Faster","og_description":"Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.","og_url":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2022-11-11T01:47:27+00:00","article_modified_time":"2025-04-24T17:16:43+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO","type":"","width":"","height":""}],"author":"Abhay Parashar","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Abhay Parashar","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"How To Train Your Deep Learning Models Faster","datePublished":"2022-11-11T01:47:27+00:00","dateModified":"2025-04-24T17:16:43+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/"},"wordCount":1391,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO","articleSection":["Machine Learning","Tutorials"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/","url":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/","name":"How To Train Your Deep Learning Models Faster - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO","datePublished":"2022-11-11T01:47:27+00:00","dateModified":"2025-04-24T17:16:43+00:00","description":"Training a multilayer neural network entails much more than just creating connections between neurons, however. It can be a prohibitively lengthy iterative process, and often requires additional computational power that many do not have access to on their home computers.","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#primaryimage","url":"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO","contentUrl":"https:\/\/miro.medium.com\/max\/700\/0*RPTgooN5_raho8DO"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/how-to-train-your-deep-learning-models-faster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"How To Train Your Deep Learning Models Faster"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4597","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=4597"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4597\/revisions"}],"predecessor-version":[{"id":15657,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/4597\/revisions\/15657"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=4597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=4597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=4597"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=4597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}