{"id":7860,"date":"2023-10-06T14:35:37","date_gmt":"2023-10-06T22:35:37","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7860"},"modified":"2025-04-24T17:05:48","modified_gmt":"2025-04-24T17:05:48","slug":"resnet-how-one-paper-changed-deep-learning-forever","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/","title":{"rendered":"ResNet: How One Paper Changed Deep Learning Forever"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
The ResNet Paper<\/figcaption><\/figure>\n

In December of 2015 a paper was published that rocked the deep learning world.<\/p>\n

Widely regarded as one of the most influential papers in modern deep learning, it has been cited over 110,000 times. The name of this paper will go down in the annals of deep learning history: Deep Residual Learning for Image Recognition (aka, the ResNet paper).<\/strong><\/a><\/p>\n

This paper showed the deep learning community that it was possible to construct increasingly deeper network architectures<\/em><\/strong> that can either perform well or at least the same as the shallower networks.<\/p>\n

When AlexNet hit the scene in 2012, prevailing wisdom suggested adding more layers to neural networks would lead to better results. This was evidenced by breakthroughs coming from VGGNet, GoogleNet, and others.<\/p>\n

This set the deep learning community on a quest to go deeper.<\/p>\n

It turns out, though, that learning better networks is not as easy as stacking more and more layers. Researchers observed that the accuracy of deep networks would increase up to a saturation point before leveling off. Additionally, an unusual phenomenon was observed:<\/p>\n

Training error would increase as you add layers to an already deep network.<\/em><\/p>\n

This was primarily due to two problems<\/p>\n

1) Vanishing\/exploding gradients<\/p>\n

2) The degradation problem<\/p>\n

Vanishing\/exploding gradients<\/h2>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Google search<\/figcaption>\n<\/figure>\n

The vanishing\/exploding gradients problem is a by-product of the chain rule<\/a>. The chain rule multiplies error gradients for weights in the network. Multiplying lots of values that are less than one will result in smaller and smaller values. As those error gradients approach the earlier layers of a network, their value will tend to zero.<\/p>\n

This results in smaller and smaller updates to earlier layers (not much learning happening).<\/strong><\/p>\n

The inverse problem is the exploding gradient problem. This occurs due to large error gradients accumulating during training, resulting in massive updates to model weights in the earlier layers (since multiplying lots of values that are bigger than 1 will result in larger and larger values).<\/p>\n

The reason for this issue?<\/strong><\/p>\n

Parameters in earlier layers of the network are far away from the cost function, which is the source of the gradient that is propagated backward through the network. As the error is back propagated through an increasingly deep network, a larger number of parameters contribute to the error. The net result of both of these scenarios is that early layers in the network become more difficult to train.<\/p>\n

But there\u2019s another more curious problem\u2026<\/p><\/blockquote>\n

The degradation problem<\/h2>\n

Adding more and more layers to these deep models leads to higher training errors, ending in a degradation in the expressive power of your network.<\/p>\n

The degradation problem is unexpected, because its not caused by overfitting. Researchers were finding that as networks got deeper, the training loss would decrease but then shoot back up as more layers were added to the networks. Which is counterintuitive because you\u2019d expect your training error to decrease, converge, and plateau out as the number of layers in your network increases.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Author<\/figcaption>\n<\/figure>\n

Let\u2019s imagine that you had a shallow network that was performing well.<\/p>\n

If you take a \u201cshallow\u201d network and just stack on more layers to create a deeper network, the performance of the deeper network should be at least as good as the shallow network. Why? Because, in theory, deeper network could learn the shallow network. The shallow network is a subset of the deeper network. But this doesn\u2019t happen in practice!<\/p>\n

You could even set the new stacked layers to be identity layers, and still find your training error getting worse when you stack more layers on top of a shallower model. Deeper networks were leading to higher training error!<\/strong><\/p>\n

Both of these issues \u2014 the vanishing\/exploding gradients and degradation problems \u2014 threatened to halt progress of deep neural networks, until the ResNet paper came out.<\/p>\n

The ResNet paper introduced a novel solution to these two pesky problems that plagued the architects of deep neural networks.<\/p>\n

The skip connection<\/h2>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Image Source: Sachin Joglekar<\/figcaption>\n<\/figure>\n

Skip connections, which are housed in residual blocks, allow you to take the activation value from an earlier layer and pass it to a deeper layer in a network. Skip connections enable deep networks to learn the identity function. Learning the identity function allows a deeper layer to perform as well as an earlier layer, or at the very least it won\u2019t perform any worse. The result is smoother gradient flow, ensuring important features are preserved in the training process.<\/p>\n

The invention of the skip connection has given us the ability to build deeper and deeper networks while avoiding the problem of vanishing\/exploding gradients and degradation.<\/p>\n

Here\u2019s how it works\u2026<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Demonstrating information flow in a plain network. Source: EasyLearn.AI<\/figcaption>\n<\/figure>\n

Instead of the previous layers output being passed directly onto the next block, a copy of that output is made, then that copy is passed through a residual block. This residual block will process the copied output matrix, X \u2014 with a 3×3 convolution, followed by batch norm and ReLU to yield a matrix Z.<\/p>\n

Then, X and Z would be added together, element by element, to yield the output to the next layer\/block.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Demonstrating information flow in a residual network. Source: EasyLearn.AI<\/figcaption>\n<\/figure>\n

Doing this helps us ensure that any added layers in a neural network are useful for learning. Worst case scenario is that the residual block could output a bunch of zeros, which leaves the X+Z to end up still being X, since X+Z=X if Z is just the zero matrix.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

ResNet in action!<\/h1>\n

Now it\u2019s time to see ResNet in action.<\/p>\n

You could try to implement ResNet from scratch, train it on ImageNet, and try to find the optimal training parameters yourself\u2026 but why do that when you can use something out of the box? That\u2019s what SuperGradients<\/a> gives you: A pre-trained ResNet model with a robust set of training parameters that is ready for you to use with minimal configuration!<\/p>\n

You\u2019ll learn a bit about transfer learning and how to use the SuperGradients<\/a>training library on the MiniPlaces dataset to perform image classification. SuperGradients<\/a> is a PyTorch based training library that has pre-trained models for classification, detection, and segmentation tasks.<\/p>\n

You can follow along here, or open up this notebook on Google Colab to get hands on:<\/p>\n

Install dependencies<\/h2>\n

You\u2019ll need to install the following dependencies.<\/p>\n

!pip install super_gradients==3.0<\/span>.0<\/span> gwpy &> \/dev\/null\n!pip install matplotlib==3.1<\/span>.3<\/span> &> \/dev\/null\n!pip install torchinfo &> \/dev\/null<\/span><\/pre>\n
Import packages<\/h2>\n
Now you can import all the packages necessary for this tutorial.<\/p>\n
import<\/span> os\nimport<\/span> requests\nimport<\/span> zipfile\nimport<\/span> random\nimport<\/span> numpy as<\/span> np\nimport<\/span> torchvision\nimport<\/span> pprint\nimport<\/span> torch\nimport<\/span> pathlib\n\nfrom<\/span> matplotlib import<\/span> pyplot as<\/span> plt\nfrom<\/span> torchinfo import<\/span> summary\nfrom<\/span> pathlib import<\/span> Path, PurePath\nfrom<\/span> torchvision import<\/span> transforms\nfrom<\/span> torchvision import<\/span> datasets\nfrom<\/span> torch.utils.data import<\/span> DataLoader\nfrom<\/span> PIL import<\/span> Image\nfrom<\/span> typing import<\/span> List<\/span>, Tuple<\/span>\nimport<\/span> super_gradients\nfrom<\/span> super_gradients.training import<\/span> models\nfrom<\/span> super_gradients.training import<\/span> dataloaders\nfrom<\/span> super_gradients.training import<\/span> Trainer\nfrom<\/span> super_gradients.training import<\/span> training_hyperparams<\/span><\/pre>\n
Download data<\/h2>\n
You\u2019ll use use miniplaces<\/strong> dataset. You can learn more about the dataset here<\/a>.<\/p>\n
torchvision.datasets.utils.download_and_extract_archive('https:\/\/dissect.csail.mit.edu\/datasets\/miniplaces.zip'<\/span>,'datasets'<\/span>)<\/span><\/pre>\n
Configurations<\/h2>\n
The following class houses configuration values that will be helpful as you progress along the tutorial.<\/p>\n
class<\/span> config<\/span>:\n    EXPERIMENT_NAME = 'resnet_in_action'<\/span>\n    MODEL_NAME = 'resnet50'<\/span>\n    CHECKPOINT_DIR = 'checkpoints'<\/span>\n\n    # specify the paths to training and validation set <\/span>\n    TRAIN_DIR = 'datasets\/miniplaces\/train'<\/span>\n    VAL_DIR = 'datasets\/miniplaces\/val'<\/span>\n\n    # set the input height and width<\/span>\n    INPUT_HEIGHT = 224<\/span>\n    INPUT_WIDTH = 224<\/span>\n\n    # set the input height and width<\/span>\n    IMAGENET_MEAN = [0.485<\/span>, 0.456<\/span>, 0.406<\/span>]\n    IMAGENET_STD = [0.229<\/span>, 0.224<\/span>, 0.225<\/span>]\n\n    NUM_WORKERS = os.cpu_count()\n\n    DEVICE = 'cuda'<\/span> if<\/span> torch.cuda.is_available() else<\/span> 'cpu'<\/span>\n\n    FLIP_PROB = 0.25<\/span>\n    ROTATION_DEG = 15<\/span>\n    JITTER_PARAM = 0.25<\/span>\n    BATCH_SIZE = 64<\/span><\/span><\/pre>\n
Experiment setup<\/h2>\n
When working with SuperGradients first thing you have to do is initialize your trainer<\/strong>.<\/p>\n
The trainer<\/strong> is in charge of pretty much everything from training, evaluation, saving checkpoints, plotting etc. The experiment name<\/strong> argument is important as all checkpoints, logs and tensorboards will be saved in a directory with the same name.<\/p>\n
This directory will be created as a sub-directory of ckpt_root_dir<\/strong> as follow:<\/p>\n
ckpt_root_dir\n|\u2500\u2500\u2500 experiment_name_1\n\u2502       ckpt_best.pth                     # Model checkpoint on best epoch\n\u2502       ckpt_latest.pth                   # Model checkpoint on last epoch\n\u2502       average_model.pth                 # Model checkpoint averaged over epochs\n\u2502       events.out.tfevents.1659878383... # Tensorflow artifacts of a specific run\n\u2502       log_Aug07_11_52_48.txt            # Trainer logs of a specific run\n\u2514\u2500\u2500\u2500 experiment_name_2\n        ...<\/span><\/pre>\n
trainer = Trainer(experiment_name=config.EXPERIMENT_NAME, ckpt_root_dir=config.CHECKPOINT_DIR, device=config.DEVICE)<\/span><\/pre>\n
Create dataloaders<\/h2>\n
The SG trainer is compatible with PyTorch dataloaders, so you can use your own dataloaders.<\/p>\n
def<\/span> create_dataloaders<\/span>(\n    train_dir: str<\/span>,\n    val_dir: str<\/span>,\n    train_transform: transforms.Compose,\n    val_transform:  transforms.Compose,\n    batch_size: int<\/span>,\n    num_workers: int<\/span>=config.NUM_WORKERS\n<\/span>):\n  \"\"\"Creates training and validation DataLoaders.\n  Args:\n    train_dir: Path to training data.\n    val_dir: Path to validation data.\n    transform: Transformation pipeline.\n    batch_size: Number of samples per batch in each of the DataLoaders.\n    num_workers: An integer for number of workers per DataLoader.\n  Returns:\n    A tuple of (train_dataloader, val_dataloader, class_names).\n  \"\"\"<\/span>\n  # Use ImageFolder to create dataset<\/span>\n  train_data = datasets.ImageFolder(train_dir, transform=train_transform)\n  val_data = datasets.ImageFolder(val_dir, transform=val_transform)\n\n  print<\/span>(f\"[INFO] training dataset contains {len<\/span>(train_data)}<\/span> samples...\"<\/span>)\n  print<\/span>(f\"[INFO] validation dataset contains {len<\/span>(val_data)}<\/span> samples...\"<\/span>)\n\n  # Get class names<\/span>\n  class_names = train_data.classes\n  print<\/span>(f\"[INFO] dataset contains {len<\/span>(class_names)}<\/span> labels...\"<\/span>)\n\n  # Turn images into data loaders<\/span>\n  print<\/span>(\"[INFO] creating training and validation set dataloaders...\"<\/span>)\n  train_dataloader = DataLoader(\n      train_data,\n      batch_size=batch_size,\n      shuffle=True<\/span>,\n      num_workers=num_workers,\n      pin_memory=True<\/span>,\n  )\n  val_dataloader = DataLoader(\n      val_data,\n      batch_size=batch_size,\n      shuffle=False<\/span>,\n      num_workers=num_workers,\n      pin_memory=True<\/span>,\n  )\n\n  return<\/span> train_dataloader, val_dataloader, class_names<\/span><\/pre>\n
Transforms<\/h2>\n
This next code block will instantiate a transformation pipeline for both training and validation.<\/p>\n
# initialize our data augmentation functions<\/span>\nresize = transforms.Resize(size=(config.INPUT_HEIGHT,config.INPUT_WIDTH))\n\nhorizontal_flip = transforms.RandomHorizontalFlip(p=config.FLIP_PROB)\n\nvertical_flip = transforms.RandomVerticalFlip(p=config.FLIP_PROB)\n\nrotate = transforms.RandomRotation(degrees=config.ROTATION_DEG)\n\nnorm = transforms.Normalize(mean=config.IMAGENET_MEAN, std=config.IMAGENET_STD)\n\nmake_tensor = transforms.ToTensor()\n\n# initialize our training and validation set data augmentation pipeline<\/span>\ntrain_transforms = transforms.Compose([resize, horizontal_flip, vertical_flip, rotate, make_tensor, norm])\nval_transforms = transforms.Compose([resize, make_tensor, norm])<\/span><\/pre>\n
Instantiate dataloaders<\/h2>\n
Now you can instantiate your dataloaders.<\/p>\n
train_dataloader, valid_dataloader, class_names = create_dataloaders(train_dir=config.TRAIN_DIR,\n                                                                     val_dir=config.VAL_DIR,\n                                                                     train_transform=train_transforms,\n                                                                     val_transform=val_transforms,\n                                                                     batch_size=config.BATCH_SIZE)\n\nNUM_CLASSES = len<\/span>(class_names)<\/span><\/pre>\n
Some heuristics for transfer learning<\/h2>\n
1) Your dataset is small and similar to the dataset the model was pre-trained on.<\/strong><\/p>\n
When your images are similar its likely that low-level features (like edges) and high-level features (like shapes) will be similar.<\/p>\n
What to do: Freeze the weights up to last layer, replace the fully connected layer, and retrain. Why? Less data means you can overfit if you train the entire network.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
Comet Artifacts lets you track and reproduce complex multi-experiment scenarios, reuse data points, and easily iterate on datasets. Read this quick overview of Artifacts<\/a> to explore all that it can do.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n
2) Your dataset is large and similar to the dataset the model was pre-trained on.<\/strong><\/p>\n
What to do: Freeze the earlier layer weights. Then retain later weights with a new fully connected layer.<\/p>\n
3) Your dataset is small and different from the dataset the model was pre-trained on.<\/strong><\/p>\n
This is the most difficult situation to deal with. The pre-trained network is already finely-tuned at each layer. You don\u2019t want any of the high-level features and<\/em><\/strong> you can\u2019t<\/strong> afford to retrain because you run the risk of overfitting.<\/p>\n
What to do: Remove the fully connected layers and convolutional layers closer to the output. Retrain the convolutional layers closer to the input.<\/p>\n
4) Your dataset is large and different from the dataset the model was pre-trained on.<\/strong><\/p>\n
You should instantiate the pre-trained model weights to speed up training (lot\u2019s of the low-level convolutions will have similar weights).<\/p>\n
What to do: Retrain the entire network from scratch, making sure to replace the fully connected output layer.<\/p>\n
To train ResNet from scratch in SG all you need to do is omit the pretrained_weights=\"imagenet\"<\/code> argument from the models.get<\/code> method.<\/p>\n
For this example, we will go with option 2.<\/strong><\/p>\n
Instantiate your ResNet model<\/h2>\n
Using SuperGradients makes changing the classification head of your model simple. All you have to do is pass in the number of classes for your use case to the num_classes<\/code> argument and the classification head will automatically be changed for you.<\/p>\n
resnet50_imagenet_model = models.get(model_name=config.MODEL_NAME, num_classes=NUM_CLASSES, pretrained_weights=\"imagenet\"<\/span>)<\/span><\/pre>\n
This next block of code will freeze the early layers and batch norm layers of the instantiated model.<\/p>\n
for<\/span> param in<\/span> resnet50_imagenet_model.conv1.parameters():\n    param.requires_grad = False<\/span>\n\nfor<\/span> param in<\/span> resnet50_imagenet_model.bn1.parameters():\n    param.requires_grad = False<\/span>\n\nfor<\/span> param in<\/span> resnet50_imagenet_model.layer1.parameters():\n    param.requires_grad = False<\/span>\n\nfor<\/span> param in<\/span> resnet50_imagenet_model.layer2.parameters():\n    param.requires_grad = False<\/span><\/span><\/pre>\n
Training setup<\/h2>\n
The training parameters in SuperGradients were optimized per dataset and architecture with consideration for the type of training used(from scratch\\transfer learning).<\/p>\n
For more recommended training params you can have a look at our recipes here<\/a>.<\/p>\n
You won\u2019t need to tune hyperparamters in this example, but you are more than welcome to play around with some!<\/p>\n
training_params =  training_hyperparams.get(\"training_hyperparams\/imagenet_resnet50_train_params\"<\/span>)\n\n#overriding the number of epochs to train for<\/span>\ntraining_params[\"max_epochs\"<\/span>] = 3<\/span><\/span><\/pre>\n
Training and evaluation<\/h2>\n
The results of the training epochs are kept in your CKPT path that was defined. Let\u2019s go ahead and train the model.<\/p>\n
You\u2019ll notice a few metrics displayed for each epoch:<\/p>\n
    \n
  1. Accuracy<\/code><\/li>\n
  2. LabelSmoothingCrossEntropyLoss<\/code><\/li>\n
  3. Top5<\/code><\/li>\n<\/ol>\n
    Let\u2019s define the two that you may not be familiar with.<\/p>\n
    LabelSmoothingCrossEntropyLoss.<\/h2>\n
    In classification problems, sometimes our model learns predict the training examples extremely confidently. This is not good for generalization. Label smoothing is a regularization technique for classification problems that prevent a model from predicting the training examples too confidently. This is used to increase robustness for classification problems.<\/p>\n
    Top5<\/strong><\/h2>\n
    Top-5 accuracy means one of the model\u2019s top 5 highest probability predictions match the ground truth. If it does, you count it as a correct prediction.<\/p>\n
    To train your model all you have to do is run the following code:<\/p>\n
    trainer.train(model=resnet50_imagenet_model,\n              training_params=training_params,\n              train_loader=train_dataloader,\n              valid_loader=valid_dataloader)\n\n# Load the best model that we trained<\/span>\nbest_model = models.get(config.MODEL_NAME,\n                        num_classes=NUM_CLASSES,\n                        checkpoint_path=os.path.join(trainer.checkpoints_dir_path,\"ckpt_best.pth\"<\/span>))<\/span><\/pre>\n
    Let\u2019s see how well our model predicts on the validation data<\/h1>\n
    Now that the model is trained you can examine how well it predicts on validation data. This next block of code will predict on images from the validation set.<\/p>\n
    # 1. Take in a trained model, class names, image path, image size, a transform and target device<\/span>\ndef<\/span> pred_and_plot_image<\/span>(model: torch.nn.Module,\n                        image_path: str<\/span>,\n                        class_names: List<\/span>[str<\/span>],\n                        image_size: Tuple<\/span>[int<\/span>, int<\/span>] = (config.INPUT_HEIGHT, config.INPUT_WIDTH),\n                        transform: torchvision.transforms = None<\/span>,\n                        device: torch.device=config.DEVICE<\/span>):\n\n\n    # 2. Open image<\/span>\n    if<\/span> isinstance<\/span>(image_path, pathlib.PosixPath):\n      img = Image.open<\/span>(image_path)\n    else<\/span>:\n      img = Image.open<\/span>(requests.get(image_path, stream=True<\/span>).raw)\n\n    # 3. Create transformation for image (if one doesn't exist)<\/span>\n    if<\/span> transform is<\/span> not<\/span> None<\/span>:\n        image_transform = transform\n    else<\/span>:\n        image_transform = transforms.Compose([\n            transforms.Resize(image_size),\n            transforms.ToTensor(),\n            transforms.Normalize(mean=config.IMAGENET_MEAN,\n                                 std=config.IMAGENET_STD),\n        ])\n\n    ### Predict on image ### <\/span>\n\n    # 4. Make sure the model is on the target device<\/span>\n    model.to(device)\n\n    # 5. Turn on model evaluation mode and inference mode<\/span>\n    model.eval<\/span>()\n    with<\/span> torch.inference_mode():\n      # 6. Transform and add an extra dimension to image (model requires samples in [batch_size, color_channels, height, width])<\/span>\n      transformed_image = image_transform(img).unsqueeze(dim=0<\/span>)\n\n      # 7. Make a prediction on image with an extra dimension and send it to the target device<\/span>\n      target_image_pred = model(transformed_image.to(device))\n\n    # 8. Convert logits -> prediction probabilities (using torch.softmax() for multi-class classification)<\/span>\n    target_image_pred_probs = torch.softmax(target_image_pred, dim=1<\/span>)\n\n    # 9. Convert prediction probabilities -> prediction labels<\/span>\n    target_image_pred_label = torch.argmax(target_image_pred_probs, dim=1<\/span>)\n\n    #actual label<\/span>\n    ground_truth = PurePath(image_path).parent.name\n\n    # 10. Plot image with predicted label and probability <\/span>\n    plt.figure()\n    plt.imshow(img)\n    if<\/span> isinstance<\/span>(image_path, pathlib.PosixPath):\n      plt.title(f\"Ground Truth: {ground_truth}<\/span> | Pred: {class_names[target_image_pred_label]}<\/span> | Prob: {target_image_pred_probs.max<\/span>():.3<\/span>f}<\/span>\"<\/span>)\n    else<\/span>:\n      plt.title(f\"Pred: {class_names[target_image_pred_label]}<\/span> | Prob: {target_image_pred_probs.max<\/span>():.3<\/span>f}<\/span>\"<\/span>)\n    plt.axis(False<\/span>);\n\n# Get a random list of image paths from test set<\/span>\nnum_images_to_plot = 30<\/span>\ntest_image_path_list = list<\/span>(Path(config.VAL_DIR).glob(\"*\/*.jpg\"<\/span>)) # get list all image paths from test data <\/span>\ntest_image_path_sample = random.sample(population=test_image_path_list, # go through all of the test image paths<\/span>\n                                       k=num_images_to_plot) # randomly select 'k' image paths to pred and plot<\/span>\n\n# Make predictions on and plot the images<\/span>\nfor<\/span> image_path in<\/span> test_image_path_sample:\n    pred_and_plot_image(model=best_model,\n                        image_path=image_path,\n                        class_names=class_names,\n                        image_size=(config.INPUT_HEIGHT, config.INPUT_WIDTH))<\/span><\/pre>\n
    Now that the training is complete you can use the trained model to predict on an unseen image!<\/p>\n
    Let\u2019s see what place the model thinks this image is:<\/p>\n
    \n
    \"\"<\/figure>
    <\/picture><\/div>\n<\/figure>\n
    pred_and_plot_image(model=best_model,\n                    image_path=\"https:\/\/cdn.pastemagazine.com\/www\/articles\/2021\/05\/18\/the-office-NEW.jpg\"<\/span>,\n                    class_names=train_dataloader.dataset.classes,\n                    image_size=(config.INPUT_HEIGHT, config.INPUT_WIDTH))<\/span><\/pre>\n
    If you enjoyed this tutorial and think SuperGradients<\/a> is a cool tool to use, then consider giving it a star on on GitHub<\/a>. Looking for a community of deep learning practitioners? Then hang out at the Deep Learning Daily<\/a> community, where deep learning practitioners come to learn new skills and solve their most difficult problems.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
    The ResNet Paper In December of 2015 a paper was published that rocked the deep learning world. Widely regarded as one of the most influential papers in modern deep learning, it has been cited over 110,000 times. The name of this paper will go down in the annals of deep learning history: Deep Residual Learning […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"customer_name":"","customer_description":"","customer_industry":"","customer_technologies":"","customer_logo":"","footnotes":""},"categories":[6],"tags":[],"coauthors":[135],"class_list":["post-7860","post","type-post","status-publish","format-standard","hentry","category-machine-learning"],"yoast_head":"\nResNet: How One Paper Changed Deep Learning Forever - Comet<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"ResNet: How One Paper Changed Deep Learning Forever\" \/>\n<meta property=\"og:description\" content=\"The ResNet Paper In December of 2015 a paper was published that rocked the deep learning world. Widely regarded as one of the most influential papers in modern deep learning, it has been cited over 110,000 times. The name of this paper will go down in the annals of deep learning history: Deep Residual Learning […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/\" \/>\n<meta property=\"og:site_name\" content=\"Comet\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cometdotml\" \/>\n<meta property=\"article:published_time\" content=\"2023-10-06T22:35:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-24T17:05:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png\" \/>\n<meta name=\"author\" content=\"Harpreet Sahota\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Cometml\" \/>\n<meta name=\"twitter:site\" content=\"@Cometml\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Harpreet Sahota\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"ResNet: How One Paper Changed Deep Learning Forever - Comet","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/","og_locale":"en_US","og_type":"article","og_title":"ResNet: How One Paper Changed Deep Learning Forever","og_description":"The ResNet Paper In December of 2015 a paper was published that rocked the deep learning world. Widely regarded as one of the most influential papers in modern deep learning, it has been cited over 110,000 times. The name of this paper will go down in the annals of deep learning history: Deep Residual Learning […]","og_url":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/","og_site_name":"Comet","article_publisher":"https:\/\/www.facebook.com\/cometdotml","article_published_time":"2023-10-06T22:35:37+00:00","article_modified_time":"2025-04-24T17:05:48+00:00","og_image":[{"url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png","type":"","width":"","height":""}],"author":"Harpreet Sahota","twitter_card":"summary_large_image","twitter_creator":"@Cometml","twitter_site":"@Cometml","twitter_misc":{"Written by":"Harpreet Sahota","Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#article","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/"},"author":{"name":"Team Comet Digital","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf"},"headline":"ResNet: How One Paper Changed Deep Learning Forever","datePublished":"2023-10-06T22:35:37+00:00","dateModified":"2025-04-24T17:05:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/"},"wordCount":1895,"publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png","articleSection":["Machine Learning"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/","url":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/","name":"ResNet: How One Paper Changed Deep Learning Forever - Comet","isPartOf":{"@id":"https:\/\/www.comet.com\/site\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#primaryimage"},"image":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#primaryimage"},"thumbnailUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png","datePublished":"2023-10-06T22:35:37+00:00","dateModified":"2025-04-24T17:05:48+00:00","breadcrumb":{"@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#primaryimage","url":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png","contentUrl":"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Hb4l_paTxs7mFuLENeR3HQ.png"},{"@type":"BreadcrumbList","@id":"https:\/\/www.comet.com\/site\/blog\/resnet-how-one-paper-changed-deep-learning-forever\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.comet.com\/site\/"},{"@type":"ListItem","position":2,"name":"ResNet: How One Paper Changed Deep Learning Forever"}]},{"@type":"WebSite","@id":"https:\/\/www.comet.com\/site\/#website","url":"https:\/\/www.comet.com\/site\/","name":"Comet","description":"Build Better Models Faster","publisher":{"@id":"https:\/\/www.comet.com\/site\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.comet.com\/site\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.comet.com\/site\/#organization","name":"Comet ML, Inc.","alternateName":"Comet","url":"https:\/\/www.comet.com\/site\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2025\/01\/logo_comet_square.png","width":310,"height":310,"caption":"Comet ML, Inc."},"image":{"@id":"https:\/\/www.comet.com\/site\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/cometdotml","https:\/\/x.com\/Cometml","https:\/\/www.youtube.com\/channel\/UCmN63HKvfXSCS-UwVwmK8Hw"]},{"@type":"Person","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/6266601170c60a7a82b3e0043fbe8ddf","name":"Team Comet Digital","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.comet.com\/site\/#\/schema\/person\/image\/4f0c0a8cc7c0e87c636ff6a420a6647c","url":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","contentUrl":"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/08\/Screen-Shot-2023-08-12-at-8.58.50-AM-96x96.png","caption":"Team Comet Digital"},"sameAs":["https:\/\/www.comet.ml\/"],"url":"https:\/\/www.comet.com\/site\/blog\/author\/teamcometdigital\/"}]}},"_links":{"self":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7860","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/comments?post=7860"}],"version-history":[{"count":1,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7860\/revisions"}],"predecessor-version":[{"id":15509,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/posts\/7860\/revisions\/15509"}],"wp:attachment":[{"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/media?parent=7860"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/categories?post=7860"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/tags?post=7860"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.comet.com\/site\/wp-json\/wp\/v2\/coauthors?post=7860"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}