![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*ckmHcR0yC0LSVkQMJSVafQ.jpeg\")
<\/figure>AI tools such as ChatGPT, DALL-E, and Midjourney are increasingly becoming a part of our daily lives. These tools were developed with deep learning techniques. Deep learning <\/strong>is a subfield of AI that aims to extract knowledge from data. Today, I\u2019ll walk you through how to perform an end-to-end deep learning project using PyTorch, Comet ML, and Gradio. Here are the topics we\u2019ll cover in this blog:<\/p>\n By the end of this article, you\u2019ll learn step-by-step the life cycle of a deep learning project on how to perform image classification using the cat vs. dog dataset. After finishing the project, our app will look like this:<\/p>\n Before diving into the project, let me explain the libraries I\u2019m going to use in this analysis. Let\u2019s start with PyTorch:<\/p>\n Two frameworks are generally used for deep learning: TensorFlow and PyTorch. TensorFlow is mostly used in industry, while PyTorch is used for academic research. I\u2019m going to use PyTorch for this project because of its user-friendliness, flexibility, and robust community support. Let\u2019s move on and take a look at another library.<\/p>\n When implementing deep learning projects, you\u2019ll need to track your hyperparameters, visualize performance metrics, monitor models, and share experiments with others. This is when Comet ML comes into play.<\/p>\n Comet ML is an machine learning platform that allows you to manage, visualize, compare and optimize models. We\u2019re going to use Comet ML to track our hyperparameters and to monitor our model. Let\u2019s move on and have a look at what Gradio is.<\/p>\n Deep learning projects that are not moved to production are dead projects. Gradio is an open-source Python library that helps you build easy-to-use demos for your ML model that you can share with other people.<\/p>\n Beautiful! We briefly talked about the libraries we\u2019ll use. Let\u2019s go ahead and start loading our dataset.<\/p>\n Beginners often start with clean datasets like the MNIST dataset to learn deep learning. It\u2019s good to start with these datasets, but real-world datasets aren\u2019t always clean. One of the challenges in deep learning is loading and working with a custom dataset. The dataset we\u2019ll use is the cat and dog dataset<\/a>, which contains images of cats and dogs.<\/p>\n Before loading this dataset, let\u2019s launch an Experiment in Comet ML..<\/p>\n If you don\u2019t have a Comet ML account yet, you can create a free account here<\/a>. To follow along with the code in this blog, you can access the notebook I used in this project here<\/a>.<\/p>\n Awesome! We started our experiment. Now let’s take a look at our version of Torch and check if we have access to CUDA (GPU). CUDA is a parallel computing platform developed by NVIDIA that makes calculations faster. You can use it for free (in limited amounts) on Google Colab or Kaggle notebooks.<\/p>\n Now let\u2019s create our training and test paths that we\u2019ll use while loading data.<\/p>\n Excellent! Now let\u2019s get a random image and look at the features of this image.<\/p>\n This is a cat image and the size of this image is 217*179. What a cute cat, right? I love cats. Let\u2019s go ahead and create the necessary functions to load the dataset.<\/p>\n So far, we created variables for the dataset paths and explored an image from the dataset. Note that images in a dataset may not all be the same size. In this case, we\u2019ll need to do some data preprocessing to standardize the size, shape, and format of th stylee pictures.<\/p>\n Transforming data, also known as preprocessing, helps you prepare quality data. With transforming, you can improve the performance of your model and reduce the risk of bias. Let\u2019s transform our dataset with Here, we used the Nice, we determined how to transform our dataset. Now we\u2019re ready to create our own custom torch <\/a> You can find many ready-made datasets such as MNIST, and CIFAR100 in the We can use this function because the format of our dataset is as shown above. Let’s load images from train and test folders into Note that PyTorch has two great functions for loading the dataset: Awesome! We have prepared the necessary functions to load the dataset. We are now ready to build a CNN-based model.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Want to see the evolution of AI-generated art projects? Visit our public project to see time-lapses, experiment evolutions, and more<\/a>!<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Convolutional neural network (CNN) is a deep learning technique often used to extract patterns in visual data. CNN consists of at least three layer types: the convolutional layer, the pooling layer and then the fully connected layer.<\/p>\n\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*uF37Jln4iYi-EMlirSbngw.gif\")
<\/figure>Introduction<\/h1>\n
PyTorch<\/a><\/h2>\n
<\/figure>
Comet ML<\/a><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*e-HOiPQNHDUImZT8i6yVuw.png\")
<\/figure>Gradio<\/a><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*8uLrpAWKCrZCRgZpN5AeZg.png\")
<\/figure>Loading the Dataset<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*nG1Ahtu7A4neez6fxonJGA.png\")
<\/figure># Installing comet_ml<\/span>\n# !pip install comet_ml<\/span>\n\n# Importing the comet_ml library<\/span>\nimport<\/span> comet_ml\nfrom<\/span> comet_ml import<\/span> Experiment\n\n# Building an experiment with your API key<\/span>\nexperiment = Experiment(\n api_key= my_api_key,\n workspace=\"tirendaz-academy\"<\/span>,\n project_name=\"experiment-tracking\"<\/span>)\n\n# Setting hyperparameters<\/span>\nhyper_params = {\"seed\"<\/span>: 42<\/span>, \"batch_size\"<\/span>: 32<\/span>, \"num_epochs\"<\/span>: 20<\/span>,\n \"learning_rate\"<\/span>: 1e-3<\/span>,\"image_size\"<\/span>: 224<\/span>}\n\n# Logging hyperparamters<\/span>\nexperiment.log_parameters(hyper_params)<\/span><\/pre>\nimport<\/span> torch\nfrom<\/span> torch import<\/span> nn\n\n# Make sure torch >= 1.10.0<\/span>\nprint<\/span>(\"The version of torch:\"<\/span>, torch.__version__)\n\n# Setuping cuda<\/span>\ndevice = \"cuda\"<\/span> if<\/span> torch.cuda.is_available() else<\/span> \"cpu\"<\/span>\nprint<\/span>(\"The type of device: \"<\/span>,device)\n\n# Output:<\/span>\nThe version of torch: 1.11<\/span>.0<\/span>\nThe type<\/span> of device: cuda<\/span><\/pre>\n# Creating our paths<\/span>\nmy_train_dir = \"\/kaggle\/input\/cat-and-dog\/training_set\/training_set\"<\/span>\nmy_test_dir = \"\/kaggle\/input\/cat-and-dog\/test_set\/test_set\"<\/span><\/span><\/pre>\nimport<\/span> random\nfrom<\/span> PIL import<\/span> Image\nimport<\/span> glob\nfrom<\/span> pathlib import<\/span> Path\n\n# Setting seed<\/span>\nrandom.seed(hyper_params[\"seed\"<\/span>])\n\n# Creating our image path<\/span>\nimage_path= glob.glob(f\"{image_path}<\/span>\/*\/*\/*\/*.jpg\"<\/span>)\n\n# Getting random a path<\/span>\nrandom_image_path = random.choice(image_path)\n\n# Creating a variable for the path <\/span>\nimage_class = Path(random_image_path).parent.stem\n\n# Let's open the image<\/span>\nimage = Image.open<\/span>(random_image_path)\n\n# Let's print our metadata<\/span>\nprint<\/span>(\"Random image path: {}\"<\/span>.format<\/span>(random_image_path))\nprint<\/span>(\"Image class: {}\"<\/span>.format<\/span>(image_class))\nprint<\/span>(\"Image height: {}\"<\/span>.format<\/span>(image.height))\nprint<\/span>(\"Image width: {}\"<\/span>.format<\/span>(image.width))\nimage<\/span><\/pre>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*NcX7Z0_zzZIcLSsvzboAFA.png\")
<\/figure>Transforming data<\/h2>\n
torchvision.transforms<\/a><\/code> in PyTorch.<\/p>\nfrom<\/span> torchvision import<\/span> transforms\n\n# Setting our image size<\/span>\nIMAGE_SIZE=(hyper_params[\"image_size\"<\/span>], hyper_params[\"image_size\"<\/span>])\n\n# Creating a transform for training using TrivialAugment<\/span>\nmy_train_transform = transforms.Compose([\n transforms.Resize(IMAGE_SIZE),\n transforms.TrivialAugmentWide(),\n transforms.ToTensor()])\n\n# Creating a transform for testing <\/span>\nmy_test_transform = transforms.Compose([\n transforms.Resize(IMAGE_SIZE),\n transforms.ToTensor()])<\/span><\/pre>\nTrivialAugmentWide<\/a><\/code> function, which is a data augmentation technique. Now let\u2019s take a step back and talk about what data augmentation is. Data augmentation<\/strong> is a method used to artificially increase the diversity of your data by modifying your existing data. This technique is often utilized when the dataset is small. I encourage you to examine the examples of the various transforms here<\/a>.<\/p>\nDataset<\/a><\/code> class<\/a>.<\/p>\nCreating a Custom
Dataset<\/code><\/h2>\ntorchvision.datasets<\/a><\/code> module. But in most cases, you need to handle real-world datasets. If you want, you can create your own class to load the dataset in PyTorch. But, the good news is that you can use the ImageFolder<\/a><\/code> function if the format of your dataset is as shown below:<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*V1IinsPrz8SWHMIapw2uaw.png\")
<\/figure>Datasets<\/code> with the ImageFolder<\/code> function.<\/p>\n# Converting our image folders ito Datasets<\/span>\nfrom<\/span> torchvision import<\/span> datasets\n\n# Converting our image folders ito Datasets<\/span>\nmy_train_data = datasets.ImageFolder(my_train_dir, transform=my_train_transform)\nmy_test_data = datasets.ImageFolder(my_test_dir, transform=my_test_transform)<\/span><\/pre>\nDataset<\/a><\/code>and <\/a>DataLoader<\/a><\/code>. The samples and their related labels are stored in Dataset<\/code>. DataLoader<\/code> iteratively wraps the Dataset<\/code> for easy access to samples. We built our custom Dataset<\/code>s. It\u2019s time to turn our custom Dataset<\/code>s into DataLoader<\/code>s. Show time!<\/p>\nfrom<\/span> torch.utils.data import<\/span> DataLoader\n\n# Setting some parameters<\/span>\ntorch.manual_seed(hyper_params[\"seed\"<\/span>])\nNUM_WORKERS = os.cpu_count()\n\n# Creating a training DataLoader<\/span>\nmy_train_dataloader = DataLoader(my_train_data,\n batch_size=hyper_params[\"batch_size\"<\/span>],\n shuffle=True<\/span>,\n num_workers=NUM_WORKERS)\n\n# Creating a test DataLoader<\/span>\nmy_test_dataloader = DataLoader(my_test_data,\n batch_size=hyper_params[\"batch_size\"<\/span>],\n shuffle=False<\/span>,\n num_workers=NUM_WORKERS)<\/span><\/pre>\nModel Building<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*vn_hdIrYC9fqL-6AiwxoQQ.gif\")
<\/figure>