![\"\"\/](\"https:\/\/miro.medium.com\/v2\/resize:fit:2572\/1*0RspkFABiHzobtJlX1X9aw.jpeg\")
<\/figure>\n\n\n\nIntroduction<\/h1>\n
Machine learning has expanded computers\u2019 ability to understand images and extract different information from visual data. In this article, different computer vision tasks will be presented alongside explanations for how each has been tackled using machine learning.<\/p>\n
A lot of machine learning research has been done in the field of computer vision throughout the last 3 decades. Different topics, tasks, and problems have been studied thoroughly; however, we\u2019ll focus on the core problems of computer vision, and we\u2019ll briefly present some of the more advanced hot topics in computer vision towards the end.<\/p>\n
Classification<\/h1>\n
Image classification is the first computer vision task to be tackled by machine learning\u2014in the 1950s, the perceptron algorithm was implemented in the Mark 1<\/a> Perceptron machine, which was used for image classification.<\/p>\n Although this algorithm was efficient for structured data problems, it could only perform well on trivial tasks such as classifying different geometric shapes. A few decades later, the SVM <\/a>algorithm was introduced, which was able to tackle high-dimensional data with a minimum amount of samples, such as small image datasets.<\/p>\n Finally, what has really revolutionized computer vision is the introduction of convolutional neural networks (CNNs) by Yann Lecun<\/a> in his model LeNet<\/a>, which was proven to be superior to other vision-based ML techniques in 2012, when AlexNet <\/a>was the first CNN based model to win the famous ImageNet <\/a>competition.<\/p>\n This part will be short since problem formulation is quite simple for classification. A classification problem generally involves classifying images into 2 or more classes.<\/p>\n In the case of using just two classes (ex: cat and no cat, text and background, etc.), the problem is known as a binary classification<\/em> problem\u2014for which, the last layer of the network will contain 1 neuron with a sigmoid activation function.<\/p>\n And in the case of using more than 2 classes (ex: digits, animals, vehicles, etc.), the problem is deemed a multi-class classification<\/em> problem, for which the last layer will contain The most famous dataset is the MNIST<\/a> Handwritten Digits dataset, which has been used in the early age of computer vision and is still used as an introduction to image classification problems. Although this dataset has played an essential role in the development of computer vision, the task is considered trivial for the current state of the field and industry.<\/p>\n Currently, one of the most significant datasets is ImageNet<\/a>, which consists of 1M samples of 1K classes of different animal species. This dataset is really important for image classification for two reasons.<\/p>\n First, It\u2019s used as a benchmark for evaluating new network architectures through a yearly competition active from 2010 until 2017 and is still used for evaluating of new subsequent architectures. And second, it\u2019s widely used to set pre-trained weights of other networks, as this dataset is so variant it can teach networks to detect important features in images that can be used in other computer vision tasks.<\/p>\n There are a lot of other datasets for image classification that have been used repeatedly through research\u2014such as STL-10<\/a>, CIFAR-10, and CIFAR-100<\/a>\u2014in a manner similar to ImageNet, but with less data and smaller image sizes.<\/p>\n As for medical research, a lot of datasets have been developed for different specific tasks such as ISIC<\/a>, MURA<\/a>, and DermNet<\/a>. Medical datasets are harder to collect which is challenging for complicated tasks, as most of the time it\u2019s not very feasible to collect large datasets when needed.<\/p>\n Although datasets can affect classification performance by presenting different variations in the data, the model architecture is also critical, as it affects the speed and the ability of the network to fit the data.<\/p>\n The LeNet architecture is considered the father of CNNs. It has a simple, shallow architecture, but it demonstrated the ability of convolution layers to learn good features from image data.<\/p>\n This network showed that the deeper the network, the more it can learn. It has been proven efficient through experiments using different numbers of layers in the network. Although this network is considered quite small right now, it still can handle different tasks while using ImageNet for pre-training (weights are already available<\/a>).<\/p>\n In this work, they have successfully built an efficient network that can handle more complex tasks more efficiently, using a special building block (Inception Block) that contains 4 parallel paths, each containing different ConvLayers with different kernel sizes.<\/p>\n This technique enabled the network to utilize different kernel sizes in each layer while giving more weight to the more suitable kernel sizes. Another feature of this network is the use of intermediate classifiers, which could handle the problem of vanishing gradients<\/a> better.<\/p>\n Using residual layers (link), this network attempts to solve the problem of depth\u2014 the deeper the network the harder it is to train. So by adding shortcut connections to the network that skip one or more layers, the network can perform identity mapping, which means it can\u2019t perform worse than a network with fewer layers. Using this technique, they successfully trained a network 8 times deeper than VGG.<\/p>\n A move towards speed was made in this work, as researchers at Google used separable layers that decreased the number of computations required without affecting the model\u2019s performance significantly. This technique made fitting convolutional neural networks on mobile devices<\/a> much more achievable.<\/p>\n Object localization and detection is a computer vision problem in which, given an image, the algorithm has to decide the locations of one or more target objects, outputting bounding boxes for each that appears in the image or video frame.<\/p>\n This task is used heavily in different applications such as self-driving cars, robotics, augmented reality, and medical applications.<\/p>\n This problem can be formulated in different ways depending on the architecture being used. But generally, the network should output a class for each target object in the image using a sigmoid or a softmax activation function.<\/p>\n And to localize the object, the network outputs four variables representing the bounding box, which can be Predicting the bounding box is considered a regression problem. Most of the algorithms require outputting another variable indicating whether an object exists in the selected area or not, as the output is produced for different parts of the image either via CNN implementation or using a sliding window.<\/p>\n Datasets for object localization and detection require more work than for image classification, as the bounding box is annotated around each target object in a given image. Let\u2019s quick review a few datasets.<\/p>\n COCO<\/a>, PASCAL<\/a>, and ImageNet<\/a> datasets are considered the main datasets used for evaluating new object detection architectures. They consist of large numbers of images of general objects like people, animals, cars, planes, etc., with annotations describing bounding boxes and classes for objects in the image. These datasets are also used for segmentation, which we\u2019ll discuss in the next part.<\/p>\n Different techniques are used for building models for object detection. Some models rely on extracting region proposals and classifying each region separately, others use regions of interest (ROI) as an input to the model, while other approaches just use a single-shot network to handle the problem.<\/p>\n Some models have taken a sliding window approach to implement detection, in which a window slides through the image with a certain stride while inserting each block of the image into a classifier. The window is applied to different resolutions of the image to detect objects of different sizes. This approach is considered quite slow compared to the following examples, as the classification model is run once for every possible bounding box in the network.<\/p>\n The R-CNN approach relies on extracting region proposals via selective search. Then each region is wrapped and forwarded to a CNN pre-trained on ImageNet for feature extraction. Finally, the extracted features are forwarded to an SVM to classify each region. This approach has proved more accurate than sliding window approaches, but inference takes longer to process given the network\u2019s separate stages.<\/p>\n Other architectures based on region proposals have followed the R-CNN approach. Fast R-CNN has better performance and speed relative to R-CNN, as it has merged feature extraction and classification into the same CNN. So the network has an image and multiple ROIs as input and it outputs a prediction for both the class and the bounding box for each ROI.<\/p>\n Faster R-CNN took it a step further by extracting ROIs through the network, which improved the accuracy and speed again. The reason behind this is that the network has more freedom to solve the problem. The weights can be updated throughout the network using end-to-end training, so the network has full control over ROIs and feature extraction.<\/p>\n You only look once (YOLO) is a single-shot network that\u2019s designed for optimal performance. It applies a CNN version of the sliding window approach by dividing the input image into an For each cell, there are A single-shot multi-box detector can achieve a faster and more accurate performance than YOLO. It uses features from different levels in the network, which help detect objects with different sizes. Adding to that, the fast implementation of non-maximum suppression is essential for the network, as it outputs a large number of boxes for each image.<\/p>\n Image segmentation is used in various applications (medical, robotics, satellite imagery analysis, etc.) to not only understand the locations of objects in images and video frames, but to more precisely map the boundaries between different objects in the same image.<\/p>\n Image segmentation can be divided into two categories: semantic segmentation and instance segmentation, both of which require pixel-level labels.<\/p>\n For semantic segmentation, the solution requires objects\u2019 pixels for each class of targets in the image to be labeled with the same value. Meanwhile, instance segmentation requires separating different instances of the same class by assigning their pixels different values. Some approaches handle the occlusion of objects so that the occluded part of the object is also represented.<\/p>\n Needless to say, segmentation datasets are of the most time-consuming datasets to build. Again, COCO<\/a> and PASCAL<\/a> datasets are two of the largest datasets for image segmentation, using general objects, as mentioned earlier.<\/p>\n Other datasets are built for more specific applications. For medical applications, there are a lot of datasets such as BraTS<\/a> and Lits<\/a>, which target tasks like tumor segmentation in different parts of the body and different types of diseases. Datasets like SpaceNet<\/a> and the Agriculture-Vision<\/a>Database consist of satellite images, which can have a variety of applications used for labeling large-scale things like streets, buildings, water bodies, etc.<\/p>\n For models handling segmentation problems, a different concept of ConvLayers should be used in addition to the traditional one\u2014transpose convolution or deconvolution<\/mark>. Transpose convolution can output frames with larger spatial sizes than its input, which is needed for segmentation since the network infers the segmented image from features with smaller spatial size.<\/p>\n The U-Net architecture is a full CNN that uses ConvLayers while reducing the spatial size and increasing the depth, and then the reverse by using transpose ConvLayers. Also, an important detail that affects the accuracy of the network is the forward propagation used between the early layers of the network and the later layers with the same spatial sizes, which can provide missing information due to the reduction of the window size.<\/p>\n This network is an extension of Faster R-CNN, adding an extra branch predicting an object mask parallel to the bounding box prediction branch. In addition to this network being the state of the art for segmentation, it has been extended to perform several different tasks such as human pose estimation and rigid object pose estimation.<\/p>\n Computer vision can also tackle different tasks, which are a bit out of the scope of this article; however, we\u2019ll offer simple descriptions for some examples of other tasks.<\/p>\n By learning the distribution of a dataset using approaches like GANs<\/a>, we can generate new images that look real and can be used in new datasets. For example, by going to this website<\/a>, you will see a new picture of a person that looks real\u2014but has just generated by a CNN.<\/p>\n Approaches like GANs and VAEs can be used to transform images from a source domain (street view in summer) to a target domain (street view in winter), which is very beneficial for generalizing networks\u2019 performance on different tasks without annotating new data. This is also used for cool applications like deepfakes.<\/p>\n Another cool application of CNNs is neural style transfer, which uses a content image and a style image to output an image with the same content of the content image, but with the style of the style image. Using this, we can transform normal pictures into version that appear to be created by Van Gogh or Picasso.<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*QqblVCEB2TjuuJhh.png\")
<\/figure>Problem Formulation<\/h1>\n
n<\/code> neurons (n = Number Of Classes)<\/code> with a softmax activation function.<\/p>\nDatasets and Benchmarks<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*zPTiJDBSFU5lkCE1.png\")
<\/figure>Significant Models<\/h1>\n
LeNet5<\/strong> <\/a>(LeCun et al.)<\/h2>\n
VGG<\/strong><\/a> (Simonyan, Zisserman)<\/h2>\n
GoogleNet<\/strong><\/a> (Google) \u2014 Aka InceptionNet<\/h2>\n
RestNet<\/strong><\/a> (Microsoft)<\/h2>\n
MobileNet<\/strong><\/a> (Google)<\/h2>\n
Object Localization and Detection<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:653\/1*52aTV_JJZYdIUXnF-P91Vg.jpeg\")
<\/figure>Problem Formulation<\/h1>\n
(x, y, w, h)<\/code>, with the x, y<\/code> representing either the center or the top right corner of the bounding box.<\/p>\nDatasets and Benchmarks<\/h1>\n
Significant Models<\/h1>\n
Sliding Window (Deformable Parts Models)<\/strong><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*CkyqO-KuYC8HTYdj.png\")
<\/figure>R-CNN<\/strong><\/a><\/h2>\n
R-CNN Improvements<\/strong><\/h2>\n
YOLO<\/strong><\/a><\/h2>\n
ss<\/code> by reducing the image size through the network into ss<\/code> cells with the same depths, where each cell represents a grid in the original image.<\/p>\nB5 + C<\/code> tensors representing B<\/code> bounding box predictions and an array with length C<\/code> representing the class. Because of this implementation detail, the network is limited in the number of nearby objects it can predict, as for each cell of the ss<\/code> grid, it can only predict one class.<\/p>\nSSD<\/strong><\/a><\/h2>\n
Segmentation<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:535\/1*V3YHZCm4EqI8E4E4Qbk-mQ.jpeg\")
<\/figure>Problem Formulation<\/h1>\n
Datasets and Benchmarks<\/h1>\n
Significant Models<\/h1>\n
U-Net<\/strong><\/a><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*EaY7ImL1gy-KFWdBIfYi8g.png\")
<\/figure>Mask R-CNN<\/strong><\/a><\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:274\/1*0DkaX5fQUNIYcCRMTAkDAw.png\")
<\/figure>More Advanced Tasks<\/h1>\n
Data Generation<\/h2>\n
Domain Adaptation<\/h2>\n
Neural Style Transfer<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Ld5_hOPgcLQOn5ICcnknlw.jpeg\")
<\/figure>