{"id":6430,"date":"2023-06-19T19:00:23","date_gmt":"2023-06-20T03:00:23","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=6430"},"modified":"2025-04-24T17:15:21","modified_gmt":"2025-04-24T17:15:21","slug":"deep-learning-for-image-segmentation-u-net-architecture","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/deep-learning-for-image-segmentation-u-net-architecture\/","title":{"rendered":"Deep Learning for Image Segmentation: U-Net Architecture"},"content":{"rendered":"\n\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\n
\n
\n

\u201cMy life seemed to be a series of events and accidents. Yet when I look back, I see a pattern.\u201d Benoit Mandelbrot<\/a><\/p><\/blockquote>\n

U-Net, a kind of Convolutional Neural Networks (CNN) approach, was first proposed by Olaf Ronneberger, Phillip Fischer, and Thomas Brox in 2015 with the suggestion of better segmentation on biomedical images.<\/p>

The paper we\u2019ll be exploring is U-Net: Convolutional Networks for Biomedical Image Segmentation<\/em><\/a><\/p><\/blockquote>\n

Why segmentation is needed and what U-Net offers<\/h1>\n

Basically, segmentation is a process that partitions an image into regions. It is an image processing approach that allows us to separate objects and textures in images. Segmentation is especially preferred in applications such as remote sensing or tumor detection in biomedicine<\/strong>.<\/p>\n

There are many traditional ways of doing this<\/a>. For example; point, line, and edge detection methods, thresholding, region-based, pixel-based clustering, morphological approaches, etc. Various methods have been developed for segmentation with convolutional neural networks (a common deep learning architecture), which have become indispensable in tackling more advanced challenges with image segmentation. In this post, we\u2019ll take a closer look at one such architecture: u-net.<\/p>\n

\u2753 In deep learning, it\u2019s known that we need large datasets for model training. But there are some problems we run into at this point! We often cannot afford the amount of data that needs to be collected for an image classification problem. In this context, affordability means time, money, and most importantly, hardware.<\/p>\n

For example, it isn\u2019t possible to collect many biomedical images with the camera on your mobile phone. So we need more systematic ways to collect data. There\u2019s also the data labeling process, for which a single developer\/engineer will not suffice\u2014this will require a lot of expertise and experience in classifying the relevant images. This is especially true with highly-specialized areas such as medical diagnostics.<\/p>\n

Another critical point is to provide education about the general image in classically convolutional neural networks through class labels. However, some problems require knowledge of localization\/positioning with pixel-based approaches. In areas that require sensitive approaches, such as biomedical or defense, we need class information for each pixel.<\/p>\n

\u2714\ufe0fU-Net is more successful than conventional models, in terms of architecture and in terms pixel-based image segmentation formed from convolutional neural network layers. It\u2019s even effective with limited dataset images. The presentation of this architecture was first realized through the analysis of biomedical images.<\/p>\n

\ud83d\udd0e Differences that make U-Net special!<\/h1>\n

As it\u2019s commonly known, the dimension reduction<\/strong> process in the height and width that we apply throughout the convolutional neural network\u2014that is, the pooling layer<\/em> \u2014 is applied in the form of a dimension increase<\/strong> in the second half of the model.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>
Representation: Max and Avg. Pooling<\/a><\/figcaption><\/figure>\n

The pooling layer reduces height and width information by keeping the number of channels of the input matrix constant. The calculation is a step used to reduce complexity (Each element of the image matrix is called a pixel).<\/em> In summary, the pooling layer refers to a pixel that represents groups of pixels.<\/p>\n

Note: Pooling layers can work with different approaches, including maximum, average, or median layers.<\/p><\/blockquote>\n

These layers are intended to increase the resolution of the output. For localization, the sampled output is combined with high-resolution features throughout the model. A sequential convolution layer then aims to produce a more precise output based on this information.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
U-Net Model<\/a><\/figcaption>\n<\/figure>\n

U-Net takes its name from the architecture, which when visualized, appears similar to the letter U<\/em>, as shown in the figure above. Input images are obtained as a segmented output map. The most special aspect of the architecture in the second half. The network does not have a fully-connected layer. Only the convolution layers are used. Each standard convolution process is activated by a ReLU activation function.<\/p>\n

U-Net consists of a contracting path (left side) and an expansive path (right side)!<\/p><\/blockquote>\n

\ud83c\udf6d You can read about <\/em>activation functions<\/em><\/strong> in more detail <\/em>here<\/em><\/strong><\/a>.<\/em><\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Representation of a convolution and deconvolution process in U-Net<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

The pixels in the border region are symmetrically added around the image so that images can be segmented continuously. With this strategy, the image is segmented completely. The padding (pixel adding) method is important for applying the U-Net model to large images; otherwise, the resolution will be limited by the capacity of the GPU memory. The result of padding and segmenting with the mirroring I mentioned is shown in the figure below.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Overlap-tile strategy<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n

\ud83d\udc0bThe difference between U-Net and the autoencoder architecture<\/strong><\/h2>\n

To help highlight what makes U-Net unique, it might be helpful to quickly compare it to a different traditional approach to image segmentation: the autoencoder architecture.<\/p>\n

In a classical autoencoder<\/a> architecture, the size of the input information is initially reduced, along with the following layers.<\/p>\n

At this point, the encoder part of the architecture is completed and the decoder part begins. Linear feature representation is learned in this section, and the size gradually increases. At the end of the architecture, the output size is equal to the input size.<\/em><\/strong><\/mark><\/p>\n

This architecture is ideal in preserving the output size, but one problem is that it compresses the input linearly, which results in a bottleneck in which all features cannot be transmitted.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Autoencoders Model<\/figcaption>\n<\/figure>\n

This is where U-Net differs. U-Net performs deconvolution on the decoder side (i.e. in the second half) and, in addition, can overcome this bottleneck problem, which results in the loss of features through connections from the encoder side of the architecture.<\/p>\n

\ud83c\udfca\ud83c\udffbLet\u2019s continue with U-Net!<\/strong><\/h2>\n

Let\u2019s return to our specific use case at hand\u2014biomedical image segmentation. The most common variation in tissue in a biomedical image is deformation, and realistic deformations can be efficiently simulated. In this way, the learning process is more successful with the elastic deformation approach, which helps us increase the size of our dataset.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Representation of Elastic Deformation<\/figcaption>\n<\/figure>\n

In addition, it\u2019s difficult to determine the boundaries when there are parts of the same class that touches each other. For this purpose, it\u2019s recommended to use the values that have a large weight in the loss function, while separating the information to be segmented from the background first.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
HeLa cells recorded by DIC (differential interference contrast) microscopy. a) raw image b) Ground truth segmentation. Different colors show different examples of HeLa cells. c) Created segmentation mask (black and white) d) A map with a lost weight in pixels to allow the network to learn edge pixels.<\/figcaption>\n<\/figure>\n

\ud83d\udcc8 Loss Approaches<\/h1>\n

Loss can be calculated by standard binary cross-entropy and Dice<\/a> loss, which is a frequently-used performance criterion for assessing success in biomedical images.<\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\"\"<\/picture><\/figure>\n
\"\"<\/picture>\n
Loss: Binary cross-entropy and Dice<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n

Intersection over Union (IoU)<\/strong> is a pixel-based criterion and is often used when evaluating segmentation performance.<\/p>\n

The varying pixel ratio between the target matrix and the resulting matrix is considered. This metric is also associated with the Dice calculation.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Visualization of IoU expression<\/figcaption>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
Input and image labeled by input<\/figcaption>\n<\/figure>\n

Here\u2019s a look at how U-Net performs on EM image segmentation<\/a>, as compared to other approaches:<\/h2>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Here is the U-Net<\/figcaption>\n<\/figure>\n

The success of the u-net architecture has been registered on different datasets. You can download the trained model here<\/a>!<\/p><\/blockquote>\n

Results from PhC-U373 and DIC-HeLa datasets and comparison with previous studies:<\/h2>\n<\/div>\n<\/div>\n
\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n
\n
\"\"<\/figure>
<\/picture><\/div>\n
U-net\u2019s segmentation success on PhC-U373 (a-b) and DIC-HeLa (c-d) datasets<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

Of course, segmentation isn\u2019t only used for medical images; earth sciences or remote sensing systems from satellite imagery also use segmentation, as do autonomous vehicle systems. After all, there are patterns everywhere.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n
\n
\n
\n

\ud83d\udc8e It is also very important how the data should be labeled for segmentation. Sometimes you may encounter data that is not fully labeled or the data may be imbalanced. I recommend another practical resource written by neptune.ai<\/strong><\/a>, that you can review under these situations: How to Do Data Exploration for Image Segmentation and Object Detection (Things I Had to Learn the Hard Way)<\/a><\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

\ud83c\udfa7\u201c<\/em>Pattern<\/em><\/a>\u201d \u2014 with this song, let me continue to write <\/em>\ud83d\ude05<\/h2>\n

\ud83c\udf0bTGS Salt Identification Challenge<\/h1>\n

There are large deposits of oil and gas and large deposits of salt beneath the surface in various areas of the Earth. Unfortunately, it\u2019s very difficult to know where the large salt deposits are.<\/p>\n

Professional seismic imaging requires expert interpretation of salt bodies. This leads to very subjective, variable predictions. To generate the most accurate seismic images and 3D imaging, TGS (geology data company)<\/a>hopes that Kaggle\u2019s machine learning community can create an algorithm that automatically and accurately determines whether an underground target is a salt.<\/p>\n

Here are some examples of successful u-net approaches:<\/p>\n

TGS Salt Identification Challenge<\/a><\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Salt Identification Challenge<\/a><\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

\ud83c\udf0eMapping Challenge \u2014 Building Missing Maps with Segmentation<\/h1>\n

The determination of map regions by using satellite imagery is another u-net application area. In fact, it can be said that the applications that will emerge with the development of this field will greatly facilitate the work of mapping and environmental engineers.<\/p>\n

We can use this method not only for defense industry applications but also for urban district planning applications. For example, in the competition for the detection of buildings (details of the competition can be found here<\/a>), mean accuracy of 0.943<\/code> and mean sensitivity of 0.954<\/code> is reached. You can see the u-net model of this successful study here<\/a>.<\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Result of the Mapping Challenge \u2014 Neptune.ML<\/a><\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

\ud83d\udd17 U-net\u2019s inspiration for other deep learning approaches<\/h1>\n

U-net inspired the combination of different architectures as well as other computer vision deep learning models.<\/p>\n

\n
\n