{"id":6426,"date":"2023-06-19T18:51:09","date_gmt":"2023-06-20T02:51:09","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=6426"},"modified":"2025-04-24T17:15:23","modified_gmt":"2025-04-24T17:15:23","slug":"guide-to-image-inpainting-using-machine-learning-to-edit-and-correct-defects-in-photos","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/guide-to-image-inpainting-using-machine-learning-to-edit-and-correct-defects-in-photos\/","title":{"rendered":"Guide to Image Inpainting: Using machine learning to edit and correct defects in photos"},"content":{"rendered":"\n\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

\n
\n
\n

We\u2019ve all heard the saying A picture is worth a thousand words<\/em>. But is a tarnished image with gaping holes or splotches or blurs worth a few hundred? What if you just found an age-old photograph of your grandparents\u2019 wedding, but the surface was so worn that you could barely make out their faces. Or perhaps you got photobombed in what would otherwise have been the perfect picture. Or maybe you\u2019re like me and are wondering why hasn\u2019t anyone integrated an option in a smartphone camera app to remove unwanted objects from images?<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>
This view from my school would be just the sort of thing Inpainting could improve.<\/figcaption><\/figure>\n

What is Image Inpainting?<\/h1>\n

Inpainting is the process of reconstructing lost or deteriorated parts of images and videos. In the museum world, in the case of a valuable painting, this task would be carried out by a skilled art conservator or art restorer. In the digital world, inpainting refers to the application of sophisticated algorithms to replace lost or corrupted parts of the image data. (source<\/a>)<\/p><\/blockquote>\n

This official definition of inpainting on Wikipedia<\/a> already takes into account the use of \u201csophisticated algorithms\u201d that do the same work of manually overwriting imperfections or repairing defects but in a fraction of the time.<\/p>\n

As deep learning technologies progress further, however, the process of inpainting has become automated in so complete a manner that these days, it requires no human intervention at all. Simply feed a damaged image to a neural network and receive the corrected output. Go ahead and try it out yourself, with NVIDIA\u2019s web playground<\/a> that demonstrates how their network fills in a missing portion for any image.<\/p>\n

Simply drag and drop any image file, erase a portion of it with the cursor and watch how the AI patches it up. I tried it on a few pictures lying around on my desktop. Here\u2019s one of them below, with a big chunk of my face missing and the neural network restoring it again in a matter of seconds, albeit making me look like I just got out of a street fight.<\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

You can also use it to quickly getting rid of something in a picture, too. Here\u2019s another image I had lying around. A great view of Hangzhou\u2019s west lake with the picturesque Leifeng Pagoda in the distance. The AI does a great job in envisioning a new lake with no pagoda around.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n<\/figure>\n

Traditional forms of image restoration usually evolve around some simple concept. Given a gap in pixels, fill the gap with pixels that are the same as, or similar to, neighboring pixels. These techniques are generally dependent on various factors and are more efficient for removing noise or small defects from images. They will most likely fail when the image has huge gaps or a significant amount of missing data.<\/p>\n

The most straightforward and conventional technique for image restoration is deconvolution<\/a>, which is performed in the frequency domain and after computing the Fourier transform<\/a> of both the image and the PSF to undo the resolution loss caused by the blurring factors. The result of applying this technique usually creates an imperfectly deblurred image.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Source<\/a><\/figcaption>\n<\/figure>\n

In a basic sense, inpainting does refer to the restoration of missing parts of an image based on the background information. It can be thought of as a process of filling in missing data in a designated region of visual input.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\n

A New Approach with Machine Learning<\/h1>\n

The new age alternative is to use deep learning to inpaint images by utilizing supervised image classification. The idea is that each image has a specific label, and neural networks learn to recognize the mapping between images and their labels by repeatedly being taught or \u201ctrained.\u201d<\/p>\n

When trained on huge training datasets (millions of images with thousands of labels), deep networks have remarkable classification performance that can often surpass human accuracy. Generative adversarial networks are typically used for this sort of implementation, given their ability to \u201cgenerate\u201d new data, or in this case, the missing information.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Courtesy : NVIDIA<\/figcaption>\n<\/figure>\n

The basic workflow is as follows: feed the network an input image with \u201choles\u201d or \u201cpatches\u201d that need to be filled. These patches can be considered a hyperparameter required by the network since the network has no way of discerning what actually needs to be filled in. For instance, a picture of a person with a missing face conveys no meaning to the network except changing values for pixels.<\/p>\n

To enable the neural network understand what part of the image actually needs filling in, we need a separate layer mask that contains pixel information for the missing data. The input image then goes through several convolutions and deconvolutions as it traverses across the network layers. The network does produce an entirely synthetic image generated from scratch. The layer mask allows us to discard those portions that are already presented in the incomplete image, since we don\u2019t need to fill those parts in. The new generated image is then superimposed on the incomplete one to yield the output.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Comparison of various inpainting approaches<\/figcaption>\n<\/figure>\n

A discriminator network, such as the one in a conventional GAN, can prove useful at such points. Its main use in such a scenario is to ensure that the final image obtained after filling in the gaps doesn\u2019t look obviously fake. When compared with the original image, it needs to look reasonably similar, containing minute differences.<\/p>\n

The problem here can be the absence of an original image to compare against. A discriminator network trained on thousands of image samples might be just the thing. Iizuka et al.<\/a> propose a version of a network that can be used for image inpainting pictured below.<\/p>\n

It consists of a completion network (for convolution and deconvolution) and two auxiliary context discriminator networks that are used only for training the completion network and are not used during testing. The global discriminator network takes the entire image as input, while the local discriminator network takes only a small region around the completed area as input. Both discriminator networks are trained to determine if an image is real or completed by the completion network, while the completion network is trained to fool both discriminator networks.<\/p>\n<\/div>\n<\/div>\n

\n
\n
\n
\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Globally and Locally Consistent Image Completion (Iizuka et. al.<\/a>)<\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n
\n
\n

Not long ago, researchers at NVIDIA published a thesis<\/a> on their new and improved method for inpainting irregular holes in images using partial convolutions. They claimed that previous approaches with deep learning focused on rectangular holes, usually around the image center, and often relied on expensive post-processing or \u201ctouching up\u201d.<\/p>\n

Their new method \u201coperates robustly on irregular hole patterns, and produces semantically meaningful predictions that incorporate smoothly with the rest of the image without the need for any additional post-processing or blending operation.\u201d<\/p>\n

\n
\n