![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/format:webp\/1*Wd5nSZa7p069yG8i8yCTwA.png)
<\/picture>In a previous article<\/a>, I discussed the possibilities of computer vision-based deep learning with both RNNs and CNNs.<\/p>\n Generally, ML engineers will specialize in one model architecture and let the other slide.<\/p>\n My point and purpose for writing this post is the following: learning both allows to tackle a wider range of use-cases.<\/p><\/blockquote>\n Last week, I tried the final project of the course Introduction to Deep Learning<\/a>from HSE (Higher School of Economics). In this project, we learn how to use the output of a convolutional neural network (CNN) for tasks other than image classification or regression.<\/p>\n Here, we\u2019ll instead learn how to feed this output into another neural network: a recurrent neural network (RNN). An RNN is a type of neural network that can work with sequences such as text, sound, videos, finance data, and more.<\/p>\n Combining CNNs and RNNs helps us work with images and sequences of words in this case. The goal, then, is to generate captions for a given image.<\/p>\n \ud83d\udcac For example, we could run the desired network on Conor McGregor\u2019s UFC image and get a description. This will be our objective.<\/strong><\/p>\n A picture is worth a thousand words. But sometimes we actually want the words.<\/p><\/blockquote>\n Let\u2019s pause a moment and try to understand the possibilities of image captioning.<\/p>\n If the output is a bunch of words, it means that we are going to use these words. <\/strong>Specifically, we\u2019ll use these words for contextual understanding, or to describe more detailed scenarios.<\/p>\n Let\u2019s say that you have to identify a specific type of clothing to then recommend other clothes in matching styles. This is called \u201cvisual search\u201d and could change fashion retail forever.<\/p><\/blockquote>\n In the image above, captioning can help us understand the specific clothes a person is wearing\u2014and their overall style. In this example, the detail is not strong enough. A better image to input to the network would include more specific fashion items.<\/p>\n Going to the extreme use case, we could even translate a football match in real-time and replace the on-air commentary with AI-generated voices discussing the match. I\u2019m not talking about robot voice here; we could imitate whoever we want.<\/p>\n Just look at this AI that can imitate Joe Rogan: https:\/\/fakejoerogan.com<\/a><\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n To do this, we need to use 2 different neural networks: a CNN and an RNN. Here I assume you\u2019re a bit familiar with both. Before getting into technical details, let\u2019s view the dataset and the output we want to generate.<\/p>\n The dataset is a collection of images and captions. Here, it\u2019s the COCO dataset<\/a>. For each image, a set of sentences (captions) is used as a label to describe the scene.<\/p>\n It means our final output will be one of these sentences.<\/p><\/blockquote>\n The words are converted into tokens through a process of creating what are called word embeddings<\/strong>.<\/p><\/blockquote>\n The process to convert an image into words\/token is as follows:<\/p><\/blockquote>\n TL;DR \u2014 We have images and sentences for each. Sentences are converted into vectors.<\/p><\/blockquote>\n The encoder is a convolutional neural network named Inception V3.<\/strong> This is a popular architecture for image classification.<\/p>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*Wd5nSZa7p069yG8i8yCTwA.png\")
<\/figure>Before we start<\/h2>\n
\n
Why image captioning?<\/h1>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*DKqniHLa5_5ZFwVN_ncgZw.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*XnC1B8q3jHGwtUpMhtIDRQ.png\")
<\/figure>How?<\/h1>\n
Dataset<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:534\/1*CNk55KpPjeK12qVGZ1xZxA.png\")
<\/figure>Pre-processing<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:611\/1*IitCsxlfRaEsbU_6bzU6Bw.png\")
<\/figure>\n
Encoder<\/h2>\n
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:1088\/1*7tAF6FTcycwW8jWbX2MqJQ.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:662\/1*9Jv-tjzpcKT5_CvuayLXZg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:766\/1*-YJ3Ut7W_Os01kp53CSZYA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:856\/1*r9MBLECh9btjfk_VaLoKOA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*qo28kO9eP0LlHcY9E42-6A.png\")
<\/figure>