{"id":7016,"date":"2023-08-01T06:15:38","date_gmt":"2023-08-01T14:15:38","guid":{"rendered":"https:\/\/live-cometml.pantheonsite.io\/?p=7016"},"modified":"2025-04-24T17:15:00","modified_gmt":"2025-04-24T17:15:00","slug":"introduction-to-multimodal-deep-learning","status":"publish","type":"post","link":"https:\/\/www.comet.com\/site\/blog\/introduction-to-multimodal-deep-learning\/","title":{"rendered":"Introduction to Multimodal Deep Learning"},"content":{"rendered":"\n\n\n\n\n
\n
\n
\"\"<\/picture>
Photo by Annie Spratt<\/a> on Unsplash<\/a><\/figcaption><\/figure>\n<\/div>\n
\n
\n

Our experience of the world is multimodal \u2014 we see objects, hear sounds, feel the texture, smell odors and taste flavors and then come up to a decision. Multimodal learning<\/strong> suggests that when a number of our senses \u2014 visual, auditory, kinesthetic \u2014 are being engaged in the processing of information, we understand and remember more. By combining these modes, learners can combine information from different sources.<\/p>\n

\n
\n
\"\"<\/figure>
<\/picture><\/div>\n<\/div>\n
Baseline of multimodal learning \u2014 Photo on ResearchGate<\/a><\/figcaption>\n<\/figure>\n

When it comes to deep learning, the approach of training models on only source of information\u2014be it images, text, audio, video\u2014is commonplace.<\/p>\n

But there\u2019s also a way to build models that incorporate two data types\u2014say, text and images\u2014at the same time. <\/strong>Working with multimodal data not only improves neural networks, but it also includes better feature extraction from all sources that thereby contribute to making predictions at a larger scale.<\/p>\n

Benefits of multimodal data<\/h1>\n

Modes are, essentially, channels of information. These data from multiple sources are semantically correlated, and sometimes provide complementary information to each other, thus reflecting patterns that aren\u2019t visible when working with individual modalities on their own. Such systems consolidate heterogeneous, disconnected data from various sensors, thus helping produce more robust predictions.<\/p>\n

For example, in an emotion detector, we could combine information gathered from an EEG and also eye movement signals to combine and classify someone\u2019s current mood\u2014thus combining two different data sources for one deep learning task.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Example of multimodal learning \u2014 Photo on Springer<\/a><\/figcaption>\n<\/figure>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n
\n
\n
\n

How multimodal learning works<\/h1>\n

Deep neural networks have been successfully applied to unsupervised feature learning for single modalities\u2014eg. text, images or audio. Here, we aim to do information fusion from different modalities to improve our network\u2019s predictive ability. The overall task can mainly be divided into three phases \u2014 individual feature learning, information fusion and testing.<\/p>\n

\n
\"\"<\/figure>
<\/picture><\/div>\n
Photo by kayis <\/a>on Dev<\/a><\/figcaption>\n<\/figure>\n

We\u2019ll need the following:<\/p>\n