![\"Grid](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*UghBE0Itc0NhXkbbMPKMRA.jpeg\")
<\/figure>Computer vision is one of the most widely used and evolving fields of AI. It gives the computer the ability to observe and learn from visual data just like humans. In this process, the computer derives meaningful information from digital images, videos etc. and applies this learning tosolving problems.<\/p>\n
Visual AI Emotion Recognition<\/h1>\n
Emotion recognition is one such derivative of computer vision that involves analysis, interpretation, and classification of the emotional quotient using multimodal features like visual data, body language and gestures to assess the emotional state of the person.<\/p>\n
Computer\u2019s perception of facial expressions<\/h2>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:531\/1*OjebslKaDvbIiELKlyjAKw.png\")
<\/figure><\/picture><\/div>\nSteps of facial emotion recognition [Source<\/a>]<\/figcaption>\n<\/figure>\nFacial expressions and human emotions are tightly coupled. So, for machines to be able to understand the meaning of each emotion, it’s important for them to excel in facial detection, and particularly facial landmark detection.<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*kP1AX4uWgDbwUuh8F8WkvQ.png\")
<\/figure><\/picture><\/div>\n<\/div>\nModeling representations of facial expressions by detecting facial landmarks [Source<\/a>]<\/figcaption>\n<\/figure>\nA model is trained to first identify regions of interest from visual images. It is then re-trained to derive patterns of facial expressions using negative and positive values. Genetic algorithms [1<\/a>] are one way to detect faces in a digital image, followed by the Eigenface technique to verify the fitness of the region of interest. All the valley regions are detected from the gray-scale image to derive high-level facial landmarks like eyes, eyebrows, lips etc.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nCurious to see how Comet works? Check out our PetCam scenario to see MLOps in action<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n\n\n\nApplications of an emotion recognition system<\/h2>\n
Emotion AI has applications in a variety of fields. Some of these are highlighted below:<\/p>\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:662\/1*0Ssjx60T5BGH4Hu_iJPV1g.jpeg\")
<\/figure><\/picture><\/div>\nFacial emotion recognition applications in different scenarios [Source<\/a>]<\/figcaption>\n<\/figure>\n\n- Personalized content and services<\/strong>: Provide personalized content and movie\/music recommendations based on the current emotion of the user.<\/li>\n
- Customer behavior analysis<\/strong>: Learn from the customer\u2019s emotions and expressions while looking at products\/services. Also, some intelligent robots are deployed for behavior-specific advertising.<\/li>\n
- Healthcare<\/strong>: Nurse bots that use Emotion AI observe the patient\u2019s condition and converses with them during the treatment for an overall wellbeing.<\/li>\n
- Public safety and crime control<\/strong>: Detect the state of a driver or their level of drowsiness and trigger an alert to keep the passengers safe.<\/li>\n
- Education<\/strong>: Learning prototypes have been developed and proved efficient in the field of education to adapt to the child\u2019s mood and assess the concentration of the child in a learning environment.<\/li>\n<\/ol>\n
How AI-based emotion analysis works?<\/h1>\n
On a broader level, the process of emotion recognition can be divided into the following steps:<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*i8bSV2IBVxfFtprX.png\")
<\/figure><\/picture><\/div>\n<\/div>\nWorking of emotion recognition system [Source<\/a>]<\/figcaption>\n<\/figure>\nData collection<\/h2>\n
Data is collected by converting video into image sequences by breaking down the frames. Getting a large enough dataset that covers all emotions is difficult, if not impossible, but publicly available open datasets<\/a> like AffectNet, Emotic, K-Emocon etc. are a good point to start.<\/p>\nData preprocessing<\/h2>\n
The image frames are preprocessed by applying techniques like cropping, rotating, resizing, color correction, image smoothening and noise correction to improve the feature vector and the corresponding accuracy of the model.<\/p>\n
Training and classification<\/h2>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*TuocZ_Ubd50f7WEgDeuUPg.png\")
<\/figure><\/picture><\/div>\n<\/div>\nFace detection from an image using Python [Source: Author]<\/figcaption>\n<\/figure>\nAfter pre-processing, we first detect the location of the face (as seen above). Then we detect the facial landmarks (as seen below).<\/p>\n\n\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*H7pt_jZACfmDWJSK31OLRA.png\")
<\/figure><\/picture><\/div>\n<\/div>\nFacial landmarks extraction that contributes to emotion analysis [Source: Author]<\/figcaption>\n<\/figure>\nThe facial features that are extracted can be measured by Action Units (AU), or the distances between facial landmarks like eyebrow raise distance, mouth opening or gradient features, and more. Regions of interest that show maximum expression changes contain rich facial information. To extract more crucial expression features from the facial expression feature map, an attention module like ECA-Net (Wang et al., 2020<\/a>) can be integrated to add greater weight to the core features.<\/p>\n\n
<\/figure>Facial expressions and human emotions are tightly coupled. So, for machines to be able to understand the meaning of each emotion, it’s important for them to excel in facial detection, and particularly facial landmark detection.<\/p>\n A model is trained to first identify regions of interest from visual images. It is then re-trained to derive patterns of facial expressions using negative and positive values. Genetic algorithms [1<\/a>] are one way to detect faces in a digital image, followed by the Eigenface technique to verify the fitness of the region of interest. All the valley regions are detected from the gray-scale image to derive high-level facial landmarks like eyes, eyebrows, lips etc.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Curious to see how Comet works? Check out our PetCam scenario to see MLOps in action<\/a>.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Emotion AI has applications in a variety of fields. Some of these are highlighted below:<\/p>\n On a broader level, the process of emotion recognition can be divided into the following steps:<\/p>\n Data is collected by converting video into image sequences by breaking down the frames. Getting a large enough dataset that covers all emotions is difficult, if not impossible, but publicly available open datasets<\/a> like AffectNet, Emotic, K-Emocon etc. are a good point to start.<\/p>\n The image frames are preprocessed by applying techniques like cropping, rotating, resizing, color correction, image smoothening and noise correction to improve the feature vector and the corresponding accuracy of the model.<\/p>\n After pre-processing, we first detect the location of the face (as seen above). Then we detect the facial landmarks (as seen below).<\/p>\n The facial features that are extracted can be measured by Action Units (AU), or the distances between facial landmarks like eyebrow raise distance, mouth opening or gradient features, and more. Regions of interest that show maximum expression changes contain rich facial information. To extract more crucial expression features from the facial expression feature map, an attention module like ECA-Net (Wang et al., 2020<\/a>) can be integrated to add greater weight to the core features.<\/p>\n<\/figure>Applications of an emotion recognition system<\/h2>\n
<\/figure>\n
How AI-based emotion analysis works?<\/h1>\n
<\/figure>Data collection<\/h2>\n
Data preprocessing<\/h2>\n
Training and classification<\/h2>\n
<\/figure><\/figure>
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/0*CAaQXSoJCHZVVbxO.jpg\")
<\/figure>