Artificial intelligence is actively developing in Russia and the world - including emotional. He is interested in large companies and ambitious startups that are introducing new developments in retail, marketing, education, banking, and recruiting. According to Mordor Intelligence, the emotion recognition market was valued at $ 12 billion in 2018 and will grow to $ 92 billion by 2024.
What is emotional AI
Emotion AI (Emotion AI) is an AI that enables a computer to recognize, interpret and respond to human emotions. A camera, microphone, or wearable sensor reads a person's state, and a neural network processes the data to determine an emotion.
There are two main ways to analyze emotions:
- Contact. A person is put on a device that reads his pulse, body electrical impulses and other physiological indicators. Such technologies can determine not only emotions, but also the level of stress or the likelihood of an epileptic seizure.
- Contactless. Emotions are analyzed on the basis of video and audio recordings. The computer learns facial expressions, gestures, eye movements, voice and speech.
To train a neural network, data scientists collect a sample of data and manually mark the change in a person's emotional state. The program studies patterns and understands which signs belong to which emotions.
The neural network can be trained on different data. Some companies and labs use videotapes, others study voice, and some benefit from multiple sources. But the more diverse the data, the more accurate the result.
Consider two main sources:
Promotional video:
Photos and stills from video
Images are processed first to make it easier for the AI to work with. Facial features - eyebrows, eyes, lips, and so on - are marked with dots. The neural network determines the position of the points, compares them with the signs of emotions from the template and concludes which emotion is reflected - anger, fear, surprise, sadness, joy or calmness.
There is also another approach. Markers of emotions are immediately noted on the face - for example, a smile or frowning eyebrows. Then the neural network looks for markers on the image, analyzes their combinations and determines the state of the person.
The study of emotion markers began in the 20th century. True, then they were considered separately from neural networks. Scientists Paul Ekman and Wallace Friesen developed the Facial Action Coding System (FACS) in 1978. It breaks down facial expressions into individual muscle movements, or Action Units. The researcher studies motor units and compares them with emotion.
Voice and speech
The neural network extracts many parameters of the voice from the acoustic signal - for example, tone and rhythm. She studies their change in time and determines the state of the speaker.
Sometimes a spectrogram is used for training - an image that shows the strength and frequency of a signal over time. In addition, the AI analyzes vocabulary for more accurate results.
Where is the technology used
Sales and advertising
The most obvious use of emotion recognition technology is in marketing. With their help, you can determine how an advertising video affects a person. To do this, you can, for example, install a structure with a camera that will change advertising depending on the mood, gender and age of people passing by.
A similar design was developed by startups Cloverleaf and Affectiva. They introduced an electronic shelfpoint ad called shelfPoint that collects data about shoppers' emotions. New technologies have been tested by Procter & Gamble, Walmart and other large companies. According to Cloverleaf, sales rose 10-40%, while customer engagement increased 3-5 times.
A more unusual option is a robot consultant with artificial intelligence. He will interact with clients, read their emotions and influence them. And also make personalized offers.
The service robot was presented by the Russian startup Promobot. It uses a neural network developed by Neurodata Lab, which determines emotions from several sources at once: recordings of a face, voice, movements, as well as breathing and pulse rates.
Promobot actively sells its robots abroad. In 2018, the startup signed a contract with the American company Intellitronix for $ 56.7 million, and in the next agreed to supply devices to Saudi Arabia, Israel, Kuwait and Sweden - for them the company will receive $ 1.1 million. According to Promobot, today 492 robots are working in 34 countries around the world as guides, concierges, consultants and promoters.
Banks
Emotion recognition technologies help banks get customer feedback without surveys and improve service. Video cameras are installed in the departments, and algorithms for recording determine the satisfaction of visitors. Neural networks can also analyze the voice and speech of the client and operator during a call to the contact center.
In Russia, they have been trying to implement emotional AI for a long time: it was tested at Sberbank back in 2015, and three years later, Alfa-Bank launched its pilot for analyzing emotions from video. In addition to recordings from surveillance cameras, call recordings are also used. VTB launched a pilot project to implement emotional AI in 2019. And Rosbank, together with Neurodata Lab, have already tested the determination of customers' emotions by voice and speech. The client called the bank, and the neural network analyzed his state and the meaning of the conversation. In addition, the AI noticed pauses in the operator's speech, voice volume and communication time. This allowed not only to check the satisfaction with the service, but also to monitor the work of the contact center operators.
Now Rosbank has implemented its own solution for emotion recognition. Instead of an acoustic signal, the system analyzes the text, while the accuracy remains high.
The Speech Technology Center is also involved in recognizing emotions in speech (Sberbank owns a majority stake). The Smart Logger service analyzes the voice and vocabulary of customers and operators, talk time and pauses in order to find out the satisfaction with the service.
Entertainment sphere
Emotion recognition systems can be used to gauge the audience's reaction to a movie. Disney in 2017, in collaboration with scientists, conducted an experiment: installed cameras in a cinema and connected deep learning algorithms to assess the emotions of viewers. The system could predict people's reactions by observing them for just a few minutes. During the experiment, we collected an impressive dataset: 68 markers from each of 3,179 viewers. In total, 16 million face images were obtained.
For the same purpose, YouTube video hosting has created its own AI called YouFirst. It allows video bloggers and businesses to test content prior to release to the platform. Users click on a special link, agree to shoot a video and watch the video. At this time, the neural network determines their reactions and sends the data to the channel owner.
Among Russian companies, reactions to videos can be analyzed, for example, by Neurobotics. The company has developed the EmoDetect program that recognizes joy, sadness, surprise, fear, anger, disgust and neutrality. The program studies up to 20 local facial features in freeze frames and a series of images. The system analyzes motor units and uses FACS facial coding technology. It is possible to record video from a webcam. The EmoDetect API allows you to integrate the product with external applications.
Emotional AI is also beginning to be applied in the gaming industry. It helps to personalize the game and add more interaction with the gamer.
For example, the American emotional AI company Affectiva helped create the psychological thriller Nevermind. The tension depends on the state of the player: the plot becomes darker when he is under stress, and vice versa.
Education
Emotion recognition also applies to education. It can be used to study the mood and attention of students during class.
Russian developers have applied emotional AI in Perm. The impetus for the development of technology was the attacks of students on elementary school students and the teacher. Rostelecom and the startup New Vision have developed the Smart and Safe School program to monitor the emotional state of children. This will help identify asocial adolescents before tragedy occurs.
It was based on the Paul Ekman system. The neural network analyzed the slightest muscle movements using 150 points on the face. A large amount of data was collected during the lesson: 5-6 thousand frames for each student. The program studied the dataset and calculated the emotional state of each child. According to the creators, the accuracy was 72%.
HR
Emotional AI can be useful in work with staff. It helps to determine the state of the employee, to notice his fatigue or dissatisfaction in time, and to redistribute tasks more efficiently.
In addition, technology helps with recruiting. With the help of emotional AI, you can check a candidate for a job or catch a lie during an interview.
The American company HireVue uses artificial intelligence to evaluate candidates. The applicant goes through a video interview, and the neural network determines his condition by keywords, voice intonation, movements and facial expressions. The AI highlights the characteristics that are important for the job and gives marks, and the HR manager selects the right candidates.
London-based startup Human uses video to identify emotions and match them to character traits. After the video interview, recruiters receive a report that says how honest, curious, excited, enthusiastic, or confident the candidate was and how he responded to questions.
Medicine
In this area, not only non-contact, but also contact methods of determining emotions will be useful. They are being actively implemented by foreign startups - for example, Affectiva and Brain Power. The companies' developments include AI glasses that help children and adults with autism recognize other people's emotions and develop social skills.
But neural networks can help patients without wearable sensors. Scientists at the Massachusetts Institute of Technology have created a neural network that detects depression by analyzing a person's speech. The accuracy of the result was 77%. And startup Beyond Verbal is using AI to analyze the mental health of patients. In this case, the neural network selects only voice biomarkers from the audio recording.
Cars
Massachusetts Institute of Technology is developing an AI called AutoEmotive that will determine the condition of the driver and passengers. He will not only monitor the level of stress, but also try to reduce it - by playing soft music, adjusting the temperature in the cabin or taking a less busy route.
Limitations of emotional AI
The neural network cannot take into account the context
AI has learned to identify basic human emotions and states, but so far it does not cope well with more complex situations. Scientists note that facial expressions do not always accurately show how a person really feels. His smile can be feigned or sarcastic, and this can only be determined by context.
NtechLab experts believe that it is still difficult to accurately determine the reason for this or that emotion.
NtechLab emphasizes that it is necessary to recognize not only facial expressions, but also human movements. Diverse data will make emotional AI much more efficient. Daniil Kireev, a leading researcher at the VisionLabs face recognition product development company, agrees with this. In his opinion, with a large amount of data, the accuracy of the algorithms increases.
“There are errors, their number depends on many factors: the quality of the training sample, the trained neural network, the data on which the final system works. By adding information from different sources - for example, voice - you can improve the quality of the system. At the same time, it is important to understand that by the face we rather determine its expression than the final emotion. The algorithm may try to determine the simulated emotion, but for this, the development of technology must take a small step forward,”says Daniil Kireev.
Bad equipment
External factors influence the quality of the algorithms. For the accuracy of emotion recognition to be high, video cameras and microphones must be of high quality. In addition, the result is influenced by lighting, the location of the camera. According to Daniil Kireev, uncontrolled conditions complicate the process of determining a person's states.
For emotional AI to develop, you need quality hardware. If you find good equipment and set it up correctly, the accuracy of the results will be very high. And when it becomes more accessible and widespread, emotion recognition technologies will be improved and implemented more actively.
“The accuracy of the system depends on many factors. The main one is the quality of still frames from the camera, which are given to the system for recognition. The quality of still frames, in turn, is influenced by the settings and characteristics of the camera, the matrix, lighting, the location of the device, the number of faces in the frame. With the correct configuration of the hardware and software, it is possible to achieve the accuracy of the detected emotion up to 90-95%,”notes Vitaly Vinogradov, product manager of the cloud video surveillance and video analytics service Ivideon.
Technology Perspective
Now in Russia, emotional AI is only gaining momentum. Startups develop technology and market their products, and customers test them with caution.
But Gartner estimates that by 2024, more than half of online ads will be made using emotional AI. Computer vision, which is used to detect emotions, will become one of the most important technologies in the next 3-5 years. And MarketsandMarkets predicts that the emotion analysis market will double by 2024 - from $ 2.2 billion to $ 4.6 billion.
In addition, large companies are showing interest in emotion recognition - for example, Procter & Gamble, Walmart, VTB, Rosbank, Sberbank and Alfa-Bank. And domestic startups are developing pilot projects that will become ready-made solutions for business in the future.
Evgeniya Khrisanfova
