Using technology to scan brain activity, artificial intelligence and a speech synthesizer, scientists from Columbia University (USA) have created a device capable of translating human thoughts into articulate speech. The research findings, published in Scientific Reports, represent an important step in improving brain-computer interfaces. In the future, such devices can be used by people who have lost the ability to speak as a result of injury or illness.
To develop a device that combines the capabilities of a speech synthesizer and artificial intelligence, the study author, neuroscientist Nima Mesgarani and his colleagues turned to the latest advances in deep machine learning and speech synthesis technologies. The result of their work is an artificial intelligence-based vocoder capable of quite accurately interpreting brain activity directly from the auditory cortex and then translating it into intelligible speech. The authors of the work note that speech in this case turns out to be very computerized, but people can recognize words in most cases.
According to the creators, the new device used to reproduce the resulting speech is based on the same technology used in digital assistants such as Alexa, Siri and Google Assistant.
First, experts trained the vocoder to correctly interpret human brain activity. For this, the scientists invited five volunteers who were undergoing treatment for epilepsy to participate in the experiment. All five of them had electrodes implanted in the auditory cortex to read the electroencephalogram.
“We asked epileptic patients who are already undergoing brain surgery to listen to sentences that are being said by different people. At the same time, we analyzed patterns in the patients' brain activity. The vocoder was trained on the neural models obtained,”explains Mesgarani.
The patients were asked to listen to the recordings in which the actors read out sequences of numbers from 0 to 9. At the same time, the scientists recorded the signals from the brain, and then passed them through the vocoder, the signals for which were corrected to improve clarity by a neural network that analyzed the sounds of the vocoder itself, which produced sounds in response to these signals. As a result, a robotic voice could be heard repeating a sequence of spoken numbers. To evaluate the results, the scientists invited 11 people with excellent hearing.
“It turned out that people can recognize words about 75% of the time, which is vastly superior to any previous attempts. The sensitive vocoder and powerful neural networks generated the sounds that the patients listened to with amazing accuracy,”comments Mesgarani.
Promotional video:
In the future, Mesgarani's team is going to teach the neural network to pronounce more complex words, phrases and even whole sentences. After that, they want to develop an implant that can translate a person's thoughts into full speech.
“For example, if the owner of the implant thinks:“I need a glass of water,”our system reads the brain signals and translates them into speech. This will give anyone who has lost the ability to speak due to injury or illness a new opportunity to communicate with the world around them,”adds Mesgarani.
Nikolay Khizhnyak