Engineers from the Institute for the Study of Higher Nervous Activity have been able to create a system that translates thoughts into intelligible and recognizable speech. By controlling someone's brain activity, technology reconstructs thoughts into words.
This scientific breakthrough, coupled with the power of AI and speech synthesizers, opens a new era in the interaction between the computer and the brain. Of course, this also opens up new opportunities for people who, for one reason or another, have lost the ability to speak.
“Our voices help keep in touch with our friends, family, and the world around us, so losing voice power due to injury or illness is devastating for people. However, with today's stock of knowledge, we have a potential way to restore this power. We have shown that with the right technology, people's thoughts can be deciphered and understood by any listener,”says Nima Mesgarani, Ph. D. and one of the authors of a study conducted at the Mortimer B. Zuckerman Institute of Mental Behavior at Columbia University.
Decades of research have shown that when people speak or even imagine they are speaking words, characteristic patterns of activity appear in their brains. A clear and recognizable signal pattern also arises when we listen to someone speaking, or imagine that we are listening. Experts have tried to decipher these patterns for decades, but only now a future has opened before them, in which thoughts can no longer be hidden inside the brain, but instead can be translated into spoken language at will.
But this feat was not easy to accomplish. Early attempts to decipher brain signals from Dr. Mesgarani and others focused on simple computer models that analyzed spectrograms, which are visual representations of sound frequencies.
But due to the fact that this approach did not produce anything close to intelligible speech, Dr. Mesgarani's team instead turned to a vocoder, a computer algorithm that can synthesize speech after being trained to record people's conversations.
“This is the same technology that Amazon Echo and Apple Siri use to verbally answer our questions,” said Dr. Mesgarani, who is also an associate professor of electrical engineering at the Fu Foundation's Columbia School of Engineering and Applied Science.
To teach the vocoder to interpret brain activity, Dr. Mesgarani teamed up with Ashesh Dinesh Mehta, M. D., Ph. D., neurosurgeon at Northwell Health Physician Partners' Neuroscience Institute and co-author of today's article. Dr. Mehta treats patients with epilepsy, some of whom must undergo regular surgery.
Promotional video:
“Working with Dr. Mehta, we asked epileptic patients who had already had brain surgery to listen to suggestions from different people, while we measured patterns in their brain activity,” says Dr. Mesgarani. "These neural patterns trained the vocoder."
The researchers then asked those same patients to listen to speakers pronouncing the numbers 0 through 9 while recording brain signals that could then be passed through a vocoder. The sound produced by the vocoder in response to these signals was analyzed and refined using neural networks such as artificial intelligence that mimic the structure of neurons in a biological brain.
The end result was a robotic voice repeating a sequence of numbers. To verify the accuracy of the recording, Dr. Mesgarani and her team instructed people to listen to the recording and report what they heard.
“We found that humans can understand and repeat sounds about 75% of the time, which is far superior to any previous attempts,” says Dr. Mesgarani. The improvement in intelligibility was particularly evident when comparing the new recordings with earlier attempts based on the spectrogram. "A sensitive vocoder and powerful neural networks represented the sounds that the patients initially heard with amazing accuracy."
Dr. Mesgarani and her team now plan to test more complex words and sentences. They also intend to perform the same tests on the brain signals emitted when a person speaks or imagines speech. Ultimately, they hope that their system can be part of an implant, similar to those worn by some epileptic patients, which translate the wearer's thoughts directly into words.
“In this scenario, if the owner of the chip thinks, 'I need a glass of water,' our system can take in the brain signals generated by that thought and convert them into synthesized verbal speech,” says Dr. Mesgarani. "It's a game changer and for anyone who has lost the ability to speak due to injury or illness, technology gives a new chance to communicate with the world around them."
Editorial comment
Since some of our employees have a certain relationship to neurophysiology, we can absolutely unequivocally state that solving the problem of reading thoughts and translating these thoughts into words is not a problem that some doctor of philosophy can solve together with an intelligent neurosurgeon. This is a task for the research institute, which will solve it for a hundred, two hundred or more years. Moreover, it is not at all a fact that the research institute will solve this problem - even if all NASA supercomputers are brought there, on which a crowd of engineers will begin to simulate neural networks. However, an article in a scientific journal will not lie and the fact of recognition of thoughts is definitely there. How, then, can these two facts be brought together?
Very simple. In the last 20-30 years, a huge number of completely incredible and very complex technologies like microprocessors and hard drives have appeared in the world. And every day more and more new inventions appear, which are written down to some "talented students" who, sitting in the garage, have collected a piece from scrap metal, over which research institutes have been fighting for 50 years. And in this case, we have an invention of a similar genesis. That is, a development that has been done (or handed over to people) for many years, it is not clear who, but which was recorded on the aunt and uncle from Colombia.
In fact, there is absolutely no difference at all who Wikipedia will write down as “inventors” of technology. The main thing is that the technology was presented to the world using a scientific journal. Further, some Chinese company will start producing such "gadgets" that will accustom people to the very idea that their head has now become completely transparent. Finally, the third stage will be the same Orwellian “1984” when even the thoughts of any person can be freely read from a distance. For example, by placing special sensors in apartments or integrating a special chip that responds to brain impulses into the body. Furthermore.
Each technology has, as a rule, the opposite application. For example, if there is a technology for converting nerve impulses into words, then using the same principle, you can turn any words into someone's thoughts. In this situation, the presidents of different countries have a grandiose prospect of being supported by absolutely the entire electorate, since now there is a technical possibility to make every voter even think like him - it is enough to put a correctly modulated signal into the socket.
In general, it goes without saying that the world's most wonderful prospects are opening up and we are following the development of events with interest.