“I told my daughter that the TV in her living room had problems with syncing the picture and sound. And then I noticed that the TV in the kitchen also had bad dubbing. And suddenly I realized that her own voice also sounds out of sync with articulation. The problem is not in TVs, but in me!"
Have you ever watched an old movie where the sound was out of sync with what was happening on the screen? Now imagine that every voice you hear also sounds out of order - even your own. In such a world lives a person who asked to call him PH. Soon after undergoing heart surgery, he began to notice that something was wrong in the world around him, something was wrong.
“I was visiting my daughter, and they like the TV to be on all the time. I turn to my daughter and, smiling, say: “You would need to buy a normal TV, one where the sound and the program are synchronized. And they answer: "The TV is all right."
Puzzled, he headed to the kitchen for a cup of tea. “They have another TV on the wall, and they had the same problem with it. I went to the living room and said to my daughter: "Hey, you have two TVs here that need fixing!"
And at that moment, he began to notice that his daughter's speech also did not coincide with the movements of her lips. “It wasn't about the TV, it was about me. It happened in real life."
PH is the first officially confirmed case where a person hears people's speech before noticing their lip movements. His situation provides a unique opportunity to understand how our brains unite the audible and the visible.
It is not known why PH problems began at that moment, but this could be due to acute pericarditis - an inflammation of the serous membrane of the heart, or to the surgical intervention that had to be used to treat it.
After problems with temporal synchronization appeared, a brain scan was performed, showing two lesions in areas of the brain believed to be responsible for hearing, perception of time and movement. "Where this damage came from is anyone's guess," says PH. "Maybe I have had them all my life, or they appeared as a result of intensive therapy."
Promotional video:
Disturbing audio delay
A few weeks later, PH realized that it was not only the people around him that sounded asynchronous: during the conversation, he heard his own words before he felt his jaw begin to move. “The delay seemed quite significant, and this whole situation took me by surprise. All this worried me very much. I didn't know then if the gap between sound and picture would increase, but now it seems to have stopped for a quarter of a second."
Light and sound move at different speeds; accordingly, when someone speaks, visual and auditory data reaches our eyes and ears at different times. These signals are then processed by the brain at different levels. Despite all this, we usually perceive actions as happening simultaneously, but it is still unknown how the brain achieves this effect.
To study the PH situation, Elliot Freeman of the City University of London and colleagues presented a temporal sequence judgment test. PH showed short videos of people speaking and then asked if there was sound in the video before or after the speaker's lips moved. Understandably, PH's response was that the sound came before the lips moved, and in order to achieve synchronized perception, the team had to play the sound two hundred milliseconds later than the lips began to move. The team then ran a second, more objective test based on the McGurk illusion. The essence of this illusion is as follows: a person, hearing the first syllable, observes someone pronouncing the second syllable; this combination makes him perceive the third syllable.
Since PH hears people speak before seeing their lip movements, the team expected the illusion to trigger when they delayed the sound. Therefore, they were surprised to get the opposite result: the broadcast of the sound two hundred milliseconds before the movement of the lips triggered the illusion mechanism, but this time the subject's brain processed the image before the sound was perceived.
And not only PH showed such results. When 37 other people passed both tests, many showed the same trend, although no such inconsistencies were noticed in their daily lives.
Many hours
According to Freeman, this means that the same event from the outside world is perceived by different parts of our brain as happening in different periods of time. This suggests that in addition to one holistic "now", there are many clocks in our brain - two of them were demonstrated during tests - and that all these clocks show their individual "now" relative to the general time indicator.
In the case of PH, one or more of these clocks have been significantly slowed down - altering its overall perception of time - possibly as a result of damage to the brain. According to Freeman, the temporal discrepancies in PH may be too significant and could start too unexpectedly, not allowing him to adapt to them, or simply not noticing them, which led to the realization of PH asynchrony in everyday life. As Freeman says, PH may only perceive one of his watches, since they are the only ones that this man has conscious access to.
According to PH, in general he has already learned to live with this sensory mismatch, but admits that he has difficulty in noisy places or in crowded meetings. If he hears his speech before he feels the movements of his mouth, has he ever felt that his own voice is not under his control? “No, I'm pretty sure I'm speaking on my own,” he says. It's just a rather strange feeling."
PH admits that he would gladly be cured, but he won't be too worried if the doctors do not work out any time soon. “It's not life threatening,” he says. As you get older, you get used to living with these things. I do not expect perfect functioning from my body."