The Strange 'McGurk' Effect: How Your Eyes Can Affect What You Hear

It's pretty easy to spot a badly dubbed foreign film: The sounds that you hear coming out of the actors' mouths don't seem to match up with the movements of their lips that you see.

In other words, even when our vision and hearing are being stimulated at the same time during the film, our brains do a really good job of picking up on which lip movements go  with which speech sounds.

But the brain can also be fooled. In an intriguing illusion known as the McGurk effect, watching the movements of a person's lips can trick the brain into hearing the wrong sound. [10 Things You Didn't Know About the Brain]

Latest Videos From

Watch full video here:

The McGurk effect occurs when there is a conflict between visual speech, meaning the movements of someone's mouth and lips, and auditory speech, which are the sounds a person hears. And it can result in the perception of an entirely different message.                                              

Now, in a new study, neuroscientists at the Baylor College of Medicine in Houston attempted to offer a quantitative explanation for why the McGurk effect occurs. They developed a computer model that was able to accurately predict when the McGurk effect should or should not occur in people, according to the findings, published (Feb. 16) in the journal PLOS Computational Biology. (Here is one demonstration, and another; neither of these examples were the actual video used in the study.)

In the demonstration of the McGurk effect used in the study, the participant is asked to keep his or her eyes closed while listening to a video that shows a person making the sounds "ba ba ba." Then that individual is asked to open their eyes and watch the mouth of the person in the video closely, but with the sound off. Now, the visuals look like the person is saying "ga ga ga." In the final step of the experiment, the exact same video is replayed, but this time the sound is on, and the participant is asked to keep his or her eyes open. People who are sensitive to the McGurk effect will report hearing "da da da" — a sound that doesn't match up with either the auditory or visual cues previously seen.

That's because the brain is attempting to resolve what it thinks it's hearing with a sound closer to what it visually sees. If the person closes their eyes again, and the video's sound is replayed, he or she will once again hear the original sound of "ba ba ba."

The effect was first described in an experiment done in 1976 by psychologists Harry McGurk and John MacDonald, which showed that visual information provided by mouth movements can influence and override what a person thinks he or she is hearing.

Predicting an illusion

The McGurk effect is a powerful, multisensory illusion, said study co-author John Magnotti, a postdoctoral fellow in the department of neurosurgery at Baylor.  "The brain is taking auditory speech and visual speech and putting them together to form something new," he said. [6 Foods That Are Good For Your Brain]

When people are having a face-to-face conversation, the brain is engaged in complicated activity as it tries to decide how to put lip movements together with the speech sounds that are heard, Magnotti said.

In the study, the researchers tried to understand why the brain was better able to put some syllables together to interpret the sound heard correctly but not others, Magnotti said.

To do this, their model relied on an idea known as causal inference, or a process in which a person's brain decides whether the auditory and visual speech sounds were produced by the same source. What this means is that the sounds come from one person talking, or from multiple speakers, so you are hearing one person's voice, but looking at another person who is also talking, at the same time.

Other researchers have developed models to help predict when the McGurk effect may occur, but this new study is the first one to include causal inference in its calculation, Magnotti told Live Science. Factoring in causal inference may have improved the new model's accuracy, compared with previous prediction models of the illusion.

To test the accuracy of their prediction model, the researchers recruited 60 people and asked them to listen to pairs of auditory and visual speech from a single speaker. Then the participants were asked to decide if they thought they heard the sound "ba," "da" or "ga."

Their results showed that the model they developed could reliably predict when the majority of participants involved in the experiment would experience the McGurk effect. But as expected from their calculation, there were also some people who were not susceptible to it, Magnotti said. [Eye Tricks: Gallery of Visual Illusions]

Interestingly, Magnotti said that when this same test has been done with students in China rather than people in the United States, the McGurk effect has been shown to work in other languages.

Magnotti said that he thinks the computer models developed for this study may also have some practical uses. For example, the model could be helpful to companies that build computers that assist in speech recognition, such as a product like Google Home or Amazon Echo, he said.

If these smart speakers had cameras on them, they could integrate people's lip movement into what a person was saying to increase the accuracy of their speech-recognition systems, he said.

The model may also help children with cochlear implants, by improving researchers' understanding of how visual speech affects what a person hears, Magnotti said.

Originally published on Live Science.