Eye Movements in Silent Visual Speech track Unheard Acoustic Signals and Relate to Hearing Experience

Description

Behavioral and neuroscientific studies have shown that watching the movements of a speaker's lips aids speech comprehension. Intriguingly, even when videos of speakers are silently presented, various cortical regions have been shown to track auditory features, such as the envelope. However, the cause and function of silent neural speech tracking have remained elusive. Recently, we have shown that when attentively listening to speech, eye movements track low-level acoustic information. If also present during processing of silent videos of speakers, eye movements could offer a parsimonious explanation for cortical speech tracking. Yet, whether ocular speech tracking can also be observed in silent visual speech and how this influences cortical speech tracking is unknown. Here, not only hearing but also congenitally and acquired deaf individuals were investigated to test how auditory deprivation (early vs. late onset) affects neural and ocular speech tracking in silent lip-reading. In this magnetoencephalography (MEG) study, we investigated the electrooculogram (EOG) and neural speech tracking of 73 participants observing silent videos of a speaker played in a forward and backward manner. Our main finding is a clear ocular speech tracking effect to the unheard auditory envelope with a dominance <1 Hz, that was not present for the lip movements. Similarly, we could show a <1 Hz effect of unheard auditory envelope tracking in temporal regions for hearing participants. Importantly, in this study neural tracking is not directly linked to ocular tracking. Strikingly, across different listening groups, deaf participants with auditory listening experience show higher ocular speech tracking than hearing participants, while for congenital deaf participants in a very small sample, no ocular speech tracking effect was revealed. This study extends our previous work in showing the involvement of eye movements in speech processing even in absence of acoustic input. While underscoring the role of eye movements in speech processing, this study simultaneously raises numerous questions for future research, especially regarding their functional relevance.

Period	11 Jul 2024
Event title	Salzburg Mind-Brain Annual Meeting (SAMBA) 2024
Event type	Conference
Location	Salzburg, AustriaShow on map
Degree of Recognition	International