Neural speech tracking contribution of lip movements predicts behavioral deterioration when the speaker's mouth is occluded

eNeuro, Journal Year: 2025, Volume and Issue: unknown, P. ENEURO.0368 - 24.2024

Published: Jan. 16, 2025

Observing lip movements of a speaker facilitates speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows stronger neural responses to audiovisual stimuli compared audio-only stimuli. However, the interindividual variability this contribution movement information and its consequences on behavior are unknown. We analyzed source-localized magnetoencephalographic (MEG) 29 normal-hearing participants (12 female) speech, both with without wearing surgical face mask, presence or absence distractor speaker. Using temporal response functions (TRFs) quantify tracking, we show that are, general, enhanced when is challenging. After controlling for acoustics, contribute particularly present. extent visual tracking varied greatly among participants. Probing behavioral relevance, demonstrate individuals who higher terms drop comprehension an increase perceived difficulty mouth occluded by mask. By contrast, no effect was found not occluded. provide novel insights how varies revealing negative absent. Our results also offer potential implications objective assessments perception. Significance Statement In complex auditory environments, simultaneous conversations pose challenge comprehension. investigated level, aid such situations what observing enhances rely more deterioration wears Remarkably, case mask worn findings reveal differences applications

Language: Английский

---

Atypical audio-visual neural synchrony and speech processing in early autism

Journal of Neurodevelopmental Disorders, Journal Year: 2025, Volume and Issue: 17(1)

Published: Feb. 18, 2025

Abstract Background Children with Autism Spectrum disorder (ASD) often exhibit communication difficulties that may stem from basic auditory temporal integration impairment but also be aggravated by an audio-visual deficit, resulting in a lack of interest face-to-face communication. This study addresses whether speech processing anomalies young autistic children (mean age 3.09-year-old) are associated alterations integration. Methods We used high-density electroencephalography (HD-EEG) and eye tracking to record brain activity gaze patterns 31 ASD (6 females) 33 typically developing (TD) (11 females), while they watched cartoon videos. Neural responses stimuli were analyzed using Temporal Response Functions model phase analyses for audiovisual coordination. Results The reconstructability signals was reduced compared TD, despite more restricted it similar visual both groups. Speech reception most strongly affected when information present, interference not seen TD children. These differences broader angle distribution (exceeding pi/2) the EEG theta range ASD, signaling reliability alignment. Conclusion findings show do stand alone already at very early development stage imbalance poor response encoding disrupted

Language: Английский

---

Influence of Visual Analogue of Speech Envelope, Formants, and Word Onsets on Word Recognition is not pronounced

Hearing Research, Journal Year: 2025, Volume and Issue: 460, P. 109237 - 109237

Published: March 7, 2025

In noisy environments, filtering out the relevant speech signal from background noise is a major challenge. Visual cues, such as lip movements, can improve understanding. This suggests that movements carry information about features (e.g. envelope, formants, word onsets) be used to aid Moreover, isolated visual or tactile presentation of envelope also recognition. However, evidence in this area rather mixed, and formants onsets have not been studied context. online study investigates effect different visually presented (speech during two-talker audio on The were circle whose size was modulated over time based dynamics three features. either according target speaker, distractor speaker an unrelated control sentence. After each sentence, participants` recognition tested by writing down what they heard. We show enhanced for any relative condition.

Language: Английский

---

Early Word Segmentation Behind the Mask

Frontiers in Psychology, Journal Year: 2022, Volume and Issue: 13

Published: May 9, 2022

Infants have been shown to rely both on auditory and visual cues when processing speech. We investigated the impact of COVID-related changes, in particular face masks, early word segmentation abilities. Following up our previous study demonstrating that, by 4 months, infants already segmented targets presented auditorily at utterance-edge position, and, using same familiarization paradigm, 7–9-month-old performed an audiovisual experiment two conditions: without with FFP2 mask. Analysis acoustic showed changes face-masked speech affecting amount, weight, location cues. Utterance-edge position displayed more salient than utterance-medial but were attenuated Results revealed no evidence for segmentation, not even edge regardless mask condition or presentation. However, experiment, attended screen during test trials familiarized Also, mouth less eyes In addition, advantage emerging abilities was found. Thus, information provided some support developing segmentation. compared 7–9-monthers ability observed Butler Frota pre-COVID current data. Mean looking time significantly higher unfamiliar only. Measures cognitive language development obtained CSBS scales that scored lower same-age from (pre-COVID) normative Our results suggest overall effect pandemic development, calling longitudinal studies determine how proceeds.

Language: Английский

---

Mask wearing in Japanese and French nursery schools: The perceived impact of masks on communication

Frontiers in Psychology, Journal Year: 2022, Volume and Issue: 13

Published: Nov. 7, 2022

Due to the global COVID-19 pandemic, covering mouth region with a face mask became pervasive in many regions of world, potentially impacting how people communicate and around children. To explore characteristics this masked communication, we asked nursery school educators, who have been at forefront daily interaction children, about their perception communicative interactions while wearing an online survey. We collected data from French Japanese educators gain understanding commonalities differences behavior masks given documented cultural pre-pandemic habits, scanning patterns, behavior. Participants (177 138 educators) reported perceived change own mask, decreases language quantity increases quality non-verbal cues. Comparable changes team members’ children’s behaviors were also reported. Moreover, our results suggest that these educators’ are linked attitudes toward potential difficulty communicating following its use. These findings shed light on impact pandemic-induced environment.

Language: Английский

---

Neural Speech Tracking Highlights the Importance of Visual Speech in Multi-speaker Situations

Journal of Cognitive Neuroscience, Journal Year: 2023, Volume and Issue: 36(1), P. 128 - 142

Published: Nov. 17, 2023

Visual speech plays a powerful role in facilitating auditory processing and has been publicly noticed topic with the wide usage of face masks during COVID-19 pandemic. In previous magnetoencephalography study, we showed that occluding mouth area significantly impairs neural tracking. To rule out possibility this deterioration is because degraded sound quality, present follow-up presented participants audiovisual (AV) audio-only (A) speech. We further independently manipulated trials by adding mask distractor speaker. Our results clearly show only affect tracking AV conditions, not A conditions. This shows indeed primarily impact blocking visual acoustic degradation. can highlight how spectrogram, lip movements lexical units are tracked on sensor level. benefits for spectrogram especially multi-speaker condition. While additional improvement benefit over clear (phonemes word onsets) do enhancement at all. hypothesize young normal hearing individuals, information from input less used specific feature extraction, but acts more as general resource guiding attention.

Language: Английский

---

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

PLoS Computational Biology, Journal Year: 2024, Volume and Issue: 20(9), P. e1012433 - e1012433

Published: Sept. 9, 2024

Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory visual cues to understand one another. For example, objects may make sounds as they move space, or we movement of person’s mouth better what are saying noisy environment. Still, many neuroscience experiments rely on unimodal stimuli encoding sensory features brain. The extent which information influence vice versa natural environments thus unclear. Here, addressed this question by recording scalp electroencephalography (EEG) 11 subjects listened watched movie trailers audiovisual (AV), (V) only, audio (A) only conditions. We then fit linear models that described relationship between brain responses acoustic, phonetic, stimuli. also compared whether feature tuning was same when were presented original AV format versus removed. In these stimuli, relatively uncorrelated, included spoken narration over scene well animated live-action characters talking with without their face visible. stimulus, found similar A-only conditions, similarly, for present (AV) removed (V only). cross prediction analysis, investigated trained data predicted A V test similarly data. Overall, performance using training sets sets, suggesting has smaller effect EEG. contrast, set slightly worse than matching sets. This suggests stronger EEG, though makes no qualitative difference derived tuning. effect, our results show researchers benefit from richness multimodal datasets, can be used answer more research question.

Language: Английский

---

Neural speech tracking benefit of lip movements predicts behavioral deterioration when the speaker’s mouth is occluded

bioRxiv (Cold Spring Harbor Laboratory), Journal Year: 2023, Volume and Issue: unknown

Published: April 18, 2023

Abstract Observing lip movements of a speaker is known to facilitate speech understanding, especially in challenging listening situations. Converging evidence from neuroscientific studies shows enhanced processing audiovisual stimuli. However, the interindividual variability this visual benefit and its consequences on behavior are unknown. Here, we analyzed source-localized magnetoencephalographic (MEG) responses normal-hearing participants with or without an additional distractor speaker. Using temporal response functions (TRFs), show that neural are, general, when challenging. After conducting crucial control for acoustics, effectively contribute higher tracking, particularly present. extent varied greatly among participants. Probing behavioral relevance, individuals who more movement information terms stronger drop performance increase perceived difficulty mouth occluded by surgical face mask. By contrast, no effect was found not occluded. We provide novel insights how tracking varies individuals. Furthermore, reveal relevance demonstrating negative absent. Our results also offer potential implications future objective assessments perception.

Language: Английский

---

Attention to audiovisual speech shapes neural processing through feedback-feedforward loops between different nodes of the speech network

PLoS Biology, Journal Year: 2024, Volume and Issue: 22(3), P. e3002534 - e3002534

Published: March 11, 2024

Selective attention-related top-down modulation plays a significant role in separating relevant speech from irrelevant background when vocal attributes concurrent speakers are small and continuously evolving. Electrophysiological studies have shown that such enhances neural tracking of attended speech. Yet, the specific cortical regions involved remain unclear due to limited spatial resolution most electrophysiological techniques. To overcome limitations, we collected both electroencephalography (EEG) (high temporal resolution) functional magnetic resonance imaging (fMRI) resolution), while human participants selectively audiovisual scenes containing overlapping cocktail party utilise advantages respective techniques, analysed using EEG data performed representational dissimilarity-based EEG-fMRI fusion. We observed attention enhanced modulated correlates throughout latencies studied. Further, enhancement fluctuated predictable profiles. discuss how dynamics could arise combination interactions between prediction as well plastic properties auditory cortex. fusion revealed iterative feedforward-feedback loops hierarchically organised nodes ventral object related processing stream. Our findings support models where facilitates dynamic changes cortex, ultimately aiding discrimination sounds ones conserving resources.

Language: Английский

---

Atypical Audio-Visual Neural Synchrony and Speech Processing in children with Autism Spectrum Disorder

bioRxiv (Cold Spring Harbor Laboratory), Journal Year: 2024, Volume and Issue: unknown

Published: April 19, 2024

Background: Children with Autism Spectrum Disorders (ASD) often exhibit communication difficulties that may stem from basic auditory temporal integration impairment but also be aggravated by an audio-visual deficit, resulting in a lack of interest face-to-face communication. This study addresses whether speech processing anomalies young (mean age 3.09-year-old) children ASD are associated alterations integration. Methods: We used high-density electroencephalography (HD-EEG) and eye tracking to record brain activity gaze patterns 31 (6 females) 33 typically developing (TD) (11 females), while they watched cartoon videos. Neural responses stimuli were analyzed using Temporal Response Functions model phase analyses for audiovisual coordination. Results: The reconstructability signals was reduced compared controls, despite more restricted it similar visual both groups. Speech reception most strongly affected when information present, interference not seen TD children. These differences broader angle distribution (exceeding pi/2) the EEG theta range autistic children, signaling reliability alignment. Conclusion: findings show do stand alone already at very early development stage imbalance lousier response encoding disrupted

Language: Английский

---