Modality-Specific Perceptual Learning of Vocoded Auditory versus Lipread Speech: Different Effects of Prior Information DOI Creative Commons
Lynne E. Bernstein, Edward T. Auer, Silvio P. Eberhardt

et al.

Brain Sciences, Journal Year: 2023, Volume and Issue: 13(7), P. 1008 - 1008

Published: June 29, 2023

Traditionally, speech perception training paradigms have not adequately taken into account the possibility that there may be modality-specific requirements for perceptual learning with auditory-only (AO) versus visual-only (VO) stimuli. The study reported here investigated hypothesis are differences in how prior information is used by normal-hearing participants during vocoded VO training. Two different experiments, one AO (Experiment 1) and VO, lipread, 2), effects of giving types

Language: Английский

The concept of representation in the brain sciences: The current status and ways forward DOI Creative Commons
Luis H. Favela, Édouard Machery

Mind & Language, Journal Year: 2025, Volume and Issue: unknown

Published: Feb. 26, 2025

This article outlines the motivations and main findings of Favela Machery's “Investigating concept representation in neural psychological sciences”, discusses what to do with brain sciences moving forward.

Language: Английский

Citations

1
Lipreading: A Review of Its Continuing Importance for Speech Recognition With an Acquired Hearing Loss and Possibilities for Effective Training DOI Creative Commons
Lynne E. Bernstein,

Nicole Jordan,

Edward T. Auer

et al.

American Journal of Audiology, Journal Year: 2022, Volume and Issue: 31(2), P. 453 - 469

Published: March 22, 2022

The goal of this review article is to reinvigorate interest in lipreading and training for adults with acquired hearing loss. Most benefit from being able see the talker when speech degraded; however, effect size related their ability, which typically poor who have experienced normal through most lives. Lipreading has been viewed as a possible avenue rehabilitation an loss, but approaches not particularly successful. Here, we describe theoretically motivated its training, well examples successful paradigms. We discuss some extensions auditory-only (AO) audiovisual (AV) recognition.

Language: Английский

Citations

20
Auditory cortex encodes lipreading information through spatially distributed activity DOI

Ganesan Karthik,

Cody Zhewei Cao, Michael I. Demidenko

et al.

Current Biology, Journal Year: 2024, Volume and Issue: 34(17), P. 4021 - 4032.e5

Published: Aug. 16, 2024

Language: Английский

Citations

4
Influence of Visual Analogue of Speech Envelope, Formants, and Word Onsets on Word Recognition is not pronounced DOI Creative Commons
Kaja Rosa Benz, Anne Hauswald, Nathan Weisz

et al.

Hearing Research, Journal Year: 2025, Volume and Issue: 460, P. 109237 - 109237

Published: March 7, 2025

In noisy environments, filtering out the relevant speech signal from background noise is a major challenge. Visual cues, such as lip movements, can improve understanding. This suggests that movements carry information about features (e.g. envelope, formants, word onsets) be used to aid Moreover, isolated visual or tactile presentation of envelope also recognition. However, evidence in this area rather mixed, and formants onsets have not been studied context. online study investigates effect different visually presented (speech during two-talker audio on The were circle whose size was modulated over time based dynamics three features. either according target speaker, distractor speaker an unrelated control sentence. After each sentence, participants` recognition tested by writing down what they heard. We show enhanced for any relative condition.

Language: Английский

Citations

0
Dynamic modeling of EEG responses to natural speech reveals earlier processing of predictable words DOI Creative Commons
Jin Dou, Andrew J. Anderson, Aaron Steven White

et al.

PLoS Computational Biology, Journal Year: 2025, Volume and Issue: 21(4), P. e1013006 - e1013006

Published: April 28, 2025

In recent years, it has become clear that EEG indexes the comprehension of natural, narrative speech. One particularly compelling demonstration this fact can be seen by regressing responses to speech against measures how individual words in linguistically relate their preceding context. This approach produces a so-called temporal response function displays centro-parietal negativity reminiscent classic N400 component event-related potential. shortcoming previous implementations is they have typically assumed linear, time-invariant relationship between linguistic features and responses. other words, analysis assumes same shape timing for every word – only varies (linearly) terms its amplitude. present work, we relax assumption under hypothesis may processed more rapidly when are predictable. Specifically, introduce framework wherein standard linear modulated amplitude, latency, scale based on predictability current prior words. We use proposed model recorded from set participants who listened an audiobook narrated single talker, separate attended one two concurrently presented audiobooks. show expected faster evoking lower amplitude N400-like with earlier peaks effect driven both word’s own immediately word. Additional suggests finding not simply explained quickly disambiguated phonetic neighbors. As such, our study demonstrates brain natural depend predictability. By accounting these effects, also improves accuracy which neural modeled.

Language: Английский

Citations

0
MEG Activity in Visual and Auditory Cortices Represents Acoustic Speech-Related Information during Silent Lip Reading DOI Creative Commons
Felix Bröhl, Anne Keitel, Christoph Kayser

et al.

eNeuro, Journal Year: 2022, Volume and Issue: 9(3), P. ENEURO.0209 - 22.2022

Published: May 1, 2022

Abstract Speech is an intrinsically multisensory signal, and seeing the speaker’s lips forms a cornerstone of communication in acoustically impoverished environments. Still, it remains unclear how brain exploits visual speech for comprehension. Previous work debated whether lip signals are mainly processed along auditory pathways or system directly implements speech-related processes. To probe this, we systematically characterized dynamic representations multiple acoustic speech-derived features source localized MEG recordings that were obtained while participants listened to viewed silent speech. Using mutual-information framework provide comprehensive assessment well temporal occipital cortices reflect physically presented unique aspects absent but may be critical Our results demonstrate both feature functionally specific form restoration: during reading, they unheard features, independent co-existing visible movements. This restoration emphasizes pitch signature cortex envelope predictive lip-reading performance. These findings suggest when lips, engages support comprehension by exploiting correspondences between movements spectro-temporal cues.

Language: Английский

Citations

13
A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts DOI Creative Commons
Maansi Desai, Alyssa M Field, Liberty S. Hamilton

et al.

PLoS Computational Biology, Journal Year: 2024, Volume and Issue: 20(9), P. e1012433 - e1012433

Published: Sept. 9, 2024

Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory visual cues to understand one another. For example, objects may make sounds as they move space, or we movement of person’s mouth better what are saying noisy environment. Still, many neuroscience experiments rely on unimodal stimuli encoding sensory features brain. The extent which information influence vice versa natural environments thus unclear. Here, addressed this question by recording scalp electroencephalography (EEG) 11 subjects listened watched movie trailers audiovisual (AV), (V) only, audio (A) only conditions. We then fit linear models that described relationship between brain responses acoustic, phonetic, stimuli. also compared whether feature tuning was same when were presented original AV format versus removed. In these stimuli, relatively uncorrelated, included spoken narration over scene well animated live-action characters talking with without their face visible. stimulus, found similar A-only conditions, similarly, for present (AV) removed (V only). cross prediction analysis, investigated trained data predicted A V test similarly data. Overall, performance using training sets sets, suggesting has smaller effect EEG. contrast, set slightly worse than matching sets. This suggests stronger EEG, though makes no qualitative difference derived tuning. effect, our results show researchers benefit from richness multimodal datasets, can be used answer more research question.

Language: Английский

Citations

2
Decreasing hearing ability does not lead to improved visual speech extraction as revealed in a neural speech tracking paradigm DOI Creative Commons
Chandra Leon Haider, Anne Hauswald, Nathan Weisz

et al.

bioRxiv (Cold Spring Harbor Laboratory), Journal Year: 2024, Volume and Issue: unknown

Published: March 14, 2024

Abstract The use of visual speech is thought to be especially important in situations where acoustics are unclear and individuals with hearing impairment. To investigate this a neural tracking paradigm, we measured MEG sixty-seven mid- old-age during audiovisual (AV), audio-only (A), visual-only (V) the context face masks. First, could extend previous findings by showing that not only young normal-hearing but also aging decreasing ability brain superior neurally acoustic spectrogram AV compared A presentations, multi-speaker situations. addition lip movements further increases benefit. Second, show lower levels affected more However, population, effect seems composite blocked distorted acoustics. Third, confirm findings, benefit varies strongly across individuals. We general individual predicts how much people engage difficult listening Interestingly, was correlated thresholds therefore does seem widely used compensatory strategy impaired.

Language: Английский

Citations

1
Phonological representations of auditory and visual speech in the occipito-temporal cortex and beyond DOI Creative Commons
Alice Van Audenhaege, Stefania Mattioni, Filippo Cerpelloni

et al.

bioRxiv (Cold Spring Harbor Laboratory), Journal Year: 2024, Volume and Issue: unknown

Published: July 26, 2024

Abstract Speech is a multisensory signal that can be extracted from the voice and lips. Previous studies suggested occipital temporal regions encode both auditory visual speech features but their precise location nature remain unclear. We characterized brain activity using fMRI (in male female) to functionally individually define bilateral Fusiform Face Areas (FFA), left Visual Word Form Area (VWFA), an audio-visual region in Superior Temporal Sulcus (lSTS) control Para-hippocampal Place (PPA). In these regions, we performed multivariate patterns classification of corresponding phonemes (speech sounds) visemes (lip movements). observed VWFA lSTS represent phonological information vision sounds. The representations appeared selective anterior portion VWFA, as found viseme not phoneme representation adjacent FFA or even posterior while PPA did phonology any modality. Interestingly, cross-modal decoding revealed aligned across senses lSTS, VWFA. A whole-brain searchlight analysis additionally pSTS somato-motor cortex overlapping with oro-facial articulators. Altogether, our results demonstrate are represented extending its functional coding beyond orthography. geometries do align they STS cortex, suggesting distinct distributed network. Significance statement Which representations? show (VWFA) process sounds lip movements. However, aligns senses, does not, indicating different encoding mechanisms. These findings extend role reading. An additional approach reveals shared superior network for phonology.

Language: Английский

Citations

1
Modality-Specific Perceptual Learning of Vocoded Auditory versus Lipread Speech: Different Effects of Prior Information DOI Creative Commons
Lynne E. Bernstein, Edward T. Auer, Silvio P. Eberhardt

et al.

Brain Sciences, Journal Year: 2023, Volume and Issue: 13(7), P. 1008 - 1008

Published: June 29, 2023

Traditionally, speech perception training paradigms have not adequately taken into account the possibility that there may be modality-specific requirements for perceptual learning with auditory-only (AO) versus visual-only (VO) stimuli. The study reported here investigated hypothesis are differences in how prior information is used by normal-hearing participants during vocoded VO training. Two different experiments, one AO (Experiment 1) and VO, lipread, 2), effects of giving types

Language: Английский

Citations

3