NWO project “Aud2Sem: Acoustic to semantic transformations in Human Auditory Cortex” started
2021-10-14
The NWO-funded project “Aud2Sem: Acoustic to semantic transformations in Human Auditory Cortex” led by E.Formisano has officially started on October, 1st, 2021. Find below the project abstract and stay tuned for further information…
A bird chirping, a glass breaking, an ambulance passing by. Listening to sounds helps recognizing events and objects, even when they are out of sight, in the dark or behind a wall, for example. Despite rapid progress in the field of auditory cognition, we know very little about how the brain transforms acoustic sound representations into meaningful source representations. To fill this gap, this project develops a neurobiologically-grounded computational model of sound recognition by combining advanced methodologies from information science, artificial intelligence, and cognitive neuroscience. First, we develop “Sounds”, an ontology that characterizes a large number of everyday sounds and their taxonomic relation in terms of sound-generating mechanisms and properties of the corresponding sources. Second, we develop novel ontology-based deep neural networks (DNNs) that combine acoustic sound analysis with high-level information over the sound sources and learn to perform sound recognition tasks at different abstraction levels. In parallel, we measure brain responses with sub-millimeter functional MRI (fMRI) and intracranial electroencephalography (iEEG) in human listeners as they perform the same SR tasks. We expect that the ontology-based DNNs will explain measured laminar-specific (fMRI) and spectro-temporally resolved (iEEG) neural responses in auditory cortex better than existing acoustic and semantic models. Results of this research will provide mechanistic insights into the acoustic-to-semantic transformations in the human brain. Furthermore, the ontology of everyday sounds and the methods developed for embedding ontological information in DNNs will be of broad academic interest and relevant for the rapidly-expanding societal applications of artificial hearing.