Auditory representations for long lasting sounds: Insights from neural oscillations and event-related brain potentials

Numerous studies revealed that the sound’s basic features like its frequency and intensity including their temporal dynamics are integrated in a unitary representation. That research focused on short, discrete sounds and mainly disregarded how our brain processes long lasting sounds. We review research utilizing the Mismatch Negativity (MMN) event-related potential and neural oscillatory activity for studying representations for long lasting simple sounds such as sinusoidal tones and complex sounds like speech. We report evidence for a critical temporal constraint for the formation of adequate representations for sounds lasting >350 ms. However, we present research showing that the time-variant characteristics (auditory edges) within long lasting sounds exceeding 350 ms enables the formation of auditory representations. We argue that each edge may open an integration window for a sound representation and that the representations established in adjacent temporal windows of integration can be concatenated into an auditory representation of a long sound.

Does Sound Influence Perceived Duration of Visual Motion?

Multimodal perception is a key factor in obtaining a rich and meaningful representation of the world. However, how each stimulus combines to determine the overall percept remains a matter of research. The present work investigates the effect of sound on the bimodal perception of motion. A visual moving target was presented to the participants, associated with a concurrent sound, in a time reproduction task. Particular attention was paid to the structure of both the auditory and the visual stimuli. Four different laws of motion were tested for the visual motion, one of which is biological. Nine different sound profiles were tested, from an easier constant sound to more variable and complex pitch profiles, always presented synchronously with motion. Participants’ responses show that constant sounds produce the worst duration estimation performance, even worse than the silent condition; more complex sounds, instead, guarantee significantly better performance. The structure of the visual stimulus and that of the auditory stimulus appear to condition the performance independently. Biological motion provides the best performance, while the motion featured by a constant-velocity profile provides the worst performance. Results clearly show that a concurrent sound influences the unified perception of motion; the type and magnitude of the bias depends on the structure of the sound stimulus. Contrary to expectations, the best performance is not generated by the simplest stimuli, but rather by more complex stimuli that are richer in information.

Learning a new class of multisensory associations: High-density electrophysiological mapping of the temporal course of audio-visual object processing.

Multisensory objects that are frequently encountered in the natural environment lead to strong associations across a distributed sensory cortical network, with the end result experience of a unitary percept. Remarkably little is known, however, about the cortical processes sub-serving multisensory object formation and recognition. To advance our understanding in this important domain, the present study investigated the brain processes involved in learning and identification of novel visual-auditory objects. Specifically, we introduce and test a rudimentary three-stage model of multisensory object-formation and processing. Thirty adults were remotely trained for a week to recognize a novel class of multisensory objects (3D shapes paired to complex sounds), and high-density event related potentials (ERPs) were recorded to the corresponding unisensory (shapes or sounds only) and multisensory (shapes and sounds) stimuli, before and after intensive training. We identified three major stages of multisensory processing: 1) an early, multisensory, automatic effect (<100 ms) in occipital areas, related to the detection of simultaneous audiovisual signals and not related to multisensory learning 2) an intermediate object-processing stage (100-200 ms) in occipital and parietal areas, sensitive to the learned multi-sensory associations and 3) a late multisensory processing stage (>250 ms) that appears to be involved in both object recognition and possibly memory consolidation. Results from this study provide support for multiple stages of multisensory object learning and recognition that are subserved by an extended network of cortical areas.

Formant-Based Recognition of Words and Other Naturalistic Sounds in Rhesus Monkeys

In social animals, identifying sounds is critical for communication. In humans, the acoustic parameters involved in speech recognition, such as the formant frequencies derived from the resonance of the supralaryngeal vocal tract, have been well documented. However, how formants contribute to recognizing learned sounds in non-human primates remains unclear. To determine this, we trained two rhesus monkeys to discriminate target and non-target sounds presented in sequences of 1–3 sounds. After training, we performed three experiments: (1) We tested the monkeys’ accuracy and reaction times during the discrimination of various acoustic categories; (2) their ability to discriminate morphing sounds; and (3) their ability to identify sounds consisting of formant 1 (F1), formant 2 (F2), or F1 and F2 (F1F2) pass filters. Our results indicate that macaques can learn diverse sounds and discriminate from morphs and formants F1 and F2, suggesting that information from few acoustic parameters suffice for recognizing complex sounds. We anticipate that future neurophysiological experiments in this paradigm may help elucidate how formants contribute to the recognition of sounds.

Effect of background clutter on neural discrimination in the bat auditory midbrain

The discrimination of complex sounds is a fundamental function of the auditory system. This operation must be robust in the presence of noise and acoustic clutter. Echolocating bats are auditory specialists that discriminate sonar objects in acoustically complex environments. Bats produce brief signals, interrupted by periods of silence, rendering echo snapshots of sonar objects. Sonar object discrimination requires that bats process spatially and temporally overlapping echoes to make split-second decisions. The mechanisms that enable this discrimination are not well understood, particularly in complex environments. We explored the neural underpinnings of sonar object discrimination in the presence of acoustic scattering caused by physical clutter. We performed electrophysiological recordings in the inferior colliculus of awake big brown bats, to broadcasts of pre-recorded echoes from physical objects. We acquired single unit responses to echoes and discovered a sub-population of IC neurons that encode acoustic features that can be used to discriminate between sonar objects. We further investigated the effects of environmental clutter on this population's encoding of acoustic features. We discovered that the effect of background clutter on sonar object discrimination is highly variable and depends on object properties and target-clutter spatio-temporal separation. In many conditions, clutter impaired discrimination of sonar objects. However, in some instances clutter enhanced acoustic features of echo returns, enabling higher levels of discrimination. This finding suggests that environmental clutter may augment acoustic cues used for sonar target discrimination and provides further evidence in a growing body of literature that noise is not universally detrimental to sensory encoding.

Synaesthetic Sanctuary

<p><b>Traditional scientific methods of visualising sound data have focused on techniques that attempt to capture distinct elements of the audio signal, such as volume and length. However, existing methods such as spectrograms and waveform analysis are limited in their expression of the characteristics associated with complex sounds such as bird song. This research explores strategies to visualise sound in an aesthetically engaging manner. It uses sound data from native New Zealand birds as a design tool for creating an audio-visual design system. The distinct focus on timing and pitch within these songs makes the data suitable for visual comparison. The design techniques explored throughout this research project attempt to express the unique characteristics of a variety of New Zealand bird songs and calls. It investigates how artistic audio-visual methods can be integrated with scientific techniques so that the auditory data can be made more accessible to non-specialists.</b></p> <p>More specifically, this research aims to take advantage of the natural phonaesthetic connections people make between sonic and visual elements. The final output of this research consists of a generative design system that uses auditory data to create visualisations of New Zealand bird song. These visualisations have a mathematical basis, as well as being audio-visual artworks in themselves.</p>

Synaesthetic Sanctuary

<p><b>Traditional scientific methods of visualising sound data have focused on techniques that attempt to capture distinct elements of the audio signal, such as volume and length. However, existing methods such as spectrograms and waveform analysis are limited in their expression of the characteristics associated with complex sounds such as bird song. This research explores strategies to visualise sound in an aesthetically engaging manner. It uses sound data from native New Zealand birds as a design tool for creating an audio-visual design system. The distinct focus on timing and pitch within these songs makes the data suitable for visual comparison. The design techniques explored throughout this research project attempt to express the unique characteristics of a variety of New Zealand bird songs and calls. It investigates how artistic audio-visual methods can be integrated with scientific techniques so that the auditory data can be made more accessible to non-specialists.</b></p> <p>More specifically, this research aims to take advantage of the natural phonaesthetic connections people make between sonic and visual elements. The final output of this research consists of a generative design system that uses auditory data to create visualisations of New Zealand bird song. These visualisations have a mathematical basis, as well as being audio-visual artworks in themselves.</p>

Synaesthetic Sanctuary

<p><b>Traditional scientific methods of visualising sound data have focused on techniques that attempt to capture distinct elements of the audio signal, such as volume and length. However, existing methods such as spectrograms and waveform analysis are limited in their expression of the characteristics associated with complex sounds such as bird song. This research explores strategies to visualise sound in an aesthetically engaging manner. It uses sound data from native New Zealand birds as a design tool for creating an audio-visual design system. The distinct focus on timing and pitch within these songs makes the data suitable for visual comparison. The design techniques explored throughout this research project attempt to express the unique characteristics of a variety of New Zealand bird songs and calls. It investigates how artistic audio-visual methods can be integrated with scientific techniques so that the auditory data can be made more accessible to non-specialists.</b></p> <p>More specifically, this research aims to take advantage of the natural phonaesthetic connections people make between sonic and visual elements. The final output of this research consists of a generative design system that uses auditory data to create visualisations of New Zealand bird song. These visualisations have a mathematical basis, as well as being audio-visual artworks in themselves.</p>

Linear superposition of responses evoked by individual glottal pulses explain over 80% of the frequency following response to human speech in the macaque monkey

The frequency-following response (FFR) is a scalp-recorded electrophysiological potential that closely follows the periodicity of complex sounds such as speech. It has been suggested that FFRs reflect the linear superposition of responses that are triggered by the glottal pulse in each cycle of the fundamental frequency (F0 responses) and sequentially propagate through auditory processing stages in brainstem, midbrain, and cortex. However, this conceptualization of the FFR is debated, and it remains unclear if and how well a simple linear superposition can capture the spectro-temporal complexity of FFRs that are generated within the highly recurrent and non-linear auditory system. To address this question, we used a deconvolution approach to compute the hypothetical F0 responses that best explain the FFRs in rhesus monkeys to human speech and click trains with time-varying pitch patterns. The linear superposition of F0 responses explained well over 90% of the variance of click train steady state FFRs and well over 80% of mandarin tone steady state FFRs. The F0 responses could be measured with high signal-to-noise ratio and featured several spectro-temporally and topographically distinct components that likely reflect the activation of brainstem (<5ms; 200-1000 Hz), midbrain (5-15 ms; 100-250 Hz) and cortex (15-35 ms; ~90 Hz). In summary, our results in the monkey support the notion that FFRs arise as the superposition of F0 responses by showing for the first time that they can capture the bulk of the variance and spectro-temporal complexity of FFRs to human speech with time-varying pitch. These findings identify F0 responses as a potential diagnostic tool that may be useful to reliably link altered FFRs in speech and language disorders to altered F0 responses and thus to specific latencies, frequency bands and ultimately processing stages.

Mathematical framework for place coding in the auditory system

In the auditory system, tonotopy is postulated to be the substrate for a place code, where sound frequency is encoded by the location of the neurons that fire during the stimulus. Though conceptually simple, the computations that allow for the representation of intensity and complex sounds are poorly understood. Here, a mathematical framework is developed in order to define clearly the conditions that support a place code. To accommodate both frequency and intensity information, the neural network is described as a space with elements that represent individual neurons and clusters of neurons. A mapping is then constructed from acoustic space to neural space so that frequency and intensity are encoded, respectively, by the location and size of the clusters. Algebraic operations -addition and multiplication- are derived to elucidate the rules for representing, assembling, and modulating multi-frequency sound in networks. The resulting outcomes of these operations are consistent with network simulations as well as with electrophysiological and psychophysical data. The analyses show how both frequency and intensity can be encoded with a purely place code, without the need for rate or temporal coding schemes. The algebraic operations are used to describe loudness summation and suggest a mechanism for the critical band. The mathematical approach complements experimental and computational approaches and provides a foundation for interpreting data and constructing models.