Object Familiarity and Semantic Congruency Modulate Responses in Cortical Audiovisual Integration Areas

Abstract

The cortical integration of auditory and visual features is crucial for efficient object recognition. Previous studies have shown that audiovisual (AV) integration is affected by where and when auditory and visual features occur. However, because relatively little is known about the impact of what is integrated, we here investigated the impact of semantic congruency and object familiarity on the neural correlates of AV integration. We used functional magnetic resonance imaging to identify regions involved in the integration of both (congruent and incongruent) familiar animal sounds and images and of arbitrary combinations of unfamiliar artificial sounds and object images. Unfamiliar object images and sounds were integrated in the inferior frontal cortex (IFC), possibly reflecting learning of novel AV associations. Integration of familiar, but semantically incongruent combinations also correlated with IFC activation and additionally involved the posterior superior temporal sulcus (pSTS). For highly familiar semantically congruent AV pairings, we again found AV integration effects in pSTS and additionally in superior temporal gyrus. These findings demonstrate that the neural correlates of objectrelated AV integration reflect both semantic congruency and familiarity of the integrated sounds and images.

Touch, sound and vision in human superior temporal sulcus

“…Previous studies have shown that posterior STS responds more to multisensory auditory-visual stimuli than to unisensory auditory or visual stimuli (Beauchamp et al, 2004b;Calvert, 2001;Hein et al, 2007;Noesselt et al, 2007;Raij et al, 2000;Van Atteveldt et al, 2004). Consistent with these results, we observed a larger response for multisensory auditory-tactile stimuli than unisensory auditory or tactile stimulation.…”

“…That is, the response to auditory-tactile stimuli was greater than the response to auditory or tactile stimuli in isolation, but was not greater than the summed response to auditory and tactile unisensory stimuli (Stein and Meredith, 1993). Previous fMRI studies of auditory-visual integration in STS (Beauchamp et al, 2004a;Beauchamp et al, 2004b;Hein et al, 2007;Van Atteveldt et al, 2004;van Atteveldt et al, 2007) and auditory-tactile integration in auditory cortex (Kayser et al, 2005) have also not observed super-additive changes in the BOLD signal, perhaps because only a few single neurons show super-additivity (Laurienti et al, 2005;Perrault et al, 2005). Supporting this idea, in single-unit recording studies, only a small fraction of STP neurons respond to both auditory and tactile stimulation (Bruce et al, 1981;Hikosaka et al, 1988); the same is true in multisensory regions of cat cortex (Clemo et al, 2007).…”

The balance between memory and unification in semantics: A dynamic account of the N400

“…The same pattern of BOLD responses was found by Beauchamp, Lee, Argall, and Martin (2004): increased activation in left posterior temporal cortex to the matching pictureÁsound combination as compared to the mismatching combination. The study by Hein et al (2007) provides evidence for a functional dissociation between frontal and temporal cortex that is of great relevance for our purposes. IFG showed stronger BOLD responses to incongruent familiar animal sounds and images (e.g., a meowing dog) than to congruent combinations (e.g., a barking dog).…”

Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions

“…The max-criterion (AV > max(A, V)), can be see as a compromise between the aforementioned criteria. It has been described as sufficiently strict to isolate areas of integration (Hein et al, 2007) and has the advantage of having more detection power than the superadditivity criterion (Beauchamp, 2005b;.…”