Hearing by Eye: How Much Spatial Degradation can Be Tolerated?

Abstract

In the McGurk effect (McGurk and MacDonald, 1976 Nature 264 746-748), illusory auditory perception is produced if the visual information from lip movements is discrepant from the auditory information from the voice. A study is reported of the tolerance of the effect to varying levels of spatial degradation (videotaped images of a speaker's face were quantised by a mosaic transform). The illusory effect systematically decreased with an increase in the coarseness of the spatial quantisation. However, even with the coarsest level (11.2 pixels/face) the illusion did not completely disappear. In addition, those participants who did not experience the illusion nevertheless showed the effects of auditory-visual interaction in their clarity ratings of the auditory stimulus. It is concluded that auditory-visual interaction in visible speech perception is based on relatively coarse-spatial-scale information.

Spatio‐temporal distribution of brain activity associated with audio‐visually congruent and incongruent speech and the McGurk Effect

“…On the one hand there is evidence that acoustic properties of the auditory stimulus and robustness of the visual input (MacDonald et al. 2000; Brancazio and Miller 2005), rather than speech‐specific brain processing, contribute to the magnitude of the McGurk effect. On the other hand, developmental age (5–19 years) of listeners significantly correlated with the number of trials in which the McGurk effect was evident (Trembley et al.…”

“…Spatial quantization of the visual display, degraded from intact to increasingly coarser displays, results in decreasing efficacy of the visual input and a shift in perception toward the auditory stimulus (MacDonald et al. 2000). …”

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

“…Temporal integration is certainly an important aspect to the perception of multimedia; however, perceived synchrony does not equal comprehension. As noted, the loss of visual information can put a burden on perception, making it harder to recognise people [4], text [25], and speech movements [10,24]. Auditory distortions will similarly mask important acoustical information [23], making it difficult to understand speech sounds [13].…”

“…Not surprisingly, severe loss of visual details can lead to difficulties in recognising both people [4] and text [25]. Moreover, the auditory modality will eventually dominate speech perception when visual information is distorted [10,24]. Conversely, noise and auditory compression artifacts can mask essential spectral frequencies and lead to distracting audio effects [23]; in speech perception, reduced audio quality typically leads to an increased dependency on the visual modality [13].…”

“…The incentive was to establish three corresponding distortion levels for each modality to be carried through to the two subsequent experiments. Only the audio distortion levels were selected in this manner due to the experimental set-up; video distortion levels were chosen as best-fits to previous research [10,24], with their appropriateness tested in comparison to audio distortion.…”