Multisensory integration with a head-mounted display: background visual motion and sound motion.

Human factors

PubMedID: 20653227

Harrison WJ, Thompson MB, Sanderson PM. Multisensory integration with a head-mounted display: background visual motion and sound motion. Hum Factors. 2010;52(1):78-91.
OBJECTIVE
The aim of this study was to assess how background visual motion and the relative movement of sound affect a head-mounted display (HMD) wearer's performance at a task requiring integration of auditory and visual information.

BACKGROUND
HMD users are often mobile. A commercially available speaker in a fixed location delivers auditory information affordably to the HMD user. However, previous research has shown that mobile HMD users perform poorly at tasks that require integration of visual and auditory information when sound comes from a free-field speaker. The specific cause of the poor task performance is unknown.

METHOD
Participants counted audiovisual events that required integration of sounds delivered via a free-field speaker and vision on an HMD. Participants completed the task while either walking around a room, sitting in the room, or sitting inside a mobile room that allowed separate manipulation of background visual motion and speaker motion.

RESULTS
Participants' accuracy at counting target audiovisual events was worse when participants were walking than when sitting at a desk, p = .032. Compared with when they were sitting at a desk, participants' accuracy at counting target audiovisual events showed a trend to be worse when they experienced a combination of background visual motion and the relative movement of sound, p = .058.

CONCLUSION
Multisensory integration performance is least effective when HMD users experience a combination of background visual motion and relative movement of sound. Eye reflexes may play an important role.

APPLICATION
Results apply to situations in which HMD wearers are mobile when receiving multimodal information, as in health care and military contexts.