Enhancing spatial perception of virtual object through mid-air feedback for disabled users

There is an increasing demand to enhance the user’s ability to better identify the location of sound events and virtual objects in 3D space. The demand is not only within the gaming industry but also with users with disabilities looking to engage with VR technologies. In doing so, users can increase their ability to interact with virtual and digital worlds.

Wearable technologies that provide haptic feedback experience to users, such as gloves and gaming controllers, often require time to ‘put them on’; users are aware of the technology on their body; require calibration to the individual user and often restrict movement.

Our research explores the relationship between aural and haptic feedback modalities and how they may compensate each other. Specifically, we examine ways in which mid-air haptic feedback, using ultrasound technology, is used to improve our understanding regarding the position and direction of virtual sound objects. A prototype system, developed in Unity, enables objects on screen to provide information about their virtual location, direction and size through haptic feedback received on the palm as well as through binaural audio. For example, one may locate more accurately the direction and distance of a conversation taking place behind him through the combination haptic and aural feedback rather than with audio only. Our system can support visually-impaired children to learn about the world around them such as distance and shape of objects as well as improving accessibility in immersive applications for people with hearing loss.

Users are asked to identify to the position of sound objects through binaural listening with and without haptic feedback. We measure how well they perform in terms of accuracy. We have also examined how users experience movement such as left and right, front and back, and diagonal. Preliminary results show a higher rate of success when haptic feedback is used with the ability to enhance the spatial perception of users.