Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

Published online: Mar 22, 2018
Full Text: PDF (7.35 MiB)
DOI: 10.24138/jcomss.v14i1.410
Authors:
Ahmet Kose, Aleksei Tepljakov, Sergei Astapov, Dirk Draheim, Eduard Petlenkov, Kristina Vassiljeva

Abstract

In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia—the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener.We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by the loudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented.

Keywords

virtual reality, synesthesia, acoustic localization, microphone array, therapy
Creative Commons License 4.0
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.