Immersive orchestras: Audio processing for orchestral music VR content

Janer Mestres, Jordi; Gómez Gutiérrez, Emilia, 1975-; Martorell Domínguez, Agustín; Miron, Marius; de Wit, Benjamin

doi:http://dx.doi.org/10.1109/VS-GAMES.2016.7590352

Immersive orchestras: Audio processing for orchestral music VR content

Citation

Janer J, Gomez E, Martorell A, Miron M, de Wit B. Immersive orchestras: Audio processing for orchestral music VR content. In: 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-Games); 2016 Sep 7-9; Barcelona, Spain. [New York]: IEEE; 2016. [2 p.]. DOI: 10.1109/VS-GAMES.2016.7590352

Abstract

This paper combines Audio Signal Processing and Virtual Reality (VR) content to create novel immersive experiences for orchestral music audiences. In VR, the auralization of sound sources of recorded live content remains still a rather unexplored topic. We aim to build a multimodal experience, where visual and audio cues bring a sonic augmentation of the real scene. In the particular scenario of orchestral music content, our goal is to acoustically zoom on a particular instrument when the VR user stares at it. This work aims to improve the learning aspects of music listening, either for education or for personal enrichment. We use audio signal processing to separate different sound sources (instruments) in a acoustic scene (orchestral music recording). Given the signals captured by multiple microphones and the musical score of the piece, our system is able to isolate the different instruments. From the processed separated tracks, we use a binaural rendering technique to emphasize a give instrument. For these experiments we used original content from top European orchestras.

Description

Comunicació presentada a la 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-Games), celebrada els dies 7 a 9 de setembre de 2016 a Barcelona, Espanya.