RepoVizz: a multi-modal on-line database and browsing tool for music performance research

Abstract

RepoVizz is a data repository and visualization tool for/nstructured storage and user-friendly browsing of music performance/nmulti-modal recordings. The primary purpose of/nRepoVizz is to offer means for the scientific community to/ngain on-line access to a music performance multi-modal database/nshared among researchers. RepoVizz is designed to/nhold synchronized streams of heterogeneous data (audio,/nvideo, motion capture, physiological measures, extracted/ndescriptors, etc.), annotations, and musical scores. Data/nstreams are stored as single-channel PCM files. The motion-/nsequence format MJ2 is chosen for video. Annotations/nare stored as text files. MusicXML is used for musical/nscores. Data is structured by customizable XML skeleton/nfiles enabling meaningful retrieval. Skeleton files are first/ncreated during data gathering, and provide means to (re-/n)organize acquired data in any desired hierarchical structure,/nas they only hold pointers to stored data files. Once a/ndataset is created and uploaded to the server, each skeleton/nfile defines a view. Multi-track data visualization is done/nremotely via a powerful HTML5-based environment that/nenables web-driven editing (add annotations, extract descriptors)/nand downloading of datasets. A first instance of/nRepoVizz has been created within the EU FET-Open/nProject SIEMPRE, devoted to studying social interaction in/nensemble performance.