A plan for sustainable MIR evaluation

McFee, Brian; Humphrey, Eric J.; Urbano, Julián

A plan for sustainable MIR evaluation

License

Citation

McFee B, Humphrey EJ, Urbano J. A plan for sustainable MIR evaluation. In: Devaney J, Mandel MI, Turnbull D, Tzanetakis G, editors. 17th International Society for Music Information Retrieval Conference (ISMIR 2016); 2016 Aug 7-11; New York City, USA. [Place unknown]: International Society for Music Information Retrieval; 2016. p. 285-91.

Abstract

The Music Information Retrieval Evaluation eXchange (MIREX) is a valuable community service, having established standard datasets, metrics, baselines, methodologies, and infrastructure for comparing MIR methods. While MIREX has managed to successfully maintain operations for over a decade, its long-term sustainability is at risk. The imposed constraint that input data cannot be made freely available to participants necessitates that all algorithms run on centralized computational resources, which are administered by a limited number of people. This incurs an approximately linear cost with the number of submissions, exacting significant tolls on both human and financial resources, such that the current paradigm becomes less tenable as participation increases. To alleviate the recurring costs of future evaluation campaigns, we propose a distributed, community-centric paradigm for system evaluation, built upon the principles of openness, transparency, reproducibility, and incremental evaluation. We argue that this proposal has the potential to reduce operating costs to sustainable levels. Moreover, the proposed paradigm would improve scalability, and eventually result in the release of large, open datasets for improving both MIR techniques and evaluation methods.