A plan for sustainable MIR evaluation

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author McFee, Brianca
• dc.contributor.author Humphrey, Eric J.ca
• dc.contributor.author Urbano, Juliánca
• dc.date.accessioned 2016-10-06T14:54:45Z
• dc.date.available 2016-10-06T14:54:45Z
• dc.date.issued 2016
• dc.description.abstract The Music Information Retrieval Evaluation eXchange (MIREX) is a valuable community service, having established standard datasets, metrics, baselines, methodologies, and infrastructure for comparing MIR methods. While MIREX has managed to successfully maintain operations for over a decade, its long-term sustainability is at risk. The imposed constraint that input data cannot be made freely available to participants necessitates that all algorithms run on centralized computational resources, which are administered by a limited number of people. This incurs an approximately linear cost with the number of submissions, exacting significant tolls on both human and financial resources, such that the current paradigm becomes less tenable as participation increases. To alleviate the recurring costs of future evaluation campaigns, we propose a distributed, community-centric paradigm for system evaluation, built upon the principles of openness, transparency, reproducibility, and incremental evaluation. We argue that this proposal has the potential to reduce operating costs to sustainable levels. Moreover, the proposed paradigm would improve scalability, and eventually result in the release of large, open datasets for improving both MIR techniques and evaluation methods.en
• dc.description.sponsorship B.M. is supported by the Moore-Sloan Data Science Environment at NYU. J.U. is supported by the Spanish Government: JdC postdoctoral fellowship, and projects TIN2015-70816-R and MDM-2015-0502.en
• dc.format.mimetype application/pdfca
• dc.identifier.citation McFee B, Humphrey EJ, Urbano J. A plan for sustainable MIR evaluation. In: Devaney J, Mandel MI, Turnbull D, Tzanetakis G, editors. 17th International Society for Music Information Retrieval Conference (ISMIR 2016); 2016 Aug 7-11; New York City, USA. [Place unknown]: International Society for Music Information Retrieval; 2016. p. 285-91.
• dc.identifier.uri http://hdl.handle.net/10230/27393
• dc.language.iso engen
• dc.publisher International Society for Music Information Retrieval (ISMIR)ca
• dc.relation.ispartof 17th International Society for Music Information Retrieval Conference (ISMIR 2016); 2016 Aug 7-11; New York City, USA. [Place unknown]: International Society for Music Information Retrieval; 2016. p. 285-291.
• dc.relation.projectID info:eu-repo/grantAgreement/ES/1PE/TIN2015-70816-R
• dc.rights © Brian McFee, Eric J. Humphrey, Julián Urbano. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Attribution: Brian McFee, Eric J. Humphrey, Julián Urbano. “A Plan for Sustainable MIR Evaluation”, 17th International Society for Music Information Retrieval Conference, 2016.en
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri http://creativecommons.org/licenses/by/4.0/
• dc.subject.other Recuperació de la informacióca
• dc.subject.other Músicaca
• dc.title A plan for sustainable MIR evaluationen
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion