Weck, BennoSerra, Xavier2025-05-302023Weck B, Serra X. Data leakage in cross-modal retrieval training: a case study. In: Maragos P, Berberidis K, Boufounos P, editors. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023); 2023 June 4-10; Rhodes Island: Greece. [Piscataway]: IEEE; 2023. 5 p. DOI: 10.1109/ICASSP49357.2023.10094617http://hdl.handle.net/10230/70563The recent progress in text-based audio retrieval was largely propelled by the release of suitable datasets. Since the manual creation of such datasets is a laborious task, obtaining data from online resources can be a cheap solution to create large-scale datasets. We study the recently proposed SoundDesc benchmark dataset, which was automatically sourced from the BBC Sound Effects web page. In our analysis, we find that SoundDesc contains several duplicates that cause leakage of training data to the evaluation data. This data leakage ultimately leads to overly optimistic retrieval performance estimates in previous benchmarks. We propose new training, validation, and testing splits for the dataset that we make available online. To avoid weak contamination of the test data, we pool audio files that share similar recording setups. In our experiments, we find that the new splits serve as a more challenging benchmark.application/pdfeng© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. http://dx.doi.org/10.1109/ICASSP49357.2023.10094617info:eu-repo/semantics/conferenceObjectText-based audio retrievalCross-modalDuplicatesData leakageDeep learninginfo:eu-repo/semantics/embargoedAccess