Matching text and audio embeddings: exploring transfer-learning strategies for language-based audio retrieval

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Weck, Benno
• dc.contributor.author Pérez Fernández, Miguel
• dc.contributor.author Kirchhoff, Holger
• dc.contributor.author Serra, Xavier
• dc.date.accessioned 2025-05-30T05:48:41Z
• dc.date.available 2025-05-30T05:48:41Z
• dc.date.issued 2022
• dc.description.abstract We present an analysis of large-scale pretrained deep learning models used for cross-modal (text-to-audio) retrieval. We use embeddings extracted by these models in a metric learning framework to connect matching pairs of audio and text. Shallow neural networks map the embeddings to a common dimensionality. Our system, which is an extension of our submission to the Language-based Audio Retrieval Task of the DCASE Challenge 2022, employs the RoBERTa foundation model as the text embedding extractor. A pretrained PANNs model extracts the audio embeddings. To improve the generalisation of our model, we investigate how pretraining with audio and associated noisy text collected from the online platform Freesound improves the performance of our method. Furthermore, our ablation study reveals that the proper choice of the loss function and fine-tuning the pretrained models are essential in training a competitive retrieval system.
• dc.format.mimetype application/pdf
• dc.identifier.citation Weck B, Pérez M, Kirchhoff H, Serra X. Matching text and audio embeddings: exploring transfer-learning strategies for language-based audio retrieval. In: Lagrange M, Mesaros AM, Pellegrini T, Richard G, Serizel R, Stowell D, editors. Proceedings of the 7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022); 2022 Nov 3-4; Nancy: France. [Tokyo]: DCASE; 2022. p. p. 206-210.
• dc.identifier.uri http://hdl.handle.net/10230/70562
• dc.language.iso eng
• dc.publisher Detection and Classication of Acoustic Scenes and Events (DCASE)
• dc.relation.ispartof Lagrange M, Mesaros AM, Pellegrini T, Richard G, Serizel R, Stowell D, editors. Proceedings of the 7th Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2022); 2022 Nov 3-4; Nancy: France. [Tokyo]: DCASE; 2022.
• dc.rights This work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri http://creativecommons.org/licenses/by/4.0/
• dc.subject.keyword Language-based audio retrieval
• dc.subject.keyword Transfer-learning strategies
• dc.subject.keyword Text embeddings
• dc.subject.keyword Audio embeddings
• dc.title Matching text and audio embeddings: exploring transfer-learning strategies for language-based audio retrieval
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion