Learning contextual tag embeddings for cross-modal alignment of audio and tags

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Favory, Xavier
• dc.contributor.author Drossos, Konstantinos
• dc.contributor.author Virtanen, Tuomas
• dc.contributor.author Serra, Xavier
• dc.date.accessioned 2023-03-06T07:25:54Z
• dc.date.issued 2021
• dc.description Comunicació presentada a 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), celebrat del 6 a l'11 de juny de 2021 de manera virtual.
• dc.description.abstract Self-supervised audio representation learning offers an attractive alternative for obtaining generic audio embeddings, capable to be employed into various downstream tasks. Published approaches that consider both audio and words/tags associated with audio do not employ text processing models that are capable to generalize to tags unknown during training. In this work we propose a method for learning audio representations using an audio autoencoder (AAE), a general word embeddings model (WEM), and a multi-head self-attention (MHA) mechanism. MHA attends on the output of the WEM, providing a contextualized representation of the tags associated with the audio, and we align the output of MHA with the output of the encoder of AAE using a contrastive loss. We jointly optimize AAE and MHA and we evaluate the audio representations (i.e. the output of the encoder of AAE) by utilizing them in three different downstream tasks, namely sound, music genre, and music instrument classification. Our results show that employing multi-head self-attention with multiple heads in the tag-based network can induce better learned audio representations.
• dc.format.mimetype application/pdf
• dc.identifier.citation Favory X, Drossos K, Virtanen T, Serra X. Learning contextual tag embeddings for cross-modal alignment of audio and tags. In: 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): proceedings; 2021 Jun 6-11; Toronto, Canada. [Piscataway]: IEEE; 2021. p. 596-600. DOI: 10.1109/ICASSP39728.2021.9414638
• dc.identifier.doi http://dx.doi.org/10.1109/ICASSP39728.2021.9414638
• dc.identifier.issn 1520-6149
• dc.identifier.uri http://hdl.handle.net/10230/56042
• dc.language.iso eng
• dc.publisher Institute of Electrical and Electronics Engineers (IEEE)
• dc.relation.ispartof 2021 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP): proceedings; 2021 Jun 6-11; Toronto, Canada. [Piscataway]: IEEE; 2021. p. 596-600.
• dc.rights © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. http://dx.doi.org/10.1109/ICASSP39728.2021.9414638
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.subject.keyword representation learning
• dc.subject.keyword multimodal contrastive learning
• dc.subject.keyword audio classification
• dc.title Learning contextual tag embeddings for cross-modal alignment of audio and tags
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/acceptedVersion