Self-supervised learning from automatically separated sound scenes

Fonseca, Eduardo; Jansen, Aren; Ellis, Daniel P. W.; Wisdom, Scott; Tagliasacchi, Marco; Hershey, John R.; Plakal, Manoj; Hershey, Shawn; Moore, R. Channing; Serra, Xavier

doi:http://dx.doi.org/10.1109/WASPAA52581.2021.9632739

Self-supervised learning from automatically separated sound scenes

Citation

Fonseca E, Jansen A, Ellis DPW, Wisdom S, Tagliasacchi M, Hershey JR, Plakal M, Hershey S, Moore RC, Serra X. Self-supervised learning from automatically separated sound scenes. In: 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA); 2021 Oct 17-20; New Paltz, United States. [Piscataway]: IEEE; 2021. p. 251-5. DOI: 10.1109/WASPAA52581.2021.9632739

Abstract

Real-world sound scenes consist of time-varying collections of sound sources, each generating characteristic sound events that are mixed together in audio recordings. The association of these constituent sound events with their mixture and each other is semantically constrained: the sound scene contains the union of source classes and not all classes naturally co-occur. With this motivation, this paper explores the use of unsupervised automatic sound separation to decompose unlabeled sound scenes into multiple semantically-linked views for use in self-supervised contrastive learning. We find that learning to associate input mixtures with their automatically separated outputs yields stronger representations than past approaches that use the mixtures alone. Further, we discover that optimal source separation is not required for successful contrastive learning by demonstrating that a range of separation system convergence states all lead to useful and often complementary example transformations. Our best system incorporates these unsupervised separation models into a single augmentation front-end and jointly optimizes similarity maximization and coincidence prediction objectives across the views. The result is an unsupervised audio representation that rivals state-of-the-art alternatives on the established shallow AudioSet classification benchmark.

Description

Comunicació presentada a 2021 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), celebrat del 17 al 20 d'octubre de 2021 a New Paltz, Estats Units.