A critical assessment of visual sound source localization models including negative audio
Mostra el registre complet Registre parcial de l'ítem
- dc.contributor.author Juanola, Xavier
- dc.contributor.author Haro Ortega, Gloria
- dc.contributor.author Fuentes, Magdalena
- dc.date.accessioned 2025-04-11T05:49:52Z
- dc.date.available 2025-04-11T05:49:52Z
- dc.date.issued 2025
- dc.description.abstract The task of Visual Sound Source Localization (VSSL) involves identifying the location of sound sources in visual scenes, integrating audio-visual data for enhanced scene understanding. Despite advancements in state-of-the-art (SOTA) models, we observe three critical flaws: i) The evaluation of the models is mainly focused in sounds produced by objects that are visible in the image, ii) The evaluation often assumes a prior knowledge of the size of the sounding object, and iii) No universal threshold for localization in real-world scenarios is established, as previous approaches only consider positive examples without accounting for both positive and negative cases. In this paper, we introduce extended test sets and new metrics designed to complete the current standard evaluation of VSSL models by testing them in scenarios where none of the objects in the image corresponds to the audio input, i.e. a negative audio. We consider three types of negative audio: silence, noise and offscreen. Our analysis reveals that numerous SOTA models fail to appropriately adjust their predictions based on audio input, suggesting that these models may not be leveraging audio information as intended. Additionally, we provide a comprehensive analysis of the range of maximum values in the estimated audio-visual similarity maps, in both positive and negative audio cases, and show that most of the models are not discriminative enough, making them unfit to choose a universal threshold appropriate to perform sound localization without any a priori information of the sounding object, that is, object size and visibility.
- dc.description.sponsorship FPI scholarship PRE2022-101321, Maria de Maeztu CEX2021-001195-M/AEI/10.13039/501100011033, MICINN/FEDER UE project ref. PID2021-127643NB-I00, Fulbright Program and Ministerio de Universidades (Spain) funding for mobility stays of professors and researchers in foreign higher education and research centers.
- dc.format.mimetype application/pdf
- dc.identifier.citation Juanola X, Haro G, Fuentes M. A critical assessment of visual sound source localization models including negative audio. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2025 April 6-11; Hyderabad, Índia. p 1-5. DOI: 10.1109/ICASSP49660.2025.10890384
- dc.identifier.isbn 9798350368741
- dc.identifier.uri http://hdl.handle.net/10230/70127
- dc.language.iso eng
- dc.publisher Institute of Electrical and Electronics Engineers (IEEE)
- dc.relation.ispartof IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2025 April 6-11; Hyderabad, India. p 1-5. DOI: 10.1109/ICASSP49660.2025.10890384
- dc.relation.projectID info:eu-repo/grantAgreement/ES/3PE/PID2021-127643
- dc.rights © 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. http://dx.doi.org/10.1109/ICASSP49660.2025.10890384
- dc.rights.accessRights info:eu-repo/semantics/openAccess
- dc.subject.keyword Sound source localization
- dc.subject.keyword Audio-visual learning
- dc.title A critical assessment of visual sound source localization models including negative audio
- dc.type info:eu-repo/semantics/conferenceObject
- dc.type.version info:eu-repo/semantics/acceptedVersion