Communication breakdown: on the low mutual intelligibility between human and neural captioning

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Dessì, Roberto
• dc.contributor.author Gualdoni, Eleonora
• dc.contributor.author Franzon, Francesca
• dc.contributor.author Boleda, Gemma
• dc.contributor.author Baroni, Marco
• dc.date.accessioned 2023-01-24T07:07:30Z
• dc.date.available 2023-01-24T07:07:30Z
• dc.date.issued 2022
• dc.description Comunicació presentada a la 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), celebrada del 7 a l'11 de desembre de 2022 a Abu Dhabi, Emirats Àrabs Units.
• dc.description.abstract We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced IMAGECODE data-set (Krojer et al., 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading.
• dc.description.sponsorship This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreements No. 715154 and No. 101019291) and the Spanish Research Agency (ref. PID2020-112602GB-I00).
• dc.format.mimetype application/pdf
• dc.identifier.citation Dessi R, Gualdoni E, Franzon F, Boleda G, Baroni M. 2022. Communication breakdown: on the low mutual intelligibility between human and neural captioning. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022). 2022 dec 7-11; Abu Dhabi, United Arab Emirates. Stroudsburg: ACL, 2022. p. 7998–8007.
• dc.identifier.uri http://hdl.handle.net/10230/55388
• dc.language.iso eng
• dc.publisher ACL (Association for Computational Linguistics)
• dc.relation.ispartof Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022). 2022 dec 7-1; Abu Dhabi, United Arab Emirates. Stroudsburg: ACL, 2022. p. 7998–8007
• dc.relation.projectID info:eu-repo/grantAgreement/ES/2PE/PID2020-112602GB-I00
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/715154
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/101019291
• dc.rights © ACL, Creative Commons Attribution 4.0 License
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri https://creativecommons.org/licenses/by/4.0/
• dc.subject.other Neural models
• dc.subject.other Models neuronals
• dc.title Communication breakdown: on the low mutual intelligibility between human and neural captioning
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion