Challenges in including extra-linguistic context in pre-trained language models

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Sorodoc, Ionut-Teodor
• dc.contributor.author Aina, Laura
• dc.contributor.author Boleda, Gemma
• dc.date.accessioned 2023-03-15T07:20:51Z
• dc.date.available 2023-03-15T07:20:51Z
• dc.date.issued 2022
• dc.description Comunicació presentada a Third Workshop on Insights from Negative Results in NLP (Insights 2022), celebrat el 26 de maig de 2022 a Dublín, Irlanda.
• dc.description.abstract To successfully account for language, computational models need to take into account both the linguistic context (the content of the utterances) and the extra-linguistic context (for instance, the participants in a dialogue). We focus on a referential task that asks models to link entity mentions in a TV show to the corresponding characters, and design an architecture that attempts to account for both kinds of context. In particular, our architecture combines a previously proposed specialized module (an “entity library”) for character representation with transfer learning from a pre-trained language model. We find that, although the model does improve linguistic contextualization, it fails to successfully integrate extra-linguistic information about the participants in the dialogue. Our work shows that it is very challenging to incorporate extra-linguistic information into pretrained language models.
• dc.description.sponsorship This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 715154). We are also grateful to the NVIDIA Corporation for the donation of GPUs used for this research.
• dc.format.mimetype application/pdf
• dc.identifier.citation Sorodoc IT, Aina L, Boleda G. Challenges in including extra-linguistic context in pre-trained language models. In: Tafreshi S, Sedoc J, Rogers A, Drozd A, Rumshisky A, Akula A, editors. The Third Workshop on Insights from Negative Results in NLP (Insights 2022): proceedings of the Workshop; 2022 May 26; Dublin, Ireland. [Stroudsburg]: Association for Computational Linguistics; 2022. p. 134-8. DOI: 10.18653/v1/2022.insights-1.18
• dc.identifier.doi http://dx.doi.org/10.18653/v1/2022.insights-1.18
• dc.identifier.uri http://hdl.handle.net/10230/56228
• dc.language.iso eng
• dc.publisher ACL (Association for Computational Linguistics)
• dc.relation.ispartof Tafreshi S, Sedoc J, Rogers A, Drozd A, Rumshisky A, Akula A, editors. The Third Workshop on Insights from Negative Results in NLP (Insights 2022): proceedings of the Workshop; 2022 May 26; Dublin, Ireland. [Stroudsburg]: Association for Computational Linguistics; 2022. p. 134-8.
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/715154
• dc.rights © ACL, Creative Commons Attribution 4.0 License
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri http://creativecommons.org/licenses/by/4.0/
• dc.subject.other Models lingüístics
• dc.title Challenges in including extra-linguistic context in pre-trained language models
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion