Can discrete information extraction prompts generalize across language models?

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Rakotonirina, Nathanael Carraz
• dc.contributor.author Dessì, Roberto
• dc.contributor.author Petroni, Fabio
• dc.contributor.author Riedel, Sebastian
• dc.contributor.author Baroni, Marco
• dc.date.accessioned 2023-04-19T06:16:34Z
• dc.date.available 2023-04-19T06:16:34Z
• dc.date.issued 2023
• dc.description Comunicació presentada a: ICLR 2023. 11th Conference on Learning Representations (ICLR) celebrada a Kigali, Rwanda, del 1 al 5 de maig a de 2023.
• dc.description.abstract We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information. After confirming that discrete prompts induced with the AutoPrompt algorithm outperform manual and semi-manual prompts on the slot-filling task, we demonstrate a drop in performance for AutoPrompt prompts learned on a model and tested on another. We introduce a way to induce prompts by mixing language models at training time that results in prompts that generalize well across models. We conduct an extensive analysis of the induced prompts, finding that the more general prompts include a larger proportion of existing English words and have a less order-dependent and more uniform distribution of information across their component tokens. Our work provides preliminary evidence that it’s possible to generate discrete prompts that can be induced once and used with a number of different models, and gives insights on the properties characterizing such prompts.
• dc.description.sponsorship UPF has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101019291).
• dc.description.uri https://creativecommons.org/licenses/by-nc-nd/4.0/deed.es_ES
• dc.format.mimetype application/pdf
• dc.identifier.citation Rakotonirina NC, Dessì R, Petroni F, Riedel S, Baroni M. Can discrete information extraction prompts generalize across language models?. Paper presented at: ICLR 2023. Proceedings of the 11th International Conference on Learning Representations (ICLR); 2023 Mai 1-5; Kigali, Rwanda.
• dc.identifier.uri http://hdl.handle.net/10230/56495
• dc.language.iso eng
• dc.publisher International Conference on Learning Representations (ICLR)
• dc.relation.ispartof ICLR 2023. Proceedings of the 11th International Conference on Learning Representations (ICLR); 2023 Mai 1-5; Kigali, Rwanda.
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/101019291
• dc.rights © Els autors. Aquesta obra està sota Llicència CC Reconeixement-NoComercial-SenseObraDerivada 4.0 Internacional (CC BY-NC-ND 4.0)
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.subject.other Models lingüístics
• dc.subject.other Informació -- Sistemes d'emmagatzematge i recuperació
• dc.title Can discrete information extraction prompts generalize across language models?
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion