Carraz Rakotonirina, NathanaëlBaroni, Marco2024-10-152024-10-152024Carraz Rakotonirina N, Baroni M. MemoryPrompt: a light wrapper to improve context tracking in pre-trained language models. In: Calzolari N, Kan MY, Hoste V, Lenci A, Sakti S, Xue N, editors. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024); 2024 May 20-25; Torino, Italy. Brussels: ELRA and ICCL; 2024. p. 11187-95.http://hdl.handle.net/10230/61395Comunicació presentada a Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), celebrada a Turí del 20 al 25 de maig de 2024.Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. We introduce MemoryPrompt, a leaner approach in which the LM is complemented by a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with a sequence of vectors, akin to soft prompts, without requiring LM finetuning. Tested on a task designed to probe a LM’s ability to keep track of multiple fact updates, a MemoryPrompt-augmented LM outperforms much larger LMs that have access to the full input history. We also test MemoryPrompt on a long-distance dialogue dataset, where its performance is comparable to that of a model conditioned on the entire conversation history. In both experiments we also observe that, unlike full-finetuning approaches, MemoryPrompt does not suffer from catastrophic forgetting when adapted to new tasks, thus not disrupting the generalist capabilities of the underlying LM.application/pdfeng© 2024 ELRA - European Language Resources Association: CC BY-NC 4.0info:eu-repo/semantics/conferenceObjectMemory-augmented language modelPromptinginfo:eu-repo/semantics/openAccess