Emergence of a high-dimensional abstraction phase in language transformers

Mostra el registre complet Registre parcial de l'ítem

• dc.contributor.author Cheng, Emily
• dc.contributor.author Doimo, Diego
• dc.contributor.author Kervadec, Corentin
• dc.contributor.author Macocco, Iuri
• dc.contributor.author Yu, Lei
• dc.contributor.author Laio, Alessandro
• dc.contributor.author Baroni, Marco
• dc.date.accessioned 2025-06-11T06:09:44Z
• dc.date.available 2025-06-11T06:09:44Z
• dc.date.issued 2025
• dc.description.abstract A language model (LM) is a mapping from a linguistic context to an output token. However, much remains to be known about this mapping, including how its geometric properties relate to its function. We take a high-level geometric approach to its analysis, observing, across five pre-trained transformer-based LMs and three input datasets, a distinct phase characterized by high intrinsic dimensionality. During this phase, representations (1) correspond to the first full linguistic abstraction of the input; (2) are the first to viably transfer to downstream tasks; (3) predict each other across different LMs. Moreover, we find that an earlier onset of the phase strongly predicts better language modelling performance. In short, our results suggest that a central high-dimensionality phase underlies core linguistic processing in many common LM architectures.
• dc.description.sponsorship This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program (grant agreement No. 101019291). This paper reflects the authors’ view only, and the funding agency is not responsible for any use that may be made of the information it contains. Additionally, D.D. received support from the project “Supporto alla diagnosi di malattie rare tramite l’intelligenza artificiale”(CUP: F53C22001770002).
• dc.format.mimetype application/pdf
• dc.identifier.citation Cheng E, Doimo D, Kervadec C, Macocco I, Yu L, Laio A, Baroni M. Emergence of a high-dimensional abstraction phase in language transformers. In: 13th International Conference on Learning Representations (ICLR 2025); 2025 Apr 24-28; Singapore, Republic of Singapore. Appleton: ICLR; 2025. 26 p.
• dc.identifier.uri http://hdl.handle.net/10230/70669
• dc.language.iso eng
• dc.publisher International Conference on Learning Representations
• dc.relation.projectID info:eu-repo/grantAgreement/EC/H2020/101019291
• dc.rights © Els autors. Aquesta obra està sota Llicència CC BY 4.0
• dc.rights.accessRights info:eu-repo/semantics/openAccess
• dc.rights.uri http://creativecommons.org/licenses/by/4.0
• dc.subject.keyword Interpretability
• dc.subject.keyword Intrinsic dimension
• dc.subject.keyword Large language models
• dc.title Emergence of a high-dimensional abstraction phase in language transformers
• dc.type info:eu-repo/semantics/conferenceObject
• dc.type.version info:eu-repo/semantics/publishedVersion