Modeling word interpretation with deep language models: the interaction between expectations and lexical information

Aina, Laura; Brochhagen, Thomas; Boleda, Gemma

Modeling word interpretation with deep language models: the interaction between expectations and lexical information

License

Citation

Aina L, Brochhagen T, Boleda G. Modeling word interpretation with deep language models: the interaction between expectations and lexical information. In: Denison S, Mack M, Xu Y, Armstrong BC. Proceedings for the 42nd Annual Meeting of the Cognitive Science Society; 2020 Jul 29 - 1 Aug. [Toronto]: Cognitive Science Society, 2020. 1518-24 p.

Abstract

How a word is interpreted depends on the context it appears in. We study word interpretation leveraging deep language models, tracing the contribution and interaction of two sources of information that have been shown to be central to it: contextinvariant lexical information, represented by the word embeddings of a model, and a listener’s contextual expectations, represented by its predictions. We define operations to combine these components to obtain representations of word interpretations. We instantiate our framework using two English language models, and evaluate the resulting representations in the extent by which they reflect contextual word substitutes provided by human subjects. Our results suggest that both lexi cal information and expectations codify information pivotal to word interpretation; however, their combination is better than either on its own. Moreover, the division of labor between expectations and the lexicon appears to change across contexts