How a word is interpreted depends on the context it appears in. We study word interpretation leveraging deep language models, tracing the contribution and interaction of two sources of information that have been shown to be central to it: contextinvariant lexical information, represented by the word embeddings of a model, and a listener’s contextual expectations, represented by its predictions. We define operations to combine these components to obtain representations of word interpretations. We ...
How a word is interpreted depends on the context it appears in. We study word interpretation leveraging deep language models, tracing the contribution and interaction of two sources of information that have been shown to be central to it: contextinvariant lexical information, represented by the word embeddings of a model, and a listener’s contextual expectations, represented by its predictions. We define operations to combine these components to obtain representations of word interpretations. We instantiate our framework using two English language models, and evaluate the resulting representations in the extent by which they reflect contextual word substitutes provided by human subjects. Our results suggest that both lexi cal information and expectations codify information pivotal to word interpretation; however, their combination is better than either on its own. Moreover, the division of labor between expectations and the lexicon appears to change across contexts
+
