For a language model (LM) to faithfully model
human language, it must compress vast, potentially infinite information into relatively few
dimensions. We propose analyzing compression in (pre-trained) LMs from two points of
view: geometric and information-theoretic. We
demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding
length under the LM. We then show that, in
turn, high compression of a linguistic dataset
predicts ...
For a language model (LM) to faithfully model
human language, it must compress vast, potentially infinite information into relatively few
dimensions. We propose analyzing compression in (pre-trained) LMs from two points of
view: geometric and information-theoretic. We
demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding
length under the LM. We then show that, in
turn, high compression of a linguistic dataset
predicts rapid adaptation to that dataset, confirming that being able to compress linguistic
information is an important part of successful
LM performance. As a practical byproduct
of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time
on linguistic data, showing that only some encapsulate the relationship between informationtheoretic compression, geometric compression,
and ease-of-adaptation.
+
