The Impact of training order on memorization and forgetting in large language models

Abstract

This paper investigates the phenomenon of memorization in large language models (LLMs), focusing on its dynamics throughout the training process and its implications for data privacy and copyright compliance. While LLMs have demonstrated impressive performance across various natural language processing tasks, their tendency to reproduce portions of their training data verbatim raises concerns about the potential leakage of sensitive information and copyright nfringement. Our research reveals several key findings about the memorization process in LLMs. First, we observe that models tend to memorize a higher proportion of their training data during the early stages of training. This memorization rate exhibits logarithmic growth before stabilizing into a linear pattern. Notably, this logarithmic growth is attributed to an increase in the number of examples forgotten by the model at each step, while the number of newly memorized examples remains relatively constant. We demonstrate that the dynamic nature of memorization results in few examples being retained throughout the entire training process. However, forgotten examples are often re-memorized during subsequent training steps. Importantly, examples memorized early in training have a higher likelihood of remaining memorized throughout the entire process. Based on these findings, we tentatively recommend that model developers avoid including their most sensitive data at either the very beginning or end of the training process to mitigate potential risks associated with memorization.

Our study also reveals that different types of text are memorized at varying rates, although the overall memorization dynamics remain consistent across text categories.

We find that contact information is disproportionately memorized, and many examples that persist in memory throughout training, especially those containing contact information, follow a "templated" structure. These insights contribute to a deeper understanding of memorization in LLMs and provide a foundation for developing strategies to minimize the memorization of sensitive information and mitigate the risk of training data extraction attacks. Furthermore, our findings have implications for addressing copyright concerns in the development and deployment of large language models. This research advances the field by offering a more nuanced view of memorization dynamics in LLMs and provides practical recommendations for model developers to enhance data privacy and copyright compliance in their training processes.