The sleep framework: A new way for LLMs to retain knowledge in continual learning

Researchers introduce a "sleep framework" for continual learning in which LLMs alternate between an active wake phase that stores temporary information in short-term memory and an offline sleep phase that consolidates it into long-term parameters via Mixture-of-Experts parameter expansion, knowledge distillation, and a "dreaming" self-improvement loop. The authors report that this approach avoids catastrophic forgetting in continual translation, scales to 10M tokens on BABILong, reaches 48.9% on single-task and 46.2% on CPT factual-knowledge evaluations, and scores 79.2%, 69.0%, and 46.1% on AIME 24, AIME 25, and HMMT 25.