Google Research introduces sleep framework for continual LLM learning

Google Research introduces a sleep framework that lets large language models periodically absorb new information into their parameters, according to a research blog post. The approach alternates an active phase, when the model processes inputs and stores temporary information, with an offline phase for memory consolidation and reasoning practice.

During consolidation, the framework unlocks reserved Mixture of Experts memory blocks and transfers short term information into new low rank parameters. It uses knowledge distillation and reinforcement learning based imitation, then resets temporary parameters to create capacity for later sessions. A separate dreaming stage generates synthetic examples, filters them based on their expected training value, and uses selected samples for further tuning.

Google reports near perfect performance on the BABILong benchmark with contexts up to 10 million tokens. The framework also showed little performance loss while learning unseen languages sequentially and scored 48.9% on a single task factual knowledge test and 46.2% during continued pretraining. On mathematical reasoning tests, the enhanced base model scored 79.2% on AIME 2024, 69.0% on AIME 2025, and 46.1% on HMMT 2025.