Twenty-five years after my last visit, I can still recollect the seating inside the Lotus Temple, without a photograph or other persisted artifact to search. Could an AI agent keep its experience that long? Not by default: each new session starts with no memory of the last. To remember, an agent must preserve a trace, in an external record or in its learned parameters.

For a research agent on a week-long project, the challenge arrives within days: it must resume after a pause, incorporate new evidence and remember why an earlier approach failed, often while delegating to other agents. Documents can be retrieved on demand; what matters is whether the experience survives. Three approaches compete to retain it: a longer context window, organized memory such as notes or a wiki, and learned memory in the model’s weights, each shifting cost to a different part of the infrastructure.

What a Longer Window Buys

Context length describes how many tokens a model can examine at once. The window is working memory: experience reaches a later session only if saved, as transcripts, notes or curated files, and reloaded. Windows grew from GPT-3’s 2,048 tokens in 2020 to Claude’s 200K in 2023 and Gemini’s 1M in 2024; Anthropic made 1M generally available in March 2026. Meta and Mirendil report 10M support, and Magic reported 100M. But capacity is not use. At 10M tokens, Google reported over 99% recall of a single planted fact; Mirendil’s base model scored 100% on a similar test but 45% on multi-hop questions, which chain several facts. Neither shows whether an agent revises a conclusion that new evidence contradicts.

A longer window’s main appeal is that it preserves evidence whose importance becomes clear only later: a clause in a contract review, an anomalous experiment, or a constraint in a code migration. It can delay summarization and let the model compare more source material directly.

Benefit of a longer windowRemaining limitation
Revisit details that an early summary would omit.More active tokens increase memory traffic and processing cost.
Compare evidence spread across a large collection.Placement matters: Lost in the Middle found weaker use of facts in the middle of long inputs.
Retain earlier findings alongside later corrections.Extra space does not establish which account is current or correct.

From Retained Evidence to Reusable Knowledge

Organized memory takes a different route: a recent study uses dreaming, an offline review of past sessions, to correct notes and build wiki pages without changing model weights. A wiki pattern describes this representation: interlinked pages compiled from retained raw sources. On 25 questions over a 420M-token corpus, wiki plus search met 61% of graded criteria versus 53% for search alone, after 128 dreaming sessions. Those sessions are offline compute, spent before any question arrives; the infrastructure question is how often their output is reused.

Figure 1: a three-level memory stack. New input enters the context window, is absorbed into fast memory, and during sleep is distilled into slow, durable memory; a dream loop rehearses from slow memory back into fast memory. An arrow beside the stack marks memory getting slower and more durable from top to bottom.

Figure 1. Memory at three speeds, after Hope and the sleep framework. Sleep distills fast memory into slow, durable layers.

Click on the figure to zoom in.

Learned memory goes a step further, from notes to weights: through distillation, a model is trained to reproduce useful behavior without the original material in context. A sleep framework applies this to Hope, a model whose memory is a hierarchy: fast components adapt quickly and slow ones change rarely, preserving knowledge (Figure 1). During sleep, the model distills fast, recent memories into new low-rank parameters in slower layers, then dreams, using reinforcement learning to generate practice data that rehearses new knowledge and refines existing skills. On BABILong’s synthetic tests of chaining scattered facts, a Hope model with sleep, fine-tuned for the benchmark, stayed near-perfect up to 10M tokens: retention on one task, not recall of arbitrary experience.

Learning and reasoning can reinforce each other. SEAL rewards self-edits, the training data and update settings a model writes for itself, when the resulting update improves later answers, including on selected ARC puzzles. Self-Distillation Fine-Tuning trains a model on its own answers, steering them toward what the same model says after seeing expert demonstrations. Because the targets are its own outputs, updates stay close to current behavior, which RL’s Razor links to less forgetting. Taught science questions, tool use and medical questions in turn, the model kept each earlier skill; with ordinary fine-tuning, each skill degraded once training moved on. The risk is self-bias: the authors note that the model can pick up spurious patterns from its own guided answers, and repeated training on generated text can cause model collapse, where rare content vanishes. A newer study argues that on-policy training is not required: it fine-tunes a checkpoint from before post-training and grafts the scaled weight change onto the deployed model, learning new tasks while matching or improving old ones.1

Notes, wikis and learned weights cut rereading, but they keep only what seemed worth keeping; an unanticipated question may hinge on a detail they dropped. In the dreaming study described above, the wiki alone met 48% of criteria; adding search over the raw corpus lifted that to 61%. Long context helps such questions mainly through coverage, not deduction: in a Google DeepMind comparison, long-context models beat retrieval on average when given enough budget, as retrieval missed the evidence for multi-step, general and implicit queries. Yet, BABILong found that models effectively used only 10-20% of their context, and accuracy fell as reasoning steps grew. The choice shifts cost between serving each request and offline compute.

Infrastructure Implications

An agent’s full history can sit in durable storage; only the tokens in active use need GPU memory, and 10M of them change the economics. As a model reads, it caches the attention keys and values of every token in every layer, so they need not be recomputed for each new token. For an illustrative 8B model, this key/value (KV) cache takes 131,072 bytes per token: 2 (a key and a value) × 32 layers × 8 KV heads × 128 dimensions × 2 bytes per 16-bit number (Figure 2). That is 131 GB at 1M tokens and 1.31 TB at 10M for one conversation, excluding weights, more than the combined memory of sixteen 80-GB H100s. A KV cache grows with the number of layers, KV heads and dimensions per head, not with the parameter count. Llama 3 70B has almost nine times the parameters of Llama 3 8B but needs only 2.5 times the cache per token (327,680 bytes), because it keeps the same 8 KV heads of 128 dimensions; only its layer count grows, from 32 to 80.

Animation of one decoding step: a token moves through Layer 1, Layer 2, Layers 3 to 31 and Layer 32, then through output projection and token selection to the next token. In each layer it first writes its key (purple) and value (green) into that layer's KV cache; then its query attends to past and current keys and values, which are read back up. Two equations count in step: 2 (key and value) times 32 layers times 8 KV heads times 128 dimensions times 2 bytes gives 131,072 KV bytes added by this token, and 131,072 bytes per cached token times 10M tokens in context gives about 1.31 TB of KV bytes read in this step. Writes stay constant per token; reads grow with context.

Figure 2. One decoding step of the illustrative 8B model, with full attention and a 16-bit KV cache (layers 3-31 condensed; timing illustrative). In each layer, the current query attends to past and current keys and values, and the new ones are kept for future steps. Actual memory and network traffic depend on the implementation.

Click on the animation to zoom in.

With the same architecture and precision, going from 1M to 10M tokens means roughly 100× the attention arithmetic during prefill (reading the prompt) and 10× the attention work and KV reads per generated token; these are scaling estimates, not latency predictions. Systems that run prefill and decoding (generating the answer) on separate GPUs must also copy each request’s cache between them: 1.31 TB at 10M tokens. A paused agent ties up its cache too; evicting it frees GPU memory, but resuming then means reloading it from CPU memory or SSD, if it was saved there, or recomputing it from the text. Three practices keep these costs in check:

  • [Use GPU memory selectively] Prefix sharing keeps one copy of a history that several requests share; each branch needs its own entries once it diverges. Quantization to 2 bits per value shrinks the cache.
  • [Treat the KV cache as a memory tier] A saved cache lets a paused agent resume without rereading its history. DeepSeek-V4.1-Flash, built for input-heavy agent workloads, shares cache entries across layers and stores them in 4 bits, cutting its GPU cache to 890 bytes per token, under 1% of the 8B example’s. It keeps caches on SSD for at least 72 hours; a request that hits one reuses the full prefix. For a pool of coding agents, CPU memory sized to their reusable context cut recomputed prompt tokens by 93%. Letting agents drop old tool outputs from their cache, rather than summarize their history and force a re-read, cut cost by 27.7% and slightly raised scores.
  • [Budget compute and network bandwidth] FlashAttention reduces GPU memory traffic but keeps attention’s quadratic arithmetic. Ring Attention spreads one long sequence across many GPUs, which must exchange data as they compute. Placement and interconnect bandwidth matter alongside GPU capacity.

Cartridges shows the tradeoff between paying once, offline, and paying on every query. With model weights frozen, it trains a compact KV cache for one corpus. Matching in-context quality on its benchmarks, it used 38.6× less cache memory and reached 26.4× higher peak throughput. But training one cartridge for an 8B model took about 30 minutes on eight H100s in an unoptimized implementation, and its test corpora spanned 100K to 484K tokens, far short of 10M. The one-time training pays off when many queries hit a rarely changing corpus, such as a textbook or signed contracts; each change means retraining.

Learned memory pays in training instead. SEAL applies each self-edit as a LoRA update to small adapter weights; its training loop took 30-45 seconds to fine-tune and evaluate each candidate. Any weight update, even to adapters, brings training into the serving stack: backward passes to compute gradients, plus activation memory, gradient and optimizer state and saved adapter versions. The dreaming study’s offline reviews add background inference; the sleep framework adds training and new parameters every cycle. Updates also age stored KV caches: in a study of cache reuse across model updates, old caches kept useful task information but did not faithfully reproduce the updated model’s behavior, so caches must be versioned with the weights.

When agents share memory, they must coordinate writes. In a contract experiment from the same dreaming study, agents writing to one shared table overwrote one another’s notes, and only 41 of 57 graded facts survived; one page and one writer per contract, with terms quoted verbatim, preserved all 57. Longer context cannot fix write conflicts. Separate workspaces and controlled merging protect shared state; source links and observation dates distinguish an old finding from a current fact.

Looking Forward

The right yardstick is total cost per successfully completed task, with one-time offline compute spread over the tasks that reuse it: the cost-controlled evaluation that AI Agents That Matter calls for. Test systems on a task like the week-long research project described at the start: pause and resume with new evidence, revisit a decision, combine agents’ findings. Score memory with LongMemEval’s knowledge-update and multi-session accuracy, and with per-question forgetting and backward transfer: answers that flip from right to wrong (forgetting) or from wrong to right (backward transfer) after an update. Score serving with time-to-first-token (TTFT), time-per-output-token (TPOT), goodput (requests meeting both targets), KV cache hit rate, peak GPU memory and interconnect traffic as history grows.

Takeaways

  • [Context is working memory] Only what is saved, cached or learned outlives the session.
  • [Retrieval is not reasoning] Finding a fact in 10M tokens is not revising a conclusion.
  • [Each design moves the bill] Long context pays at serving time; notes, caches and weight updates pay in offline compute and storage.
  • [The KV cache is becoming a memory tier] Stored for days, it lets agents resume without rereading, until the weights change.
  • [Reuse picks the winner] Offline compute pays off only for stable material queried again and again.

Notes

  • 1 Continual learning also requires preserving the ability to learn. Loss of plasticity in deep continual learning showed that ability deteriorating in vision and reinforcement-learning experiments, not LLM reasoning. The newer language-model studies establish bounded experimental gains, not decades of reliable memory. ↩︎