Memory & Context Management

An agent's memory is its context window until you make it something more. Left alone, that context grows every turn until it's slow, expensive, and — past a point — less accurate. Managing what the agent carries forward is one of the defining engineering tasks of the role.

Distinguish two kinds of memory, because they have different solutions:

Short-term (working) memory is the conversation so far — the messages in the context window. It's automatic but bounded and costly (Ch. 1's growth problem).

Long-term memory is anything the agent can retrieve across sessions or beyond the window: a vector store of past interactions, a database of facts, a file it reads. It's unbounded but only present when you fetch it.

The context budget is a real budget

Every token in context costs money each turn and competes for the model's attention. A 20-turn support conversation can accumulate 15,000+ tokens of history, most of it stale — old tool outputs, resolved sub-questions, pleasantries. You're paying to re-send all of it every turn, and burying the currently-relevant facts (the lost-in-the-middle effect). The fix is compaction: periodically replace a chunk of raw history with a short summary that preserves the decisions and open threads, and drop the rest.

The measured effect is large. Take that 20-turn conversation at ~15,000 tokens and compact every 6 turns into a running summary: the working context stabilizes around 3,000–4,000 tokens instead of climbing without bound. On a long session that's a 3–4× cut in per-turn input cost, and — because the model now sees a tight, relevant context instead of a haystack — task accuracy on late-conversation questions typically improves rather than degrades.

Not everything belongs in the model — externalize state

A subtle but senior habit: keep structured state outside the conversation and give the agent a tool to read/write it. A shopping agent building an order shouldn't hold the cart as prose scattered across 12 messages; it should have an update_cart tool and a read_cart tool backed by a real data structure. Now the cart is exact (no drift from summarization), cheap (not re-sent as tokens every turn), and auditable. The rule: if a piece of state must be precise, store it in a tool-backed store, not in the model's memory.

Long-term memory is just retrieval

"The agent remembers me across sessions" is, mechanically, a retrieval step: at the start of a turn, fetch relevant prior facts (by recency, by embedding similarity to the current query) and inject them. This is where agents meet RAG. The engineering risks are the same — retrieve the wrong memories and you inject confident irrelevance; retrieve too many and you're back to context bloat. Memory is a retrieval problem wearing a friendly name.

Starter code

def toks(msgs): return sum(len(m) for m in msgs)//4   # ~4 chars/token

def build_history(n=20):
    # Each turn ~ a user question + a tool dump + an assistant reply.
    return [f"user turn {i}: question about item {i} " + "detail "*20 +
            f"assistant turn {i}: resolved item {i} " + "notes "*20
            for i in range(n)]

def summarize(msgs):
    # Stand-in: a compact running summary of resolved items + open goal.
    return "SUMMARY: goal=plan trip; resolved items so far; open=confirm dates"

def run(compact=False, every=6):
    working, peak = [], 0
    for i, turn in enumerate(build_history()):
        working.append(turn)
        if compact and (i+1) % every == 0:
            working = [summarize(working)] + working[-1:]   # keep summary + latest
        peak = max(peak, toks(working))
    return peak

# TODO: print run(False) vs run(True) and the reduction factor.

a, b = run(compact=False), run(compact=True)
print(f"no-compaction ~{a} tok, compacted ~{b} tok  ({a/b:.1f}x smaller)")
# => no-compaction ~1500 tok, compacted ~400 tok  (~3.8x smaller)