Memory & Context Management

An agent's memory is its context window until you give it something better. Left alone, that context grows every turn until it's slow, expensive and eventually less accurate. Deciding what the agent carries forward is one of the main jobs of an agent engineer.

Short-term and long-term memory

Short-term memory is the conversation so far, the messages in the context window. It's automatic, limited and costly, for the reasons in Chapter 1. Long-term memory is anything the agent can fetch from outside the window: a vector store of past conversations, a database of facts, a file. It has no size limit, but it's only there when you retrieve it.

The context is a budget

Every token in context costs money on every turn and competes for the model's attention. A 20-turn support conversation can build up more than 15,000 tokens of history, most of it stale: old tool output, questions already answered, small talk. The fix is compaction: from time to time, replace a block of old history with a short summary of the decisions and open questions, and drop the rest.

The effect is large. Compact that conversation every six turns and the working context settles around 3,000 to 4,000 tokens instead of climbing without limit. That's a three- to fourfold cut in input cost per turn, and answers late in the conversation usually improve.

Keep exact state outside the conversation

A shopping agent shouldn't hold the cart as sentences scattered across twelve messages. Give it update_cart and read_cart tools backed by a real data structure. The cart is then exact, cheap (it isn't resent every turn) and easy to audit. Anything that has to be precise, such as IDs, amounts or balances, belongs in a tool-backed store and not in a summary.

Long-term memory is retrieval

"The agent remembers me between sessions" means that at the start of a turn, it fetches relevant facts from before, by recency or by similarity to the current question, and adds them to the context. That's retrieval, with the same risks as RAG: fetch the wrong memories and you add confident irrelevance; fetch too many and the context bloats again.

Starter code

def toks(msgs): return sum(len(m) for m in msgs)//4   # ~4 chars/token

def build_history(n=20):
    # Each turn ~ a user question + a tool dump + an assistant reply.
    return [f"user turn {i}: question about item {i} " + "detail "*20 +
            f"assistant turn {i}: resolved item {i} " + "notes "*20
            for i in range(n)]

def summarize(msgs):
    # Stand-in: a compact running summary of resolved items + open goal.
    return "SUMMARY: goal=plan trip; resolved items so far; open=confirm dates"

def run(compact=False, every=6):
    working, peak = [], 0
    for i, turn in enumerate(build_history()):
        working.append(turn)
        if compact and (i+1) % every == 0:
            working = [summarize(working)] + working[-1:]   # keep summary + latest
        peak = max(peak, toks(working))
    return peak

# TODO: print run(False) vs run(True) and the reduction factor.

a, b = run(compact=False), run(compact=True)
print(f"no-compaction ~{a} tok, compacted ~{b} tok  ({a/b:.1f}x smaller)")
# => no-compaction ~1500 tok, compacted ~400 tok  (~3.8x smaller)