Every agent product now advertises memory, and a whole category of vendors has formed to sell it to you as a component, the way you would buy a cache or a queue. The pitch works because the gap is real: an assistant that forgets everything between sessions feels broken, and the fix sounds like infrastructure. But I have watched teams bolt a memory product onto an agent and get something worse than forgetfulness: an assistant that confidently remembers things that stopped being true in March. The infrastructure was fine. Nobody had decided what deserved to be remembered.
That is the thesis, so I will say it plainly. Memory is four decisions: what to write, how to store it, what to read back, and when to forget. Storage is the only one of the four that is infrastructure, it is the easiest of the four, and it is the only one anyone is selling. The other three are product policy, they are specific to what you are building, and no framework can make them for you.
Where Memory Actually Lives
Start from what the model actually is. The weights are frozen at training time; nothing you do at runtime changes them. The only thing the model ever sees is the context window, assembled fresh on every call, and what goes into that window is entirely your application's doing. The window empties when the session ends.
So "memory" cannot be a property of the model. It is, with no exceptions, something your application persists outside the window plus the policy that brings it back in. A store, a write path, a read path. Every memory system ever shipped, from a user-preferences column to the fanciest graph-backed product, is this one shape, and naming the shape is useful because it tells you where the difficulty lives: writing and reading are policy, and policy is where the failures are.
Four Kinds of Memory, All of Them Boring
The literature borrows its taxonomy from cognitive science, which sounds grander than the implementations deserve. Mapped to actual storage, the four kinds are mundane:
| Kind | What it holds | What it actually is |
|---|---|---|
| Working | The current task: plan, decisions, confirmed facts | The context window, plus a scratchpad file the agent maintains |
| Episodic | What happened: past sessions, actions, outcomes | An append-only log |
| Semantic | Facts that persist: preferences, entities, constraints | A structured document or a table of records |
| Procedural | How to do things: workflows, house rules, learned fixes | Instructions and skills, versioned like code |
A log, a document, some prompts. The one genuinely new engineering problem in that table is deciding what flows between the layers: which episodic events deserve to become semantic facts, and which repeated fixes deserve to become procedure. That flow is the write path, and it is where the whole game is.
The Write Path Is the Product
The naive policy is to write everything: embed every conversation turn, stuff it in a vector store, retrieve against it later. This fails slowly and then suddenly. The store fills with chitchat, abandoned plans, and corrections of things it also still contains, and retrieval starts surfacing three-month-old context with the same confidence as yesterday's. You have not built memory; you have built a hoard.
The standard upgrade is extraction: have a model read each session and write down the facts worth keeping. This is the right idea and it is also where the subtle failures move in, because extraction is generation, and generation has an error rate. A fact extracted slightly wrong does not fail once, it fails on every future session that retrieves it, and the user watching the agent repeat the same wrong belief has no idea where it came from. Extraction errors compound in a way that single-call errors do not.
And then there is the problem that separates real memory systems from demos: supersession. The user moved to Denver. An append-only store now contains "lives in Austin" and "lives in Denver," both retrievable, and which one surfaces depends on embedding luck. The write path has to treat facts as updatable state, not as a feed: detect that the new fact contradicts an old one, replace rather than accumulate, and keep the timestamp and source on everything so conflicts can be resolved by recency and provenance instead of by cosine similarity. Update-on-write is annoying to implement and it is not optional. Appending contradictions and hoping retrieval sorts it out is the single most common way agent memory goes bad.
Reading Memory Is RAG with a Messier Corpus
The read path gets the glamour, and there is less to say about it than the marketing suggests, because reading memory is retrieval, full stop. Everything in the RAG failure catalog applies unchanged: top-k returns k whether or not anything relevant exists, stale entries get served with a straight face, and an embedding model upgrade quietly severs the new queries from the old vectors. If anything, memory is the harder retrieval problem, because the corpus is self-authored, contradiction-prone, and growing without an editor.
Two reading decisions matter more than the index technology. First, what loads unconditionally versus what gets retrieved: a compact profile of durable facts is small enough to simply always include, and the systems that feel telepathic mostly do this, reserving search for the episodic long tail. Second, how candidates are scored: pure semantic similarity is the wrong ranking for memory, because when relevance ties, you want recent over old and important over incidental. Recency, importance, relevance, weighted together, is a ranking that has been reinvented by roughly everyone who has built one of these, which is a good sign it is correct.
Forgetting Is a Feature
Nobody budgets for deletion, and memory is the part of the stack where that bites hardest. Information ages at different rates: a name is durable, a job title has a half-life, "is working on the migration" is stale in a month and poisonous in six, because an agent acting on expired context does not look forgetful, it looks wrong. Expiry, supersession, and periodic review are not hygiene chores, they are what keeps the store trustworthy enough to be worth reading.
There is also the version of this with lawyers attached. A memory store is PII concentrate: it is, by design, the distilled personal facts of your users, which makes it the first place a deletion request, a breach, or a subpoena becomes expensive. Memory that users can see, correct, and delete is both the compliant shape and, conveniently, the debuggable one. If you cannot answer "what does the system believe about this user and why," you have built something you cannot operate.
The Reference Design That Keeps Winning
Here is the design I keep arriving at, and keep seeing other practitioners arrive at independently, which is a few hundred lines of code plus discipline. One structured profile document per user or project, holding the durable facts, loaded every session, updated by supersession with provenance on each entry. One append-only episode log, summarized at session end, searchable but never authoritative on its own. One small index over both for the long tail. A write policy that states, in words a human can argue with, what qualifies as worth remembering. And a review loop, because a store nobody reads is a store nobody trusts.
The pattern is almost embarrassingly file-shaped, and that is not an accident. Files a human can open are memory a human can audit, and the audit is what keeps extraction errors and stale facts from compounding quietly. The tooling I use daily works exactly this way, one fact per entry with an index loaded up front, and the most useful property it has is not recall, it is that I can read it and fix it.
Measure it like you would measure retrieval, because it is retrieval: a golden set of sessions with known-correct recalls, checked on a schedule, with the same eval discipline you point at everything else. Memory quality drifts, and un-evaluated memory drifts invisibly.
When a Framework Earns It
None of this means the memory products are useless. Multi-tenant isolation, shared memory across a fleet of agents, cross-session consistency at real scale, managed infra you would rather not run: those are honest reasons to buy. But evaluate the products as what they are, databases with an opinionated write policy attached, and interrogate the policy, because it is the part you are actually adopting. How does it detect contradictions? What carries provenance? What expires, and when? If the answers are "embeddings will sort it out," you are buying the hoard with better marketing.
The window will keep growing and the frameworks will keep multiplying, and the four decisions will not move: what to write, how to store it, what to read back, when to forget. Decide them on purpose. The storage was never the hard part.
FAQ
What is agent memory?
Anything an agent persists outside the context window plus the policy that brings it back in. The weights are frozen and the window empties between sessions, so memory is always a store, a write path, and a read path. The storage itself is ordinary: logs, documents, a small index.
How is agent memory different from RAG?
The read path of memory is RAG and inherits every RAG failure mode. What memory adds is the write path: deciding what to record, updating facts instead of appending contradictions, and forgetting on purpose. That policy is where memory systems succeed or fail.
Do agents need a vector database for memory?
Usually not at first. A structured profile that always loads, an append-only episode log, and a small index cover most products. A vector index earns its place when episodic recall over a large history becomes a real requirement.
How do you keep memory from going stale?
Supersede instead of append, carry provenance and timestamps, expire what ages poorly, keep the store human-readable and user-visible, and eval recall on a schedule. Stale memory is worse than no memory, because it is wrong with confidence.