How do we give an AI agent memory across sessions?

Giving an agent memory across sessions means persisting relevant state outside the model's context window and retrieving it at the start of each new session, since the model itself has no memory between separate API calls. The most common pattern stores conversation summaries, user preferences, and past task outcomes in a database, then uses semantic search over a vector store, such as pgvector, Pinecone or a managed alternative, to retrieve only the memories relevant to the current task rather than replaying an entire history. A second layer of memory, often called working or episodic memory, tracks the state of an in-progress multi-step task so an agent can resume after an interruption, which frameworks like LangGraph support through checkpointed graph state. Effective memory systems also need a decay or curation strategy, since unmanaged memory grows noisy and can degrade the agent's accuracy if irrelevant or outdated facts get retrieved and treated as current. Access control on stored memories matters too, particularly when different users or departments should not see each other's history. Nanobase AI implements this memory layer using the client's existing database and vector infrastructure rather than introducing a new system to maintain.

Memory is not one system, it is four

Teams building agent memory for the first time often reach for a single vector database and assume that solves persistence, but production agent memory is really four distinct layers with different lifespans, storage patterns and retrieval mechanics, borrowed loosely from how cognitive science categorizes human memory. Conflating them is why some "agent with memory" implementations feel simultaneously forgetful about important context and cluttered with irrelevant old details. Each memory layer answers a different question about what the agent needs to remember and for how long.

Working memory is checkpointed state, not a memory system at all

The layer people usually mean when they first say "the agent forgot what we were doing" is actually working memory: the state of an in-progress multi-step task. This is not retrieved from a database in the traditional sense; it is checkpointed as the task progresses so the agent can resume after an interruption, a human approval pause, or a process restart, picking up exactly where it left off rather than restarting the task from the beginning. Frameworks with explicit graph-based state management handle this natively; custom orchestration loops need to build this checkpointing deliberately.

Semantic memory needs curation, not just accumulation

Semantic memory, the durable facts and preferences retrieved by similarity search, is where most agent memory systems degrade over time if left unmanaged. Every new fact stored without a decay or review mechanism increases the odds that an outdated or contradicted fact gets retrieved and treated as current, and unlike a human, the agent has no innate sense that a six-month-old preference might no longer apply. A memory system that only grows and never curates will eventually make the agent less accurate, not more, despite having more information available. Practical curation includes expiring or down-weighting older entries, deduplicating near-identical facts, and periodically reviewing what gets retrieved most often to catch systematically wrong entries early.

Building the memory layer step by step

The four layers do not need to arrive at once; building them in order of actual need avoids maintaining infrastructure nothing yet uses.

1. Start with working memory only: checkpoint task state so interruptions and approval pauses do not force a restart.
2. Add episodic memory for any workflow where past interactions with the same user or account meaningfully change how the current one should be handled.
3. Add semantic memory once you have identified specific, durable facts worth retrieving, rather than storing every conversation wholesale.
4. Treat procedural memory as a slower-moving asset, refining tool descriptions and prompts based on patterns across many episodic records rather than updating it per session.

Access control is a memory design decision, not an afterthought

Memory that spans users or departments raises access questions that are easy to overlook until an incident forces the issue: a support agent's semantic memory should not surface one customer's account details while helping a different customer, and a department-specific agent's episodic memory should not leak across departments that should not see each other's history. Scoping memory retrieval by the same permission boundaries that apply to the underlying data, rather than treating the memory store as a separate, unscoped system, avoids this becoming a compliance gap discovered too late.

Frequently asked questions

Do we need a vector database for agent memory, or will a regular database work?

A vector database or vector-search extension is useful specifically for semantic memory's similarity-based retrieval, but working memory and episodic memory are typically better served by a regular structured database, since they are retrieved by task ID or recency rather than semantic similarity. Most production systems use both, not one or the other.

How much memory should be loaded into context for a given task?

Only what is relevant to the current task, not the full history available. Retrieval should rank and filter by relevance, then load a bounded amount, since flooding the context window with tangentially related memories degrades the model's focus on the actual task at hand.

Can memory cause an agent to behave inconsistently over time?

Yes, if outdated or contradictory facts accumulate without curation, the agent can retrieve different, conflicting memories for similar queries at different times. This is why a decay or review policy is a core design requirement, not an optional refinement added later.

Is procedural memory the same as fine-tuning?

They overlap but are not identical. Procedural memory can be as simple as refining a prompt or tool description based on observed patterns, while fine-tuning is a more formal, model-level update. Both aim at the same goal, encoding "how this task should generally be handled," at different levels of effort.

How Nanobase AI helps

Nanobase AI, a Silicon Valley enterprise AI engineering company, implements this layered memory architecture using a client's existing database and vector infrastructure rather than introducing a new system to maintain, with access control scoped to match existing data permissions. This connects to the broader question of giving agents persistent state through orchestration and to retrieval-augmented generation as a design pattern.

Ready to discuss your project? Contact Nanobase AI or email hello@bumu.tech.