Memorymemoryarchitecturevectorkey-valuegraph

Memory Architectures

A memory system is a write path (extract → dedupe → store) plus a read path (retrieve → rank → inject) over a store chosen for the access pattern — key-value, vector or graph — always scoped per user.

Interview question
Progress

The write path

Memory is written, not accumulated. After a turn or a completed task, an extraction step decides what — if anything — is worth keeping: a small model call with a JSON schema (Structured Outputs) that returns candidate facts and episode summaries, each with a confidence and a source span. Most turns yield nothing, and that is correct.

Candidates then pass through dedupe and reconciliation: is this fact already stored? does it contradict an existing one (plan changed from Pro to Enterprise)? Reconcile by key for semantic facts (newer wins, keep history), by similarity threshold for episodes (cosine > 0.92 → merge), and only then store with owner, timestamp, source and TTL.

Write path
yesnoTurn / task resultExtract candidatesWorth keeping?Dedupe + reconcileDiscardStore (scoped, timestamped)
UserLLMAgentToolDataDecisionHumanGuardrail

The read path

On each call, the read path decides what memory enters the context. Retrieval pulls candidates: semantic facts by exact key for the current user, episodes by hybrid search (vector similarity plus recency and tag filters — the same machinery as Dense, Sparse & Hybrid Retrieval and Metadata Filtering). Ranking orders them by a combined score — relevance, recency, confidence — and cuts at a token budget (Token Budgets). Injection renders the survivors into a labelled section of the context, positioned per Context Ordering & Lost in the Middle.

The read path is where "more memory is better" dies. Injecting twenty loosely related episodes buries the two that matter. A good read path returns three items with high precision; a bad one returns twenty with high recall and a worse answer.

Read path: exact facts always, episodes only above a relevance floor, everything within a budget.
1def read_memory(user_id: str, query: str, budget_tokens: int = 600) -> str:
2    facts = kv.get_all(user_id)                                   # semantic: exact, cheap
3    eps = vec.search(query, filter={"user_id": user_id}, k=10)    # episodic: candidates
4    eps = [e for e in eps if e.score >= 0.80]                     # relevance floor
5    eps.sort(key=lambda e: 0.7 * e.score + 0.3 * recency(e.at), reverse=True)
6 
7    lines = [f"- {k}: {v}" for k, v in facts.items()]
8    for e in eps:
9        line = f"- [{e.at[:10]}] {e.summary}"
10        if count_tokens("\n".join(lines + [line])) > budget_tokens:
11            break
12        lines.append(line)
13    return "## What we know about this user\n" + "\n".join(lines)

Storage options

Choose the store by access pattern, not by fashion. Most production memory systems use two of these, and start with the first.

  • Key-value / relational — semantic facts and profiles. Lookup by (user_id, key), trivial to update, audit and delete. Postgres or Redis. Start here; it covers more than people expect.
  • Vector store — episodes and free-text notes retrieved by meaning. Needs embeddings (Embeddings), a metadata filter on user id, and a relevance floor. Same tooling as RAG (Vector Storage).
  • Graph — entities and relationships ("Anna is admin of tenant 4471, which uses Okta"). Worth it only when multi-hop questions are common; otherwise a few joined tables do the job with less operational cost.

Scoping and isolation

Every memory record carries an owner, and every read filters by it — enforced at the storage layer (row-level security, a mandatory filter in the query builder), not by the prompt. A vector search without a user_id filter is a data leak waiting for the first similar query from another tenant.

Scopes usually form a small hierarchy: system-wide procedural memory (playbooks), tenant-level facts (org settings), user-level facts and episodes. Reads union the scopes the caller is entitled to; writes go to exactly one. Treat memory contents as untrusted at read time — a stored "fact" that says "ignore previous instructions" is an Indirect Prompt Injection payload with persistence (Secrets and Untrusted Output).

Key points

  • Write path: extract candidates with a schema, gate, dedupe and reconcile, then store with owner, time, source and TTL.
  • Read path: retrieve by key and by hybrid similarity, rank by relevance × recency × confidence, cut at a budget, inject labelled.
  • Most turns should write nothing; precision on read matters more than recall.
  • Key-value first, vector for episodes, graph only for genuine multi-hop needs.
  • Scope enforcement lives in the store, not the prompt; memory is untrusted content at read time.

When to use — and when not to

Use it when
  • Cross-session personalisation or continuity is a product requirement.
  • The agent must reconcile changing facts (plan tiers, preferences, ownership) over time.
  • Past incidents should inform current diagnosis and the volume justifies retrieval.
Avoid it when
  • The facts already live in a system of record you can query — a tool call beats a stale copy.
  • Low-volume, short-lived use cases where a session store and a profile table suffice.
  • When you cannot run evals on the read path — unmeasured memory quietly degrades answers.

Failure modes

  • Extraction promotes everything; the store fills with trivia and retrieval precision collapses.
  • No reconciliation: two contradictory facts are both retrieved and the model picks one at random.
  • Vector search without a user filter returns another tenant's episode.
  • Ranking by similarity only; a two-year-old episode outranks last week's identical issue.
  • Stored text treated as trusted instruction and replayed into every future context.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Two probabilistic steps (extraction, retrieval) per turn; each needs its own eval.

← Memory TypesMemory Pitfalls →