Session, Store or Weights: Where Should Your Agent Remember Things?

One of my agents called my cat “her” once. He’s a boy. I corrected it, it fixed the draft, and the next evening’s run would have made the exact same mistake, because nothing from that conversation survived the session.

What fixed it for good was one line in a markdown file called learnings.md: the cat is male, he/him, always. The agent reads that file before it writes a single word. It never learned anything about my cat; it just looks it up every time.

I’ve been chewing on memory since, and the way I keep explaining it to myself is the brain. Not because a model is a brain (it isn’t, and real brains don’t store vectors), but because the brain does two jobs that map surprisingly well onto an AI stack.

Two jobs: storing and connecting

Your brain stores memories, and it connects them. A song, a street, the summer you first heard it: keeping each one and linking them together are different jobs.

An AI stack splits them the same way. Storing is the knowledge base: documents, databases, a folder of notes. Connecting is the model. Embeddings turn text into vectors, related ideas land close together in that space, and the network does the linking. (If vectors still feel abstract, I wrote a plain guide to vectors that starts with a map direction.) It works a bit like association, which is why it feels like recall even when it’s math.

Here’s the whole map in one table, before we walk through it:

Short-term memory: the session

Short-term memory is the context window and the session state: the chat so far, every tool result fetched this turn, whatever’s sitting in a cache. It stays alive as long as something is using it, even across threads in the same session, and when the thread ends, it’s gone.

That forgetting is a feature. Context costs tokens on every single turn, so carrying everything forever gets expensive fast. Old tool results go stale. And a session that forgets is a session that can’t leak one user’s details into the next user’s chat.

It’s also the memory that hurts most when it disappears by accident. In Where Does Turn 12 Go? I wrote about an AgentDesk session landing on a cold orchestrator that has to rebuild everything from history. The fix there was keeping session context in a shared store, so it survives a server going away without becoming permanent. Short-term memory can be durable for the length of a session and still be short-term.

Long-term memory, level 1: look it up

The first kind of long-term memory lives outside the model: a vector store, a knowledge base, an MCP server, a tool the agent can call. When the agent needs something, it fetches it and drops it into short-term memory for this turn. That’s RAG, plainly.

My learnings.md file is the scrappy version of this. A rule I give my agent once (the cat is male, hearts are always 🖤) holds for every draft after, because the file gets read at the start of every run.

Lookup memory is fresh: change the file, and the next run knows. It’s citable: you can point at exactly where a fact came from. And it’s cheap to fix when it’s wrong. What it costs is the fetch itself, a little latency and a few tokens every time, plus the risk that retrieval grabs the wrong chunk and the agent confidently uses it.

Long-term memory, level 2: learn it

The second kind lives inside the model, in the weights. That’s fine-tuning. No lookup at all; the model just knows, the way you know how to ride a bike.

In June I fine-tuned Llama 2 on 7,431 of my own tweets, on my laptop, and named it Khushi. What it picked up was the shape of my writing: casual and short, because every tweet was casual and short. It can’t write my tech threads because it never saw one. And when I asked it, in chat format, what it did today, it opened with “I’m just an AI, I don’t have personal experiences or emotions.” Eight epochs of me, and the base model’s habits still won, because I trained it on raw tweets and then talked to it through a chat template it had never practiced.

That’s the honest lesson about learned memory. Fine-tuning is good at style, format and behavior, and much less reliable at facts. It’s slow to update (the last epoch alone took about ten hours on my laptop), and it’s hard to make it forget. You can’t open the weights and delete one line.

Where should a fact live?

This is the rule I use now:

1. If it only matters right now, keep it in the session and let it go.
2. If it changes, or someone needs to check where it came from, retrieve it.
3. If it’s a stable skill, tone or format, that’s when fine-tuning earns its cost.

“The cat is male” is a stable fact, and I’d still never train it in. It belongs in a file I can read and edit, because facts change and I’d rather fix one line than run a fine-tune overnight.

The same goes for prompts, by the way. If you find yourself retyping the same correction into every session, that’s a fact stuck in the wrong drawer, and a cousin of the problem in Direction vs. Destination: the agent can’t hit a target it was never given, or was given once and then lost.

What should it forget?

The part people skip is the forgetting. Every memory you add is one you have to keep correct and keep private: one user’s details shouldn’t drift into another user’s session, a stale price shouldn’t outlive the week it was true, and anything you train into the weights, you’ve basically agreed to keep.

So before giving your agent a better memory, decide what it’s allowed to forget, and where.

The short version

Most agent memory problems are filing problems: the fact was real, it just lived in the wrong place.

Where does your agent’s memory live right now: in the prompt, in a store it can search, or in the weights? I’d genuinely like to know what’s working for you.