How do you give an AI chatbot "memory"?

When I started building the first AI chatbot for a client, I assumed the model “remembered” the conversation the same way a person would. It doesn’t, and understanding that changes quite a bit about how you design this.

A language model doesn’t store anything between one question and the next by default. Every time you send it a message, you’re actually sending it the entire previous conversation again (or a summary of it) along with the new message. What we call “memory” is, in most of the chatbots we use daily, simply resending the full history every time you write something.

That has a practical limit: the longer the conversation, the more text has to be resent, and models have a cap on how much text they can process at once, the so-called “context window.” For long conversations, some tools summarize the old parts and keep only the recent bits verbatim, so they don’t run out of space.

Then there’s real memory, the kind that persists across separate conversations, an assistant remembering something you told it last week. That isn’t magic from the model — it’s a separate database where specific facts are stored (your name, a preference, a detail about your company) and then injected into the model at the start of every new conversation, like a cheat sheet.

For a customer service chatbot, this translates into a very concrete design decision: which customer data is worth storing and re-injecting (their name, an order’s status) and which doesn’t need to be, and shouldn’t be, kept permanently. Storing too much is also a privacy risk, not just unnecessary technical overhead.

The first time a client asked us why the chatbot didn’t remember what it had been told the day before, I had to explain that the answer wasn’t a bug — it was that nobody had asked the system to keep that information from one day to the next. It can be done, but it has to be designed that way on purpose.