LLMs Don't Remember You!

When you interact with a tool like Claude or ChatGPT, it feels like it remembers what you're talking about from exchange to exchange, and sometimes across chats as well, just like a real person would. What would it be like if ChatGPT didn't remember what you're talking about from line to line?

We went through some of this content during a video meeting, in case you’d like to explore it that way.

Let's say you were in the market for a new TV and started talking with ChatGPT about this.

This feels robotic. And, spoiler alert, this is the extent to how LLMs work. While it's impressive that the bot can understand text and respond appropriately to individual inputs, its memory gets wiped from line to line. It can't handle anything beyond a single request. This is essentially how Siri works currently: set a timer, show me a contact—but these are one-off interactions.

Can you think of a way that you could "hack" this to make it more useful if you're the user who is trying to buy a TV?

You'd probably write a new line connecting the dots for the bot. You might go back to the chat and copy and paste together your full request, using some of the information you got in dribs and drabs from the bot itself. Youd start to link together the information you want with the information you already learned to meet your objective:

If LLMs don’t remember anything from exchange to exchange, then how does ChatGPT remember what I previously wrote? It’s essentially doing this “hack”.

Once again, ChatGPT, Gemini, Claude–they are not LLMs. They are programs that interact with an LLM, but the LLM is just one of many external mechanisms that these programs interact with. And whenever you type in a new line in the chat, these applications quietly add in a lot of your previous exchanges. This is called context: the amazingly useful trick these applications use to simulate human interaction.

If we pasted the entire chat history back into the LLM, yes we would get our results, but those exchanges would also get super heavy really quickly and burn an enormous number of tokens. So these applications have figured out all sorts of ways to cleverly optimize this process. There are a number of different techniques being used, but we'll cover those in another article.

Meanwhile, I hope this demystified how it is that ChatGPT appears to remember you! It's not a magical genie. It's an application that helps you interact with an LLM–and instead of leaving you with a robot that has amnesia on every turn of the chat, it quietly packages up your entire history and makes it appear that youre interacting with something that remembers!