If you only have 60 seconds to spend on this, just read these four dot points
At any given point in time, a Large Language Model (an LLM, e.g. GPT 5.6, Fable 5 etc.) only has knowledge of the things it was trained on, and of the things that have been loaded into its context window.
A context window is a document that the LLM’s harness (the program that sits between you and the LLM, e.g. ChatGPT, Claude.ai, Claude Code, Codex, Cursor etc.) is invisibly attaching underneath every message you send. This document contains (among other things) a transcript of every word that was said in every prompt and every response across the entire session (including every document, website, or line of code that the LLM chose to read or write).
This is how the frontier labs (Anthropic, OpenAI) create the illusion of “chat apps”, when in fact, there is no actual chat happening. The LLM has no memory. It is not [really] learning about you and your needs as the chat session goes on. The LLM is simply re-reading the entire “chat history” every single time it prepares a response.
The naive use of LLMs will result in context windows getting massive really fast. Context windows that are filled with unnecessary irrelevant noise degrade the LLM’s performance, and escalate its cost to use. If a worker can manage their context windows intelligently (keeping them small, and focussed on a single task), they will get much better results (write better code, draft up better plans, make better decisions etc.). If a company can make it easy for its workers to do this on aggregate (e.g. by investing in more modular codebases, more agent-friendly design systems, and more-structured organisational memory etc.), it will be at a severe advantage, and this advantage will only compound as LLMs improve.
The easiest way to visualise the problem of LLM memory is with an example from daily life.
Let’s pretend we are using one big ChatGPT chat session to plan out a vacation to a place that we’ve never been before. Pretty soon, this session will get filled up with discussion topics like: where do we stay. How do we get from the airport to the hotel. Where do we want to go. Where are the best places to eat near the places that we want to go to. How do we get from one place to another place. All of these topics will have many decisions, with many options, organised in many ways.
If you’ve ever done this before (or something similar), you might have noticed that as this one big ChatGPT session gets longer and longer, the LLM’s responses get slower, and dumber, and more expensive as you go. If you’re on a free plan, you’re getting capped (or quietly bumped down to a cheaper model which is not very good at reasoning), or if you’re on a paid plan, your token limits will get used up very quickly.
The reason why this is happening is that the LLMs used in ChatGPT (and the ones used in Claude.ai, and Claude Code etc.) don’t have any sort of built-in memory system. Not really.
Every time you send a new message into a ChatGPT chat session, behind the scenes, the ChatGPT app (aka the ‘harness’) is just taking the entire chat session, and secretly copy pasting it underneath your message. Every single question. Every single answer. Every link, every image, every search result, every tool call and every tool response. Every decision and every revised decision. Every ‘that’s a great idea!’. Every ‘you’re absolutely right! I did mess that up completely!’. Every fact and every hallucination. All of that information and non-information is being processed from scratch every single time you drop a new message in the chat. It gets slow and expensive and dumb because this is a lot of information to digest and respond to coherently. And even if you have unlimited time and money, your results are going to get worse and worse because there is a huge volume of irrelevant noise that is becoming salient in the “mind” of the LLM.
You might be thinking ‘ChatGPT actually does seem to remember some things about me between chat sessions though?'. There are two ways that LLM harnesses simulate this “remembering”:
When an LLM infers that the best thing to do is to refer to a previous chat session, it will write some code to search for this session, and its harness will execute this code, and dump that session’s contents into its context window too (@developers: yes, this is more or less a code injection attack that the LLM does to its harness; all LLM tool use is). This process stitches two big dumb noisy context windows together, making one bigger dumber noisier context window. It’s also not really memory. It’s the same pretend memory extended across multiple sessions.
ChatGPT, or Claude, or Claude Code etc. all have tools for writing little ‘memory’ note files for themselves to reference. For web-based apps like ChatGPT and Claude.ai, these files live in their servers and are attached to your account. For local harnesses like Claude Code etc., these files live in a hidden folder on your computer (e.g. if you’re on Mac, go to ~/.claude/projects/<project>/memory in your root user folder, you will need to press cmd + shift + . to see it in the finder app). This is less obviously bad, and technically is a memory system, but it is an extremely brittle and faulty memory system that we should not rely on because:
a) The memory note files are opaque. For Claude Code, you have to go out of your way to see them. They are not version controlled in any way. They cannot be easily shared with others. The LLM makes a judgement call on when to write them and when to update them and what goes in them. For ChatGPT and Claude.ai it’s even worse, because the files are not even on your computer, and not transferable between different LLMs, even though they are effectively just little Apple notes files that the LLM wrote for itself.
b) Because an LLM exclusively knows about things that are in its training data or in its context window, the existence of these little note files must itself be secretly loaded into the context window for every turn of every session (not the full content of each file, just a catalogue of them). This means that a mature Claude.ai account where the user has prompted for 20 different unrelated things could have a catalogue of 20 or more unrelated memory files loaded into context for every single chat session.
Don’t let the name “ChatGPT” trick you. The “chat” is an illusion. It’s not like chatting with a human, who gradually builds up a mental picture of who you are and what you want as the chat session goes on. An LLM has no memory. And so instead of relying on this faked “chat” memory illusion that is built into ChatGPT and Claude, we’re much better off keeping sessions small and focussed, and pointing them at an externalised memory system that we own and control. I am going to show you what that looks like right now.
Let’s do the same vacation planning activity again. Now imagine that, instead of doing the entire vacation plan in one big chat thread, we created a vacation plan Google doc. This doc contains information about where we want to go, when we are going, and a description of the sort of trip we want to have.
We can then open up a ChatGPT session, paste a link to our Google doc (and enable the Google drive MCP / “connector” in the settings menu), and ask it to help us figure out our flights. When we are happy with the flights, we can ask ChatGPT to update the Google doc with our flight plan, and, with any other generally useful context we created along the way. We can then start another fresh chat session, link the doc again, and work through hotels in the same way. Note that the chat session that’s helping with hotels does not need to know about the hundreds/thousands of words sent back and forth about if we should go with airline a, b, or c, or if we wanted a window seat or an aisle seat, or how many airline points we were willing to spend to upgrade to premium economy etc. The hotel session just needs to know where we’re going, and when we’ll be there, which it can grab from the Google doc. When hotels are sorted, the hotel chat session can update the Google doc, and we can then repeat this process for every other discrete logical grouping of decisions (e.g. figuring out the itinerary for the first city, then the second city, then how to get from the first to the second city etc.), each time keeping the doc updated.
In this scenario, the Google doc is functioning as our external memory system. It allows us to split the job up into smaller chat sessions that each focus on solving one specific problem, while knowing just enough about the big picture goal. And notice how, if we had a wide enough computer monitor, we could run all of these sessions in separate tabs all at the same time. We could even divide the work up between two people with two ChatGPT accounts (or one person with a ChatGPT account, and another with a Claude account). All of these simultaneous sessions would stay in alignment because they are pointing at the same external memory system that we own and keep.
If you have no interest in how software is made, you’re good for now. Thanks for reading. In subsequent writings, I will drop a bit more information on more robust and general external memory solutions than ‘just put it in a Google doc’, but you’ve got the core idea now. Please reach out to me if anything here doesn’t make sense, and feel free to share this with friends and family if you think it will help.
If you are interested in how software is made (e.g. you are an engineer, designer, product manager etc.), you have probably already picked up on the fact that the travel planning document is effectively a codebase that agents can build on top of. Having multiple chat sessions all working on this document at once is a multi-agent / team-based workflow. The ‘more robust solution’ to ‘just put it in a Google doc’ is git. Let’s keep extending this Google doc analogy in three ways:
Imagine that the Google doc was not just a travel plan, and that instead it was a more complex document with instructions for a watchmaker on how to assemble a watch, or instructions for a shipyard on how to assemble a ship, or instructions for a computer on how to compile an app. This document would be huge. A monolith. If we wanted to use a coding agent to change just one, small, isolated component of the watch, or the ship, or the app, the agent would have to load the entire monolithic document into its context window (or in the best case, load much more than it needs for just its small change). To solve this, we would ideally want to break up our monolithic Google doc into a folder, with smaller Google docs inside, and with one central Google doc that functions as a map of the folder. Coding agents read the map, and find just the doc they need to update.
Even if we broke up the doc into a folder, this still doesn’t solve the problem of consistency and coherence. If we have many agents working for many people across many documents over a long period of time, these agents are very likely to diverge from each other in style (fonts, tone of voice, grammar patterns etc.). The solution to this is to have another separate doc (which is also referenced in the map), which tells all agents how to write in the same style. In the context of a software application, this doc is our design system (really, its own entire sub-folder, and not just one doc), and it contains rules for fonts, spacing, colours, icons etc. that agents can refer to instead of inventing their own. Before coding agents can merge their changes into the main Google doc that everyone shares (coding agents and human devs work in an offline copy, and then merge their changes when they’re done), other (AI or non-AI) programs check if they have respected the design system and automatically reject their merge if they haven’t.
Even if we made it trivial for agents to work on our complex folder of Google docs, with its map and its style guide, how would we know what work we wanted to do in the first place? How would we communicate and align this work across many teams of humans, where each human is running many coding agents all at once? In the case where the Google doc is simply a travel plan, we know that all that needs to be done is just flights, hotels, itinerary. In the case where the doc (or folder of docs) is instructions for assembling a watch/ship/app, it’s much harder to keep teams of people aligned on what specific improvements need to be made, why they are being made (and made now), and what good looks like. The solution is, as you’ve probably already guessed, yet another document (really, another folder of documents). This folder is our context repository. It contains a hierarchy of crystal clear explanations of what’s being done, why it’s being done, and what good looks like. Developers take documents from this context repo, and use them to generate smaller documents that are focussed on just one specific task, which they then force-feed into the context windows of their coding sessions.
The monolith break-up, the design system, and the context repo are the big deliverable pieces which will accelerate an AI-assisted product development process. Understanding the implementation specifics of these deliverables is a lot of work (and I will write about how to do it soon!), but the above is a very simple explanation of their general shape and why they are needed. All three are some sort of solution to the one fundamental problem of LLMs having no memory. Without this understanding, it’s much harder to justify this work, and much easier to do it wrong. The companies that do it right will be able to create software of such throughput and quality that they will reshape their fields entirely.
If you’re interested, there are two important detail questions about context windows that I’ll answer here:
Wouldn’t there be a physical limit to how long the context window could get?
There is. The latest models have context window caps of “1 million tokens”. Let’s unpack what that means. Whenever you send a prompt to the LLM, a part of the LLM called the ‘tokeniser’ converts the prompt (as well as the invisible context window doc) into a format the LLM can work with. To do this, it treats the prompt and the context window as one big slab of text, and splits it up into small chunks, which are then converted into numbers. These chunks are called ‘tokens’.
The LLM cannot process more than 1M tokens, and so when the context window approaches this limit, the harness compacts it, by either truncating it (chopping off the beginning of the doc), or by asking another LLM to summarise it. You might have experienced LLM quality suddenly falling off a cliff in long sessions. This is what happens when compaction deletes or dumbs down context that was important to you but not saved in a place that you control.
Wouldn’t it get unfeasibly expensive to have to process the entire session every time?
It does. If you were to resume a big old chat session with Fable 5, assuming a usage-credits/pay-as-you-go option (which is what most companies use for software developers), just sending one more prompt which only contained the word ‘hey’ would cost you (or your employer) at least $10 (Fable 5 is priced at $10 per million input tokens; that’s not including the cost of it writing a response). It’s worth understanding what frontier labs do to minimise this cost: caching.
Before an LLM can generate a response to the prompt and the context window, which the tokeniser has converted into a series of numbers, the part of the LLM responsible for inferring meaning (called the 'transformer’) has to transform that series of numbers into a different series of numbers, which has meaning encoded into it (an explanation of how this actually works is coming in a future doc). This transformed series of numbers is housed in the memory (‘cache’) of Nvidia chips sitting in a data centre somewhere. The LLM reads the numbers from this memory, and infers a response. This cache is the most expensive real estate in the world, and Anthropic* allows you to hold your numbers there for exactly 5 minutes**. If you send another prompt within 5 minutes of the previous one, the cache is still ‘hot’, and the transformer will not have to transform the entire session history from scratch, just the new part. If more than 5 minutes have passed, the cache is dumped (‘cold’), and the full transformer cost must be paid again. The 5 minute timer resets on every message.
*OpenAI and Google have similarly shaped cache rent models.
**Anthropic does offer an option to keep cache hot for 60 minutes, but, this comes at the cost of each individual cache write operation costing 60% more. This is sometimes worth it for some people in some scenarios. Do not enable this unless you know what you are doing.
Upcoming writings:
An explanation of the major programs of work which will enable a team or a company to adopt an effective AI-assisted product development process. This doc introduces the common vocabulary for that doc.
A comprehensive guide on how anyone who works with a computer could evolve their workflow in the light of an improved understanding of LLM memory. This doc is foundational knowledge for that doc.
A tutorial on understanding how an LLM generates a response (and how frontier labs train them to generate good responses). Anyone who works with AI, i.e. now almost everyone who works with a computer, will benefit from a clearer mental picture of how LLMs actually run. Consumer apps like ChatGPT and Claude.ai have given people a faulty mental picture, to what I believe is an enormous aggregate detriment. An accurate (but simplified) mental picture is well within the reach of anyone (and a full and perfect mental picture is within the reach of anyone who is open to a tiny bit of very simple maths). Absolutely none of this is required for understanding that LLMs have no memory.
👋🏿 Thanks for reading! Feel free to reach out to me if I can help, or to subscribe if you want to be notified about future writings.