Why not just put everything in the context window?

Context windows are finite and expensive per token, and long contexts dilute attention. For long-horizon agents the memory grows without bound, so retrieval over an external store — not an ever-larger prompt — is what scales.

← Writing

How Do AI Agents Remember Long Conversations?

Q: How do AI agents remember long conversations?

They store past turns in an external memory and retrieve the relevant pieces on each new query, rather than keeping everything in the context window. The quality-and-cost tradeoff is set by the retriever: a good design is a cascade that answers easy queries with a cheap lexical match and only escalates to expensive dense retrieval when needed.

Q: What is cascade retrieval?

Cascade retrieval runs a cheap retriever first (for example, a vectorized BM25 lexical index) and checks a confidence signal; if the top candidates are clearly good, it returns early, and only otherwise escalates to a costlier dense or reranked stage. This makes average latency track query difficulty instead of always paying the worst-case cost.

By Aojie (Justin) Yuan · August 2026 · ~6 min read

TL;DR — An agent remembers a long history by storing it in an external memory and retrieving the relevant pieces per query, not by stuffing everything into the context window. The hard part is cost: dense retrieval is accurate but slow. AgentIR uses a workload-adaptive cascade — a vectorized BM25 stage answers easy queries in ~0.9 ms and returns early; only harder queries escalate to a dense + reranked stage (~53 ms). Average latency then tracks how hard the query actually is.

Ask an agent "what did we decide last Tuesday?" after a thousand-message history and it faces a memory problem that no bigger context window really solves. The context window is finite, every token costs money and attention, and a long-lived agent's history grows without bound. The scalable answer is the same one search engines learned decades ago: don't hold everything in mind — retrieve.

Memory = storage + retrieval

Long-term memory for an agent is two things: a place to put past turns, and a way to find the few that matter for the current query. Retrieval quality is the whole game, and it comes with a brutal tradeoff:

Lexical retrieval (e.g. BM25) is extremely fast but shallow — it matches words, not meaning.
Dense retrieval (embeddings) captures meaning but is far slower and heavier per query.

Pick one and you either miss paraphrased memories or pay dense-retrieval cost on every trivial lookup. Most queries don't need the expensive path — but some absolutely do.

The idea: make retrieval adaptive to the query

Instead of a fixed pipeline, use a cascade that spends effort in proportion to difficulty. AgentIR runs three substrate stages concurrently — a SIMD-vectorized BM25 posting list, a dense BGE-small channel, and a time-partitioned temporal index — behind a cascade trigger:

Cheap when the query is easy; thorough when the query is hard.

Early exit — if the fast BM25 stage is confident (a clear score gap), return in ~0.9 ms. Most queries stop here.
Escalate — if it isn't, fall through to dense retrieval with reciprocal-rank fusion and a recency signal (~53 ms) for the top-k answer.

The trigger is the key: a small confidence test decides, per query, whether the cheap answer is good enough. Average latency collapses toward the cheap path while the hard queries still get the thorough one.

Why temporal partitioning matters

Conversational memory has a time axis that generic RAG ignores. "What did we decide last Tuesday?" is a recency-and-time query, not just a semantic one. A time-partitioned index lets the retriever bias toward the right window instead of searching all of history uniformly — a second adaptive axis alongside the cheap/thorough one.

Where this fits

Retrieval is how an agent's memory scales; auditing is how its actions stay accountable; charter governance is how a whole team of agents stays in budget. Cascade retrieval is the memory piece: the substrate that lets a long-horizon agent recall the right thing without re-reading everything.

Frequently asked questions

How do AI agents remember long conversations?

They store history in an external memory and retrieve the relevant pieces per query; a cascade retriever answers easy queries cheaply and escalates only hard ones.

What is cascade retrieval?

Run a cheap retriever first, check confidence, return early if it's clearly good, and only escalate to a costlier dense/reranked stage otherwise — so latency tracks query difficulty.

Why not just enlarge the context window?

Context is finite and costly per token, long prompts dilute attention, and an agent's history is unbounded — retrieval over a store scales where a bigger prompt doesn't.

Written by Aojie (Justin) Yuan. AgentIR is described in arXiv:2605.25092.