Memory Tax
Dissecting the Net Value of Agent Memory

01Leaderboard

Performance (%) of 11 memory systems on LongMemEval-s, PersonaMem-v2 and MINT-Eval.
Each run on Qwen3.5-4B, Qwen3.6-27B and GPT-5.6 Luna (low).

Memory systems usually pay far more cost than RAG, but often without a matching gain in accuracy.

Token cost

LLM calls

Bold = best, underline = second best per column. Cells are shaded by how far they fall below or rise above RAG (2rd).

02Memory Benefit vs. Memory Tax

We match each memory system against naive RAG (1 round) on the same questions. Some questions only the memory system gets right. That is the memory benefit. Other questions it fails but RAG gets right. That is the memory tax. What is left after the tax is the net value of building memory.

The same design choice wins one question and loses the next

• Any nontrivial memory process introduces deviations from the raw history, such as extracting atomic facts, linking sessions, anchoring time or rewriting content.
• These deviations are intended to better serve future queries, yet some may also have unintended harmful effects due to particular design choices.
• We identify some representative mechanisms of agent memory below, and the same mechanism can bring memory benefit on one question (left) and memory tax on another (right).

03Inside the Memory Tax

Memory benefit largely reflects the intended value of a memory design.
So we focus on the overlooked side — memory tax, and trace it along three dimensions, while also considering some evaluation-side factors.

1 · Faithfulness - How well is the information preserved?

Memory operations such as compression and rewriting may drop or distort evidence before retrieval begins, and such loss cannot be recovered later. To measure it, we first locate the raw turns that hold the evidence (gold raw turns) and the memory units built from them (gold memory units). We then answer each question with only its gold memory units. This takes retrieval out of the picture, so any remaining tax comes from information lost during construction.

Case studies: what each system kept from one sentence

Takeaway: Information loss with varying severity during the memory process becomes a direct cause of memory tax.

2 · Accessibility - How easily can stored information be accessed?

Even when evidence survives construction, it still has to be found at query time. We replay retrieval round after round over each memory store and record when all gold memory units have come back. Compared with raw chunks, memory systems recover fewer questions in the first round, so the evidence is buried deeper. Case studies point to three patterns: evidence is fragmented across units, diluted by unrelated content, or rephrased so it no longer matches the query.

Case studies: in which retrieval round the evidence comes back

RAG (1rd)

Takeaway: Memory process can reduce evidence accessibility by fragmenting, diluting, or rephrasing it, creating memory tax even when evidence is not lost.

Benchmark Headroom - The evaluation side makes the tax more visible.

Across more long-term memory benchmarks, simple retrieval already scores high: an IDF-keyword matcher passes 50% on some, and RAG-Hybrid reaches about 70%. Such results suggest some current evaluation settings are relatively retrieval-friendly, leaving quite limited headroom for heavy memory to improve over naive retrieval.

How the four simple baselines work

The four baselines are No Memory, IDF-keyword, RAG-Emb and RAG-Hybrid. IDF-keyword takes the two rarest words of the query (highest IDF in the corpus) and fills the top-k budget with chunks containing both, then either one, with no embedding, reranker or LLM involved.

When such a crude matcher already scores above 50%, many questions can be answered by matching one or two distinctive words, which leaves little headroom for memory processing.

Takeaway: When naive RAG already solves many queries, memory has less room for gains and even modest losses become apparent.

On the other hand, many real-world queries are also naturally retrieval-friendly, making such accurate recall tasks essential and meaningful in its own right. Current benchmarks contain a large share of such queries, and future evaluations could complement them with harder settings. Moreover, reporting memory systems alongside naive RAG can make remaining improvement room more explicit, which also helps clarify the contribution of built memory. RAG scores can also serve as a reference for calibrating difficulty and leaving meaningful room for more reliable evaluations.

04Generalizing Memory Tax

The benefit-tax view is not tied to conversational memory: any nontrivial memory operation can deviate from the source and have the double-edged effects.

1 · Experiential memory

Agents accumulate experience on training tasks and reuse it on test tasks (Qwen3.5-4B), across embodied (ALFWorld), web (WebShop, Mind2Web), coding (BigCodeBench) and tool use (AppWorld, τ²-Bench). Distilled workflows beat naive trajectory RAG on web tasks, while several memory methods fall behind it on tool use.

2 · World-model memory

For long-horizon scene consistency, directly retrieving and reinjecting past frames stays strong, while latent- or cache-based memories do not consistently beat it.

Method	Mem. form	C-as-M-eval		MemoBench
		SSIM ↑	1−LPIPS ↑	SSIM ↑	1−LPIPS ↑
I3DM	naive frames	.560	.704	.430	.476
WorldKV	raw KV	.372	.459	.379	.420
ReWorld	learned KV	.378	.510	.387	.422
MagicWorld	visual latents	.388	.341	.433	.412
FramePack	visual latents	.217	.312	.279	.303
HyDRA	mem. tokens	.320	.319	.355	.326

Example frame sequences for testing long-horizon scene consistency.

05Design Implications

We also provide some simple changes that reduce or compensate for memory tax on existing systems, without a new architecture and without test-time scaling such as deeper retrieval or stronger backbones.

BibTeX

@misc{he2026memorytaxdissectingnet,
      title={Memory Tax: Dissecting the Net Value of Agent Memory},
      author={Xuan He and Sining Zhoubian and Ruozhen Yang and Yuhao Wen and Zhineng Wang and Yuyang Hu and Ke Yang and Yuyu Zhang and Ping Nie and Jiawei Han and ChengXiang Zhai},
      year={2026},
      eprint={2610.34073},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.34073},
}