Virtual memory for an
LLM's attention

Long runs die the same way

The model fills its window, starts compressing its own history, and silently drops the one detail that mattered three steps ago. You get the runaway PR, the function rewritten twice, the build that falls apart at hour two. A bigger window only delays it, and a crammed one rots in the middle anyway.

The fix isn't more window. It's to stop throwing the overflow away — encode it to a pool on disk, and recover the exact slice when it's needed.

It's virtual memory, for attention

Map it onto an operating system and it clicks. The pager runs while the model generates, so most of the fetch hides behind its own thinking.

• MPO context chain — recall pulls the connected thread, not isolated nearest neighbours.
• Any backend — Ollama, llama.cpp, Hugging Face, or your own API-backed model.
• numpy-only core — vectors live on disk, mmap'd; only the index graph stays resident.

Measured, not asserted

One live, paid run: a reasoning model driven through a 40-turn agent session that overflows its window — 2,000-token window, 60 real microsoft/vscode issues, engine off vs on. $0.19, 14 June 2026. Raw artifacts are committed in the repo.

Scope, honestly. That run used a hosted reasoning model, not a local one — the mechanism is backend-agnostic, but the headline number is not a local number. The 2,000-token window is deliberately tiny to force overflow, so a realistic window shows a smaller (still real) gain. N = 20 recall turns, single run. It measures the engine, not the MPO chain: on this single-fact task the chain ties plain recall, and its multi-slice edge is synthetic-only so far. Full write-up →

Disk in, reach out

One number to choose. Roughly 2.2 KB per 512-token slice works out to ~233M tokens of reach per gigabyte, and RAM stays a formula rather than a mystery.

Session counts are for isolated pools, and roughly double on a 16 GB machine. Share one pool instead and the index is paid once — dozens of sessions fit, with CPU as the limit rather than memory, at the cost of isolation. A bigger pool always buys reach, never more concurrent sessions.

Honest about the word "unlimited"

It means reach, not attention. Your model keeps its native context window; the engine makes it reach a billion-token pool in slices, through fast retrieval. The whole thing rides on hit rate — when that's high the pool feels like one seamless context, and when it isn't, a miss looks like forgetting. That's the trade, stated plainly.